Application Data Sync with WebDAV
Use WebDAV in your application to sync files (sqlite in my case) between devices. Storage backed by NextCloud, AWS S3, Google Drive, Dropbox and more
Context
I was recently building an Android Application where I wanted some very basic data sync between devices. I didn't want to spend any time building a backend, so I was looking for an alternative. AWS S3 has a library that looks pretty handy, however I may open source my application at some point and don't want to add the dependency.
A few months ago I setup a personal NextCloud instance as a replacement for Dropbox. Through this I was introduced to WebDAV for file syncronisation. It looked to be the perfect solution. I've been testing using it for other applications such as NextCloud, Joplin, DAVx5
Ben Brown
According to the docs cloud storage support includes:
Jackpot!
I based my implementation on the NextCloud WebDAV docs where they use cURL to manage the files:
Code
My example code here was Android/Kotlin and NextCloud. But the API calls should work from any language and you will be able to find the correct url by checking with the provider.
curl -u user:pass "https://<host>/nextcloud/remote.php/dav/files/<username>/directory/the-file.txt"
fun download(url: String, user: String, password: String): ByteArray {
val connection: URLConnection = URL(url).openConnection()
val credentials: String = Base64
.getEncoder()
.encodeToString("$user:$password".toByteArray(StandardCharsets.UTF_8))
connection.setRequestProperty("Authorization", "Basic $credentials")
val response: InputStream = connection.getInputStream()
val responseCode = (connection as HttpURLConnection).responseCode
if (responseCode >= 300) {
throw java.lang.Exception(connection.responseMessage)
}
val data = response.readBytes()
response.close()
return data
}curl -X PUT -T <file-to-upload> -u user:pass "https://<host>/nextcloud/remote.php/dav/files/<username>/directory/the-file.txt"
fun upload(url: String, user: String, password: String, data: ByteArray) {
val userAndPass = "$user:$password"
val connection: HttpURLConnection = URL(url).openConnection() as HttpURLConnection
connection.requestMethod = "PUT"
val encoded: String =
Base64.getEncoder().encodeToString(userAndPass.toByteArray(StandardCharsets.UTF_8))
connection.setRequestProperty("Authorization", "Basic $encoded")
try {
connection.doOutput = true
connection.setChunkedStreamingMode(0)
val out: OutputStream = BufferedOutputStream(connection.outputStream)
val start = System.currentTimeMillis()
out.write(data, 0, data.size)
out.flush()
out.close()
val responseCode = connection.responseCode
if (responseCode >= 300) {
throw java.lang.Exception(connection.responseMessage)
}
} finally {
connection.disconnect()
}
}
I'm using this to sync a SQlite database file between a few people.
- Pull latest when application loaded
- Save latest locally using path from <context>.getDatabasePath(<db_name>)
- Push file whenever changes are made
This solution isn't without faults, but for what I needed it's been perfect. Because of how little usage the file gets I'm not worried about any kind of race conditions. Users are unlikely to make modifications at the same time.
One extra benefit is now when I open source the application I can provide it with sync functionality without building/hosting/running any backend infrastructure.
Troubleshooting
NextCloud taking 20+ seconds for upload/download
While developing I noticed that upload/download calls started taking 20+ seconds to complete. In my case what was happening is I was being rate limited by NextClouds brute force protection:
kfincei
I was testing invalid credential handling which obviously triggered the restrictions. The post above gives some more information, but the way I fixed the issue was to remove all the rows in the NextCloud bruteforce_attempts table.
Invalid Credentials gives 404
I had expected using invalid credentials to return an Unauthorized (401) or Forbidden (403), however it actually returns File Not Found (404). The problem with this is that I needed to know whether or not to create a new Database file.
Are the credentials correct, and there is no database file on the remote server?
Are the credential incorrect, and there may be a database file on the remote server?
The way I work around this issue is to run a connection test. I simply upload a small (1 byte) file into the same directory as the file I want to download from. If this fails then I make the assumption that url, username, or password are incorrect.
WebDAV and 2FA/MFA
If you have 2FA/MFA setup with your NextCloud account you will quickly run into the issue that WebDAV only allows username/password combination. That's where Application Passwords come in (at least for NextCloud).
You can set these up from
Settings/Security/Devices & Sessions/Create new app password
This will give you a username/password combination to use with a specific app that can be individually revoked.
WebDAV and Access Controls
Following on from the App Passwords piece; one major drawback of WebDAV is filesystem access controls. As far as I can see if you give an application WebDAV access to one file, you give it access to every file for that user. This may be different for other providers (e.g. AWS S3).
I tightly control who has access to these credentials so I have not taken any action to mitigate this. One thing you could do is create separate NextCloud accounts for different apps.
Hopefully this is something we can have improved in future.