Application Data Sync with WebDAV

Context

I was recently building an Android Application where I wanted some very basic data sync between devices. I didn't want to spend any time building a backend, so I was looking for an alternative. AWS S3 has a library that looks pretty handy, however I may open source my application at some point and don't want to add the dependency.

A few months ago I setup a personal NextCloud instance as a replacement for Dropbox. Through this I was introduced to WebDAV for file syncronisation. It looked to be the perfect solution. I've been testing using it for other applications such as NextCloud, Joplin, DAVx5

According to the docs cloud storage support includes:

• Amazon S3
• NextCloud
• Google Drive
• OneDrive
• Box.com
• DropBox

Jackpot!

I based my implementation on the NextCloud WebDAV docs where they use cURL to manage the files:

Code

My example code here was Android/Kotlin and NextCloud. But the API calls should work from any language and you will be able to find the correct url by checking with the provider.

curl -u user:pass  "https://<host>/nextcloud/remote.php/dav/files/<username>/directory/the-file.txt"

fun download(url: String, user: String, password: String): ByteArray {
        val connection: URLConnection = URL(url).openConnection()
        val credentials: String = Base64
                .getEncoder()
                .encodeToString("$user:$password".toByteArray(StandardCharsets.UTF_8))
        connection.setRequestProperty("Authorization", "Basic $credentials")
        val response: InputStream = connection.getInputStream()
        val responseCode = (connection as HttpURLConnection).responseCode
        if (responseCode >= 300) {
            throw java.lang.Exception(connection.responseMessage)
        }
        val data = response.readBytes()
        response.close()
        return data
    }

curl -X PUT -T <file-to-upload> -u user:pass  "https://<host>/nextcloud/remote.php/dav/files/<username>/directory/the-file.txt"

fun upload(url: String, user: String, password: String, data: ByteArray) {
        val userAndPass = "$user:$password"
        val connection: HttpURLConnection = URL(url).openConnection() as HttpURLConnection
        connection.requestMethod = "PUT"
        val encoded: String =
            Base64.getEncoder().encodeToString(userAndPass.toByteArray(StandardCharsets.UTF_8))
        connection.setRequestProperty("Authorization", "Basic $encoded")

        try {
            connection.doOutput = true
            connection.setChunkedStreamingMode(0)
            val out: OutputStream = BufferedOutputStream(connection.outputStream)
            val start = System.currentTimeMillis()
            out.write(data, 0, data.size)
            out.flush()
            out.close()

            val responseCode = connection.responseCode
            if (responseCode >= 300) {
                throw java.lang.Exception(connection.responseMessage)
            }
        } finally {
            connection.disconnect()
        }
    }
    

I'm using this to sync a SQlite database file between a few people.

• Pull latest when application loaded
• Save latest locally using path from <context>.getDatabasePath(<db_name>)
• Push file whenever changes are made

This solution isn't without faults, but for what I needed it's been perfect. Because of how little usage the file gets I'm not worried about any kind of race conditions. Users are unlikely to make modifications at the same time.

One extra benefit is now when I open source the application I can provide it with sync functionality without building/hosting/running any backend infrastructure.

Troubleshooting

NextCloud taking 20+ seconds for upload/download

While developing I noticed that upload/download calls started taking 20+ seconds to complete. In my case what was happening is I was being rate limited by NextClouds brute force protection:

I was testing invalid credential handling which obviously triggered the restrictions. The post above gives some more information, but the way I fixed the issue was to remove all the rows in the NextCloud bruteforce_attempts table.

Invalid Credentials gives 404

I had expected using invalid credentials to return an Unauthorized (401) or Forbidden (403), however it actually returns File Not Found (404). The problem with this is that I needed to know whether or not to create a new Database file.

Are the credentials correct, and there is no database file on the remote server?

Are the credential incorrect, and there may be a database file on the remote server?

The way I work around this issue is to run a connection test. I simply upload a small (1 byte) file into the same directory as the file I want to download from. If this fails then I make the assumption that url, username, or password are incorrect.

WebDAV and 2FA/MFA

If you have 2FA/MFA setup with your NextCloud account you will quickly run into the issue that WebDAV only allows username/password combination. That's where Application Passwords come in (at least for NextCloud).

You can set these up from

Settings/Security/Devices & Sessions/Create new app password

This will give you a username/password combination to use with a specific app that can be individually revoked.

WebDAV and Access Controls

Following on from the App Passwords piece; one major drawback of WebDAV is filesystem access controls. As far as I can see if you give an application WebDAV access to one file, you give it access to every file for that user. This may be different for other providers (e.g. AWS S3).

I tightly control who has access to these credentials so I have not taken any action to mitigate this. One thing you could do is create separate NextCloud accounts for different apps.

Hopefully this is something we can have improved in future.