Docker is using all my inodes
We recently had an issue where one of our servers was getting out of disk space errors when trying to pull new Docker images. This caught us by surprise as none of our disk usage alerts had fired. Once we got on the box to investigate we realised that it wasn't out of disk space in terms of bytes, it was out of disk space in terms of inodes.
This article does a good job of explaining what these are.
Dave McKay
but the quick summary is that you not only have a limit in terms of disk space, you also have a limit in terms of number of files. Inodes are metadata entries for files.
Back to our Docker issue. Further investigation showed that other nodes had inode usage at around 50%, this one particular node was up at 100%. We then found that the high usage node was still using the overlay filesystem instead of the newer overlay2 filesystem.
This article gives more information on the filesystem
The key point being
Note: If you use OverlayFS, use the overlay2 driver rather than the overlay driver, because it is more efficient in terms of inode utilization.
Sounds promising. So we decided to migrate this node to overlay2.
The steps involved in this were:
- Removing this node from any kind of LoadBalancer
- Stopping all Docker containers
- Make a backup of any volume data
- Run a Docker system/volume prune
- Stop the docker daemon
- Change the storage driver in docker config (see Docker Documentation posted above).
- Start the docker daemon
- Verify Docker is using overlay2 using 'docker info'
- Check the /var/lib/docker/overlay2 has been created
- Delete /var/lib/docker/overlay
- Start all your containers again
- Copy your backup data into the relevant new volumes
After this the issues were resolved and we had regained a large percentage of our inodes.