The service refuses to write, the system says the disk is full. You look: 60% used. You delete the biggest log file you can find, and nothing moves.
Four causes cover almost every one of these situations. Taken in order, they take a few minutes to diagnose.
1. The deleted file that still occupies space
By far the most frequent cause, and the most counter-intuitive.
On Linux, deleting a file only removes its directory entry. The blocks are freed only once the last open descriptor on that file is closed.
The scenario is a classic: a service writes to a log that has grown out of all proportion. You delete it to make room. The service keeps its descriptor open — and cheerfully goes on writing into a now-invisible file, which goes on growing.
The signature is unmistakable: the usage tool reports a full disk, while walking the directory tree turns up nothing that justifies it. The gap matches exactly what was deleted without being closed.
The fix: identify the processes holding deleted files, then restart them or ask them to reopen their logs. And going forward, set up log rotation rather than deleting by hand — rotation tells the service, deletion does not.
2. Exhausted inodes
Here the disk genuinely is not full — something else is.
Every file consumes an inode, which stores its metadata. Their count is fixed when the filesystem is created and cannot be increased afterwards.
A system that accumulates millions of small files — application caches, web sessions, mail, queues — can exhaust that table while disk space remains plentiful.
The error message still talks about insufficient space, which sends you down the wrong path. Checking inode usage settles it: if it is at 100% while space is not, you have your answer.
The remedy is deleting the small files you do not need — often a forgotten cache directory. If the problem is structural, you have to recreate the filesystem with a higher inode density, or pick a filesystem that allocates them dynamically.
3. The blocks reserved for root
By default, ext4 reserves 5% of the blocks for the root user. An ordinary user therefore sees a full disk while space remains — space they are not allowed to use.
That reserve has two sound justifications: letting a system service still write when the disk fills up, and limiting fragmentation by avoiding a completely full filesystem.
On a system partition it is useful. On a multi-terabyte data volume, 5% means tens of gigabytes tied up for no real benefit.
The reserve can be adjusted live, without unmounting the volume. Bring it down to 1% or to zero on a pure data volume; leave it in place on the root partition.
4. The mount that hides files
Rarer, but thoroughly disorienting when it happens.
You write data into a directory. Later, you mount a disk on that same directory. The original files become invisible — the mount covers them — but they still occupy space on the underlying volume.
The typical scenario: a data directory filled for weeks before the disk meant for it was finally mounted. Tens of gigabytes vanish from view while still being counted.
To find them again, mount the root of the volume somewhere else and inspect the hidden path. The files are there, intact, simply covered up.
The false gaps
Two mechanisms produce legitimate differences between the stated size of files and the space actually consumed.
Sparse files declare a large size but only occupy the regions actually written: common for virtual machine disk images. They take up less than they advertise.
Snapshots do the opposite: they hold on to blocks that appear in no current file. The space is consumed, and no visible file accounts for it. On a system that handles snapshots, that is always the first thing to check.
The diagnostic order
- Compare the usage the system reports against the real sum of the directories. A gap points straight at deleted-but-open files or at snapshots.
- Check the inodes: it is instant, and it rules the second cause in or out.
- Hunt for deleted files still held open, and restart the processes involved.
- Check the mounts and inspect any directory that might have been filled before it was mounted over.
And run the analysis with the privileges it needs: a tool that is not allowed to read a directory skips it silently, and you end up hunting for missing space that was simply out of reach of the measurement.
Comments