No space left on device but df shows free space

ENOSPC when df -h shows free space means the free space is not the thing that ran out. Three candidates, in order of frequency.

1. Inodes

Every file consumes an inode, and the count is fixed when the filesystem is created. Run out and you cannot create another file, however many gigabytes are free.

df -i

Filesystem      Inodes   IUsed  IFree IUse% Mounted on
/dev/nvme0n1p1 6553600 6553600      0  100% /

IUse% 100 is the answer. Find where they went:

sudo find / -xdev -type f -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head -20

-xdev keeps it on one filesystem, which stops it wandering into /proc and network mounts.

The usual sources are PHP sessions in /var/lib/php/sessions, a mail spool, a cache directory with millions of tiny entries, or a container runtime's layer store.

sudo find /var/lib/php/sessions -type f -mtime +7 -delete

Deleting many files at once can be slow and I/O heavy. find -delete is better than rm with a huge argument list, which fails with Argument list too long.

You cannot add inodes to an ext4 filesystem after creation. If this recurs, either fix what is creating the files or recreate the filesystem with mkfs.ext4 -i 8192 for a higher density. XFS allocates inodes dynamically and does not have this limit, which is one reason it is a better default for anything file-heavy.

2. A deleted file still held open

rm on a file removes the directory entry. The space is only reclaimed when the last file descriptor closes. A log rotated by deletion while the writer still holds it open frees nothing.

sudo lsof +L1

COMMAND   PID USER   FD   TYPE DEVICE     SIZE/OFF NLINK    NODE NAME
nginx    1247 www    12w   REG  259,1  42949672960     0  393218 /var/log/nginx/access.log (deleted)

NLINK 0 with a live process is exactly this: 40GB held by a file nobody can see.

Three ways out, in increasing order of disruption:

# 1. Ask the process to reopen its logs. Best.
sudo systemctl reload nginx
sudo kill -USR1 1247        # nginx's reopen signal

# 2. Truncate through /proc, freeing the space immediately
sudo truncate -s 0 /proc/1247/fd/12

# 3. Restart the service
sudo systemctl restart nginx

The /proc trick works because the descriptor is still a path into the file. It frees the space without stopping anything, and the process carries on writing at its previous offset, which leaves a sparse file that is harmless in practice.

The real fix is logrotate with copytruncate or a proper reopen signal, rather than anything deleting an open log:

/var/log/app/*.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    copytruncate
}

3. You are looking at the wrong filesystem

df -h /var/lib/docker /var/log /tmp /

A separate /var can be full while / has plenty. df -h alone shows every mount, and it is easy to read the first line and stop.

Docker is the usual culprit on a build host:

docker system df
docker system prune -a --volumes    # destructive: read what it says first

--volumes deletes named volumes not attached to a container, which can be real data. Run it without the flag first and read the reclaim estimate.

Two things that look like this and are not

Reserved blocks. ext4 reserves 5% for root by default. A filesystem at "100%" for an ordinary user still has that reserve, so root can write and nobody else can.

sudo tune2fs -l /dev/nvme0n1p1 | grep -i 'reserved block count'
sudo tune2fs -m 1 /dev/nvme0n1p1

On a large data volume, 5% is a lot of wasted space, and 1% is ample.

A full /tmp on tmpfs. /tmp is often RAM-backed and sized as a fraction of memory, so it can fill while the disk is empty:

df -h /tmp

Finding the space

sudo du -xh --max-depth=1 / 2>/dev/null | sort -rh | head -15

-x stays on one filesystem. ncdu -x / is much nicer if you can install it.

Note that du reports what files use and df reports what the filesystem has allocated. A large gap between them is the deleted-file case above.

A checklist

1. df -i. 100% IUse means inodes, not space.
2. Find the directory with the file count, and delete by age with find -delete.
3. sudo lsof +L1 for deleted files still open.
4. Free them with truncate -s 0 /proc/PID/fd/N, or reload the service.
5. df -h <path> for the specific path, not just the first line of df -h.
6. Docker host → docker system df before pruning.
7. du and df disagreeing → deleted files held open.
8. Ordinary user out of space and root not → ext4 reserved blocks.

---

Frequently Asked Questions

Why does df show free space when I cannot create a file?

Because blocks are not the only finite resource. The filesystem may be out of inodes, which df -i reports separately and which cap the number of files regardless of their size. Or a process is holding a deleted file open, so the space is allocated but invisible to du and unreclaimable until the descriptor closes. Or the path you are writing to is on a different mount from the one you checked. Run df -i and lsof +L1 before anything else.

What are inodes and why do they run out?

An inode stores a file's metadata, and every file and directory needs exactly one. On ext4 the total is fixed when the filesystem is created, derived from its size on the assumption of average-sized files. Millions of tiny files break that assumption, so a filesystem with plenty of free bytes can have no inodes left. Typical causes are PHP session files, mail spools and cache directories. You cannot add inodes to an existing ext4 filesystem; XFS allocates them dynamically and does not have the limit.

How do I free space from a deleted file that is still open?

Get the process and descriptor from sudo lsof +L1, then either ask the process to reopen its files, with systemctl reload or a signal such as nginx's SIGUSR1, or truncate it in place with sudo truncate -s 0 /proc/<pid>/fd/<fd>. The truncate reclaims the space immediately without stopping the service; the process continues writing at its old offset, producing a sparse file that is harmless. Restarting the service also works and is the most disruptive option.

Why can root write when my user cannot?

ext4 reserves a percentage of the filesystem for the superuser, 5% by default. The reservation exists so that root can still log in and system daemons can still write when a filesystem fills, rather than the machine becoming unrecoverable. It means unprivileged writes fail at what looks like 100% while root has headroom. On a large data volume 5% is a substantial amount to hold back, and lowering it with tune2fs -m 1 is reasonable.

Why do du and df disagree about disk usage?

du walks the directory tree and adds up the files it can see, while df reports what the filesystem has allocated. Files that have been deleted but are still held open by a process appear in df and not in du, because they no longer have a directory entry. A large discrepancy between the two is therefore a strong signal to run lsof +L1. Sparse files and filesystem overhead produce smaller differences that are normal.