· Development

Deleted a file but the disk space didn't come back? unlink and open files

Deleting a large file with rm doesn't always free space in df. The POSIX rules on link counts and open files, why du and df disagree, and how to find the cause and get space back on Linux.

DevelopmentOperating SystemsLinux

The disk fills up, you rm a huge log file, and df still shows the same free space. Run du on the directory and the file’s size is gone from the total. Neither tool is wrong. The file’s name is gone, but the file itself is still there.

df -k /var                    # free space on the file system, in KB
rm /var/log/app/huge.log
df -k /var                    # free space may not change
du -sk /var/log/app           # the directory total goes down

Removing a name is not removing a file

On Unix-like file systems, a directory entry, or file name, is a link to a file. One file can have several names through hard links, and POSIX defines st_nlink in the stat structure as the file’s number of hard links.

In POSIX, rm acts on each file as if by remove(), and remove() on a path that isn’t a directory is equivalent to unlink(). unlink() removes one name and lowers the link count.

When the space comes back is spelled out in the POSIX.1-2024 description of unlink(): the link count has to reach 0, and no process may still reference the file through an open file descriptor or a memory mapping. If a process holds such a reference when the last link is removed, the link is still removed before unlink() returns, but removal of the file’s contents is postponed until no such references remain.

The close() description covers the descriptor side: when the link count is 0, the space is freed once every file descriptor for the file is closed. The unlink() description also counts memory mappings as references, though, so if a file-backed mapping is still in place after the last descriptor closes, the space isn’t back yet. When a process terminates, its open file descriptors are closed and the mappings it created are unmapped.

So rm deletes a name. If another hard link remains, or some process has the file open or mapped, the data stays on disk.

Why du and df disagree

POSIX defines du as reporting the space allocated to the file hierarchy rooted at each operand, and df as reporting the available space on file systems.

A file with no name can’t be reached by walking directories, so it drops out of du’s total. Its space hasn’t been freed yet, though, so df’s free space doesn’t grow. That gap is a reason to suspect a deleted file that something still references. It isn’t proof, because other things can make the two numbers differ.

Finding it on Linux

From here on, these are features of Linux and of specific tools, not POSIX. The /proc details follow what Linux man-pages 6.19 documents.

  • lsof +L1: the lsof 4.99.0 manual says +L1 selects open files that have been unlinked. It lists only open files whose link count is below 1. To narrow the result to one process, add -a, which ANDs the selections, and -p <PID>.
  • /proc/<PID>/fd/: one symbolic link for each of the process’s open file descriptors, named after the descriptor number.
  • /proc/<PID>/maps: if a file-backed mapping’s file has been deleted, (deleted) is appended to the pathname. The manual notes that this marker is ambiguous.
  • /proc/<PID>/exe: if a running program’s executable is unlinked, this link gets the same (deleted) suffix.

Getting the space back

Every reference to the file has to go: all open descriptors closed and all file-backed mappings unmapped. The link count is already 0, so the space is freed as soon as the last reference disappears. Process exit normally does both.

This order keeps the impact low:

  1. Find every process holding the file. More than one process can reference it, and the unlink() description counts descriptors and mappings separately, so checking only one kind may not be enough. Look at lsof +L1 together with /proc/<PID>/maps for the processes involved.
  2. If the program documents a way to reopen its log files or perform a graceful reload, use that first. The mechanism differs by program, so follow its documentation.
  3. Only if there is no such option, consider a restart as a last resort, decided per service after weighing the impact.
  4. Afterwards, run df -k to confirm the free space actually went up. If it didn’t, check again for remaining holders or another hard link.

Running rm again does nothing, because the name is already gone.

Once the space has been freed, POSIX allows the file’s serial number, st_ino, to be reused.

This post covers only the POSIX rules and how to inspect the situation on Linux. Windows, network file systems, and file-system features such as snapshots can behave differently and are left out.

Back to the blog