· Development

write() succeeded. Is the data on disk? Short writes, late errors and fsync on Linux

A successful write() does not mean the data is on disk. Handling short writes, Linux 4.13+ writeback error reporting, what fsync and fdatasync cover, and when the directory needs fsync too.

DevelopmentOperating SystemsLinux

Code that saves a config file or a job result usually ends with open(), write() and close(). When write() comes back without an error, it is tempting to treat the save as done. The NOTES section of the Linux write(2) manual page says otherwise: a successful return from write() makes no guarantee that the data has been committed to disk.

A successful write() and “still there after a power cut” are separate stages. This post walks through what sits between them: how many bytes were accepted, when write errors are reported, what fsync() and fdatasync() push out, and why the directory holding the file’s name needs its own attention.

Linux behaviour here follows the Linux man-pages 6.19 pages for write(2), fsync(2), open(2) and rename(2). Portable requirements follow POSIX.1-2024.

The return value is the number of bytes accepted

write(fd, buf, count) returns the number of bytes actually written, which can be less than count. POSIX.1-2024 only requires that the value is the number of bytes actually written and never more than count.

Short writes happen on regular files too. The Linux write(2) page lists these cases:

  • the underlying device does not have enough space left
  • the RLIMIT_FSIZE file size limit is reached
  • a signal handler interrupts the call after some bytes were written
  • the request is larger than 0x7ffff000 (2,147,479,552) bytes, the most a single write() transfers on Linux, on both 32-bit and 64-bit systems

POSIX and the Linux page agree on signals. If the call is interrupted before anything is written, it returns -1 with EINTR. If it is interrupted after at least one byte, it succeeds and returns the count written so far. Calling write() again for the rest either writes more or fails, for example because the disk is now full.

So one look at the return value is not enough. Keep writing until nothing is left.

#include <errno.h>
#include <stddef.h>
#include <sys/types.h>
#include <unistd.h>

/* Write all len bytes, handling short writes and EINTR */
static int write_all(int fd, const char *buf, size_t len)
{
    while (len > 0) {
        ssize_t n = write(fd, buf, len);
        if (n < 0) {
            if (errno == EINTR)
                continue;      /* interrupted before writing anything: retry */
            return -1;
        }
        if (n == 0) {          /* no progress: treat as failure (defensive) */
            errno = EIO;
            return -1;
        }
        buf += n;
        len -= (size_t)n;
    }
    return 0;
}

What this loop guarantees is that the kernel has accepted every byte. It says nothing about the disk.

Write errors can arrive later

After write() returns, the data can stay in kernel memory and go to the device later. fsync(2) calls this the file’s modified in-core data, that is, its modified buffer cache pages. Where that memory shows up in free output is covered in the post on buff/cache and available.

If something fails during that later transfer (writeback), the original write() call has already returned and cannot report it. The EIO entry in Linux write(2) addresses this case directly:

  • EIO may relate to writeback of data written by an earlier write(), possibly through a different file descriptor on the same file.
  • Since Linux 4.13, writeback errors may be reported by later write() calls and will be reported by a later fsync(), whether or not write() already reported them.

The EIO entry in fsync(2) points the same way. Since Linux 4.13, writeback errors are reported to every file descriptor that might have written the data. Filesystems such as NFS that track which data came through which descriptor report more precisely; most local filesystems report the error to all descriptors that were open on the file when it was recorded.

Running out of space can also surface late. write(2) notes that on some filesystems, NFS included, a successful write() does not even guarantee that space was reserved, and errors may be delayed until a later write(), fsync() or even close(). fsync(2) adds that on filesystems that do not allocate space at write() time, fsync() returns ENOSPC or EDQUOT if an earlier write failed for lack of space.

That is why the return value of close() matters too. Checking it catches a late error; it does not make close() write anything to disk. According to write(2), the only way to be sure is to call fsync() after you have written all your data.

What fsync and fdatasync flush

Per Linux fsync(2), fsync() transfers the file’s modified data to the disk (or other permanent storage device), including writing through or flushing a disk cache if there is one, and blocks until the device reports that the transfer has completed. Along with the data it flushes the file’s metadata, such as size and timestamps.

fdatasync() skips metadata that is not needed to read the data back correctly. In the page’s example, a change to only the access time (st_atime) or modification time (st_mtime) needs no flush, while a change to the file size (st_size) does. The point is to cut disk activity for programs that do not need every piece of metadata synchronized. Appending to a file changes its size, so in that case fdatasync() still flushes the size.

You can also get the same effect on every write(). With open(2)’s O_DSYNC, each write() behaves as if followed by fdatasync(); with O_SYNC, as if followed by fsync(). O_DIRECT only tries to minimize cache effects, and on its own does not give the data and metadata guarantees of O_SYNC.

If fsync() fails, treat the save as failed. POSIX.1-2024 states that when fsync() fails, outstanding I/O operations are not guaranteed to have completed. Whether to rewrite from scratch or tell the user is up to the program.

Syncing the file does not sync its name

This is the step that often gets missed. A file’s name is not stored in the file; it is an entry in the directory that contains it. How a name and the file behind it can go separate ways is covered in the post on unlink and open files.

Linux fsync(2) says that calling fsync() does not necessarily ensure that the entry in the directory containing the file has also reached disk, and that this needs an explicit fsync() on a file descriptor for the directory.

For a newly created file, then, the contents may be on disk after fsync() while its name is a separate question. A rename() changes the directory as well, so the same gap applies.

The APPLICATION USAGE for fsync() in POSIX.1-2024 also says that an application that modifies a directory, for example by creating a file in it, can call fsync() on the directory to synchronize its entries. The same paragraph adds that for most applications this should not be necessary. So it is better seen as a tool for cases where the new name has to survive a failure than as a ritual to attach to every file operation.

Atomic rename and durability are different properties

A common way to replace a file safely is to write the new contents to a temporary file and then rename() it over the original name. Linux rename(2) says that if newpath already exists it is replaced atomically, so there is no point at which another process accessing newpath finds it missing. If the operation fails, an instance of newpath is guaranteed to remain.

That promise is about what other running processes see. It is not a request to flush data to storage, and rename(2) says nothing about which version survives if the system goes down. Atomicity means no intermediate state is visible; durability means the result survives a failure. If you want both, you have to take care of each.

open temp file
  → write_all          kernel has accepted every byte
  → fsync(temp file)   flush contents and metadata, check the error
  → close              check for a late error
  → rename             other processes see a single switch
  → fsync(directory)   flush the changed name

The order matters. If the temporary file is not synced before rename(), the new name can end up pointing at a file whose contents have not reached the device yet. As shown above, a successful write() does not guarantee a disk write, and rename() is not a call that flushes data.

When you turn this flow into real code, the step that needs the most care is the first one: opening the temporary file. Opening a fixed name such as save.tmp with O_CREAT | O_TRUNC can empty a file that was already at that path, or follow a symbolic link placed there and overwrite some unrelated file. It can also collide with another process using the same name.

So the temporary file should be created fresh, under a name nobody can predict, in the same directory as the target. mkstemp() is one way to do that; a carefully designed openat() on the opened directory with O_CREAT | O_EXCL is another. Linux open(2) says that with O_CREAT and O_EXCL together the call succeeds only if it creates the file, and fails if the path is a symbolic link, wherever the link points. Keeping it in the same directory matters too: rename(2) fails with EXDEV when the two paths are not on the same mounted filesystem, and notes that it does not work across different mount points even if the same filesystem is mounted on both.

Error handling follows the same order. If a step before the rename() call fails, the temporary name still belongs to the program, so remove the temporary file and report the save as failed. An error from rename() itself is less clear-cut. The BUGS section of Linux rename(2) warns that on NFS you cannot assume a failed call means the file was not renamed: the server may have done the rename and then crashed, and the retransmitted request fails once it comes back up. The page leaves it to the application to deal with this. So do not assume nothing happened and delete the temporary name blindly; treat the outcome as unknown and let the program’s own recovery design check which name holds what before cleaning up. When fsync() fails, treating it as a failure fits the previous section better than calling it again on the spot. If the directory fsync() fails after rename(), other processes can already see the new contents, but the durability of the name is unconfirmed. Report that as a failure as well, so the caller does not mark the save as complete.

What this does not promise

Everything above is what the system calls document. What actually survives a failure also depends on the layers underneath.

  • POSIX.1-2024 leaves the nature of the fsync() transfer to the implementation. Its RATIONALE says a null implementation is permitted on systems where _POSIX_SYNCHRONIZED_IO is not defined.
  • Linux fsync(2) notes that fsync() in older kernels and lesser used filesystems may not know how to flush disk caches, and that disk caches then need to be disabled for safe operation.
  • Filesystem type, mount options, storage firmware and network filesystems can all change what is left after a failure. This post does not cover each combination.
  • Programs that must update several files as one unit, as databases do, need a design of their own; the sequence here is not enough on its own.

The return value of write() is what the kernel accepted, since Linux 4.13 writeback errors are guaranteed to show up in a later fsync(), and a new name needs a separate fsync() on its directory.

Back to the blog