· Development

O_DIRECT alignment on Linux: ask statx instead of guessing

Linux O_DIRECT alignment of buffer address, offset and length varies by filesystem and kernel. Get it with STATX_DIOALIGN (6.1) and STATX_DIO_READ_ALIGN (6.14), and check stx_mask.

linuxdirect-iostatxsystem-calls

You open a file with O_DIRECT, call pread() or pwrite(), and get EINVAL. The same code runs fine on another machine. Alignment is the first thing to suspect, and rules of thumb like “512 bytes is enough” or “4096 is always safe” do not hold everywhere.

Since Linux 6.1 you can ask the kernel for the actual values. This post goes through what has to be aligned and how to get the numbers, based on the open(2) and statx(2) manual pages (Linux man-pages 6.19).

What O_DIRECT does, and what it doesn’t

open(2) describes O_DIRECT as trying to “minimize cache effects of the I/O to and from this file.” File I/O goes directly to and from user-space buffers. The same page says this will in general degrade performance, and that it is useful in special situations, such as when an application does its own caching.

It is not a durability guarantee. The man page says O_DIRECT makes an effort to transfer data synchronously but does not give the O_SYNC guarantee that data and the necessary metadata are transferred. To guarantee synchronous I/O you need O_SYNC as well. When a write actually survives a crash is covered in the write() and fsync() post.

The man page’s own summary is cautious: O_DIRECT is potentially powerful but should be used with care, and applications should treat it as a performance option that is disabled by default.

What has to be aligned

O_DIRECT may impose alignment restrictions on three things:

  • the memory address of the user-space buffer
  • the length of the I/O (for vectored I/O, the length of each segment)
  • the file offset

The awkward part is that none of this is fixed. According to the NOTES in open(2), Linux alignment restrictions vary by filesystem and kernel version, and might be absent entirely. Handling of misaligned I/O varies too: it can fail with EINVAL, or fall back to buffered I/O.

The second outcome is the more dangerous one. Nothing reports an error, so you end up doing I/O through the page cache without knowing your alignment was wrong.

There is one more place EINVAL comes from. If the filesystem doesn’t support O_DIRECT at all, open() itself fails with EINVAL.

Linux 6.1: STATX_DIOALIGN

Since Linux 6.1, putting STATX_DIOALIGN in the mask argument of statx() requests two fields:

Field Meaning
stx_dio_mem_align required alignment, in bytes, of user memory buffers
stx_dio_offset_align required alignment, in bytes, of file offsets and I/O segment lengths

A value of 0 means direct I/O is not supported on that file. One is nonzero only if the other is nonzero.

Support is narrower than you might hope. Per statx(2), block devices are supported since Linux 6.1, while support on regular files varies by filesystem. The filesystems the man page names are ext4, f2fs and xfs, since Linux 6.1. The page says nothing about the others, so don’t assume support; check at runtime as shown below.

Always check stx_mask

statx() does not promise to fill every field you ask for. The man page puts it this way:

  • In general the kernel does not reject unknown bits in mask. Instead it tells you, through stx_mask, which values this kernel and filesystem support.
  • If the filesystem doesn’t support a field, its bit is cleared in stx_mask even if you requested it, and a dummy value may be filled in for compatibility.
  • The kernel may also fill in fields you didn’t request, in which case their bits are set.

So if the running kernel or filesystem doesn’t support the fields you asked for with STATX_DIOALIGN, that bit is absent from stx_mask, and the field values may be dummies. Read stx_dio_mem_align without checking, and you may be aligning to a meaningless number. Check stx_mask first, every time.

For the same reason the man page warns against setting mask to UINT_MAX: some bits may later be used to signal an extension of the structure.

Linux 6.14: a separate offset alignment for reads

Linux 6.14 added STATX_DIO_READ_ALIGN, which requests stx_dio_read_offset_align: the alignment of file offsets and segment lengths for direct I/O reads.

  • If it is 0, stx_dio_offset_align applies to reads as well.
  • If it is nonzero, it is less than or equal to stx_dio_offset_align, so reads may have a looser requirement than writes.
  • The memory alignment in stx_dio_mem_align is not affected.

The support the man page lists is xfs, on regular files, since Linux 6.14. Because the value only means something relative to stx_dio_offset_align, request it together with STATX_DIOALIGN, and keep using stx_dio_offset_align for writes.

A C example

This gets the alignment values for an open file descriptor and checks whether a planned I/O fits them. The statx() wrapper has been in glibc since 2.28. Older headers that don’t define STATX_DIOALIGN and the new fields won’t build it; the STATX_DIO_READ_ALIGN part sits behind #ifdef so it compiles only when the header has it.

#define _GNU_SOURCE
#include <fcntl.h>
#include <stdbool.h>
#include <stdint.h>
#include <sys/stat.h>

struct dio_align {
    uint32_t mem;          /* buffer address */
    uint32_t offset;       /* offset and length for writes (and reads by default) */
    uint32_t read_offset;  /* offset and length for reads */
};

/* 0: values known, 1: not reported (use another method), -1: statx failed */
static int query_dio_align(int fd, struct dio_align *a)
{
    unsigned int want = STATX_DIOALIGN;
#ifdef STATX_DIO_READ_ALIGN
    want |= STATX_DIO_READ_ALIGN;
#endif
    struct statx stx;

    if (statx(fd, "", AT_EMPTY_PATH, want, &stx) == -1)
        return -1;
    if (!(stx.stx_mask & STATX_DIOALIGN))
        return 1;

    a->mem = stx.stx_dio_mem_align;
    a->offset = stx.stx_dio_offset_align;
    a->read_offset = a->offset;
#ifdef STATX_DIO_READ_ALIGN
    if ((stx.stx_mask & STATX_DIO_READ_ALIGN) &&
        stx.stx_dio_read_offset_align != 0)
        a->read_offset = stx.stx_dio_read_offset_align;
#endif
    return 0;
}

static bool dio_ok(const struct dio_align *a, const void *buf,
                   uint64_t off, uint64_t len, bool is_read)
{
    uint32_t step = is_read ? a->read_offset : a->offset;

    if (a->mem == 0)   /* direct I/O not supported on this file */
        return false;
    return (uintptr_t)buf % a->mem == 0 && off % step == 0 && len % step == 0;
}

Passing an empty path with AT_EMPTY_PATH makes statx() operate on the file that dirfd refers to. You could query by pathname instead, but this way you are asking about the very descriptor you will do I/O on.

If query_dio_align() returns 1, the alignment is unknown, not unrestricted. Move on to the next section.

When statx can’t tell you

open(2) lists what comes next:

  1. Filesystem-specific interfaces, for example XFS_IOC_DIOINFO in xfsctl(3) for xfs. The man page says to use STATX_DIOALIGN instead where it is available.
  2. Failing that, direct I/O support and alignment can only be assumed from known characteristics of the filesystem, the individual file, the underlying storage devices and the kernel version.

As a starting point for that guess, the man page gives the history. In Linux 2.4, most filesystems based on block devices required the file offset and the length and memory address of all I/O segments to be multiples of the filesystem block size (typically 4096 bytes). Linux 2.6.0 relaxed this to the logical block size of the block device (typically 512 bytes). You can get the logical block size with the ioctl(2) BLKSSZGET operation, or from the shell with blockdev --getss.

Take the word “typically” seriously. These numbers are grounds for an assumption, not a guarantee.

Other caveats worth knowing

Don’t run it concurrently with fork()

If the buffer is a private mapping (anything created with MAP_PRIVATE, which includes heap and statically allocated buffers), O_DIRECT I/O must never run concurrently with fork(2). Whether the I/O was submitted through an asynchronous interface or from another thread, it should complete before fork() is called. Otherwise the result can be data corruption and undefined behavior in both parent and child. The restriction doesn’t apply if the buffer came from shmat(2) or from mmap(2) with MAP_SHARED, or if it was marked MADV_DONTFORK with madvise(2) so the child never gets it.

Don’t mix it with buffered I/O or mmap

The man page advises against mixing O_DIRECT and normal I/O on the same file, especially on overlapping byte ranges. Even when the filesystem handles the coherency correctly, overall throughput is likely to be slower than with either mode alone. Mixing mmap(2) and direct I/O on the same file should be avoided too.

NFS behaves differently

The NFS protocol can’t pass the flag to the server, so only the client bypasses its page cache; the server may still cache the I/O. The Linux NFS client places no alignment restrictions on O_DIRECT I/O.

In short

Don’t hard-code alignment. Ask statx() per file, trust the result only after checking stx_mask, and treat “not reported” as “unknown” rather than “no restriction.” The separate read value exists only from 6.14 and only on some filesystems. And O_DIRECT itself is not a durability mechanism; it is a performance option that stays off by default and gets turned on where needed.

References

Back to the blog