Linux minor vs major page faults: what ru_minflt and ru_majflt count

Run a command under /usr/bin/time -v and two lines appear partway through the output: Major (requiring I/O) page faults and Minor (reclaiming a frame) page faults. This post covers what those two numbers count on Linux, and why reading the major fault count as a number of disk reads goes wrong.

The sources are getrusage(2), proc_pid_stat(5), mmap(2), and time(1) from Linux man-pages 6.19, the pagemap page in the Linux kernel documentation (7.3.0-rc6 build), and the GNU time 1.10 manual. The field meanings described here are Linux’s. POSIX.1 specifies getrusage() but only the fields ru_utime and ru_stime, so the meaning of the fault counters doesn’t carry over to other Unix-like systems as-is.

The dividing line: was I/O needed?

In getrusage(2), the two fields in struct rusage carry these comments:

long   ru_minflt;        /* page reclaims (soft page faults) */
long   ru_majflt;        /* page faults (hard page faults) */

The man page describes them as follows:

• ru_minflt: the number of page faults serviced without any I/O activity. It explains that I/O is avoided by “reclaiming” a page frame from the list of pages awaiting reallocation.
• ru_majflt: the number of page faults serviced that required I/O activity.

For a running process, the same values are field 10 (minflt) and field 12 (majflt) of /proc/<pid>/stat. proc_pid_stat(5) describes them as minor faults that have not required loading a memory page from disk, and major faults that have. Field 11 cminflt and field 13 cmajflt hold the values for the process’s waited-for children.

So the Linux documentation uses a single criterion to separate the two kinds: whether servicing the fault needed I/O (loading a page from disk). Whether the fault took a long time, or whether the memory touched was backed by a file, is not the criterion.

Why faults happen: demand paging

mmap() creates a new mapping in the calling process’s virtual address space. A mapping existing doesn’t mean every page in that range is in memory. The rss entry in proc_pid_stat(5) says the resident set size does not include pages which have not been demand-loaded in, or which are swapped out.

The MAP_POPULATE flag in mmap(2) (since Linux 2.5.46) shows the same fact from the other side. It populates (prefaults) the page tables for a mapping, and for a file mapping it causes read-ahead on the file, which helps reduce blocking on page faults later. The mmap() call doesn’t fail if the mapping cannot be populated. Since a separate flag exists for this, it follows that by default you can’t expect the pages to be populated when mmap() returns (an inference from the flag’s description). The MAP_LOCKED entry likewise says prefaulting can fail, so major faults might happen later, and recommends mmap() plus mlock(2) when major faults are not acceptable.

So the first access to a page that isn’t ready yet raises a fault, and the kernel services it. If servicing it needed disk I/O, it counts as major; otherwise, minor.

Why the major fault count is not a count of disk reads

File input goes into a different counter. Separately from the fault counters, struct rusage has ru_inblock (since Linux 2.6.22), which getrusage(2) defines as the number of times the filesystem had to perform input. GNU time prints this separately as %I (filesystem inputs). The grep example in the GNU time manual shows 46inputs and (43major+251minor)pagefaults together. They are different fields, and the documentation defines no one-to-one relationship between them. A program that reads a file with read(2) isn’t getting the data by touching mapped pages, so there’s no basis for assuming those reads show up in the major fault count (an inference from the definitions above).

Data that came from disk still counts as minor if it’s already in memory. The criterion is whether servicing that fault needed I/O. Even for a page of a file mapping, if the data was already in memory when the fault happened and the fault was serviced without I/O, it counts as minor (an inference from the definition). For example, if you run the same program twice in a row and the second run shows fewer major faults, that fits the explanation that file data read during the first run was still in memory. A single number can’t confirm the reason, though; which pages were in memory has to be checked with the separate tools described below. The memory fields shown by free are covered in the post on free’s available column.

Different wording in different tools

The output labels and tool documentation word things a little differently from getrusage(2):

• The GNU 1.7 section of time(1) describes %F as faults where the page has to be read in from disk, and %R as faults for pages that are not valid but have not yet been claimed by other virtual pages. In that case the data in the page is still valid, but the system tables must be updated.
• The GNU time 1.10 manual describes %F as major, or I/O-requiring, faults, where the page has actually migrated out of primary memory. The -v output labels are Major (requiring I/O) page faults and Minor (reclaiming a frame) page faults.

The BUGS section of time(1) says the selection of measured items was mostly inspired by the data provided by 4.2 or 4.3BSD, and getrusage(2) notes that its structure definition was taken from 4.3BSD Reno. Wording like “reclaiming a frame” appears to come from that lineage (an inference). When interpreting the numbers on Linux, use the criterion from getrusage(2) and proc_pid_stat(5): whether I/O was needed.

What the numbers don’t tell you

• Time. Both fields are counts. struct rusage has no field for time spent servicing faults. Even with many major faults, how much they added to run time has to be judged alongside a separate time measurement. Interpreting time’s real/user/sys is covered in the post on real, user, and sys.
• Interval. The values in /proc/<pid>/stat are cumulative counts of the faults the process has made so far. To see the effect of a specific piece of work, read them twice, before and after, and take the difference (a method derived from their being cumulative). getrusage(2) also says resource usage is preserved across execve(2), so values accumulated before an exec may be included.
• Child processes. RUSAGE_CHILDREN and cminflt/cmajflt include only children that have terminated and been waited for. Faults from children still running are not included.
• Which pages. The counters don’t say at which address a fault happened. The NOTES section of mmap(2) points to mincore(2) for finding which pages of a mapping are currently resident in the page cache. The kernel’s /proc/<pid>/pagemap gives one 64-bit value per virtual page, with bit 63 meaning page present and bit 62 meaning page swapped. Since Linux 4.0, though, getting physical frame numbers (PFNs) requires the CAP_SYS_ADMIN capability. Both tools show current state only; they don’t tell you whether a fault happened in the past.
• Swap. getrusage(2) classifies ru_nswap as unmaintained on Linux, and says the kernel sets unmaintained fields to zero. nswap in proc_pid_stat(5) is also not maintained. The GNU time manual notes that not all resources are measured by all versions of Unix, so some values might be reported as zero. If GNU time’s Swaps value comes from this field, it will always be 0 on Linux, so Swaps: 0 should not be read as evidence that no swapping happened. The documents checked for this post don’t state directly where GNU time gets this value, so that part is an inference.

Checking it yourself

To run one command and see the values with GNU time:

/usr/bin/time -v ./your-program
/usr/bin/time -f '%F major, %R minor' ./your-program

Some shells, such as bash, have a built-in time, so to use these options you may need to give a path, like the /usr/bin/time that time(1) uses as an example.

For a process that’s already running, read /proc/<pid>/stat. Replace 1234 with the target PID.

python3 -c 'import sys; s = open("/proc/%s/stat" % sys.argv[1]).read(); f = s[s.rindex(")") + 2:].split(); print("minflt", f[7], "majflt", f[9])' 1234

Field 2, comm, is the executable’s filename in parentheses, so it can contain spaces. That’s why the line splits from after the last ) instead of splitting on whitespace right away. The first value after that is field 3, state, so index 7 is field 10 minflt and index 9 is field 12 majflt. Run it twice, before and after a piece of work, and the difference is the number of faults in that interval.