Finding false sharing with perf c2c: reading code that fights over one cache line

Sometimes threads touch different variables and still get in each other’s way. Reading the code, nothing is shared, so no lock seems necessary. But CPU caches don’t move data around one variable at a time. They move whole cache lines. If two variables sit in the same line and one of them is written often, every other CPU reading or writing that line has to fetch it again and again. That is false sharing.

This post covers how to find such cache lines on Linux with perf c2c, which columns of the report to look at, and what you can do once you’ve found one. It is based on the False Sharing page in the Linux kernel documentation and the perf-c2c manual. There are no measurements or speedup numbers here. Results vary too much between machines, and a number would mislead more than it would help.

When false sharing happens

The kernel documentation names two conditions for harmful false sharing:

• There is global data accessed (shared) by many CPUs.
• Among those concurrent accesses there is at least one write: a write/write or write/read pairing.

Several read-only fields can safely share a line, provided nothing in that line is written. The documentation’s example is a struct whose refcount and name share a cache line. refcount is bumped frequently by one CPU; name is set once at creation and never changes. The other CPUs reading name still have to reload the whole line every time refcount changes, even though name itself was never modified.

The kernel documentation describes the cache line size as “usually 64 bytes or more”. Check the exact value for your platform.

The recurring patterns the documentation lists are worth knowing:

• A lock (spinlock, mutex, semaphore) deliberately placed in the same line as the data it protects. With few CPUs that layout helped cache efficiency, but on a large system under heavy lock contention, the lock owner writes to the data while other CPUs spin on the lock in that same line.
• Several small global variables (say, 4-byte parameters) that naturally end up grouped in one line.
• Members of a big structure that happen to sit together without anyone noticing. The documentation’s example is mem_cgroup.

The sharing can come from two completely unrelated components, or from different code paths in the same component. That is what makes it hard to diagnose. It rarely shows up in code review; you have to measure it.

What perf c2c records

C2C stands for Cache To Cache. The manual introduces the tool as a “Shared Data C2C/HITM Analyzer” for tracking down cache line contention. The events it uses provide:

• the memory address of the access
• the type of access (load or store, with details)
• the latency of the load, in cycles

perf c2c collects this data and reports the cache lines with the most HITM accesses first, i.e. the most contended lines. The workflow is the usual perf one: record to collect, report to view.

The important part is that these events depend on hardware features. The manual describes each architecture like this:

Architecture	Mechanism	Caveats from the manual
Intel	Load latency and precise store events	Default events are cpu/mem-loads,ldlat=30/P and cpu/mem-stores/P; CPUs with an AUX event use a grouped form
AMD	IBS op PMU	Not supported on Zen3 due to hardware limitations
PowerPC	Random instruction sampling with thresholding
Arm64	SPE sampling of loads and stores	Needs hardware and kernel support; sampling is statistical, so not every memory operation is sampled

So “just run perf c2c” is only half true. Which events you can use depends on the CPU model, the PMU, the kernel and the perf build. It pays to check what this system supports first:

perf c2c record -e list

This is how the manual says to list available events. If nothing shows up, or you get errors about opening events, sort out the environment before trying to read any report. Adding -v (--verbose) shows counter open errors in more detail.

On AMD, using the load latency filter (-l, --ldlat) requires, per the manual, that /sys/bus/event_source/devices/ibs_op/caps/ldlat contains 1, and the value must be between 128 and 2048 inclusive. Multiples of 128 have slightly less profiling overhead, and the filter is off by default.

A small reproduction

To keep things concrete, here is a piece of C that creates false sharing on purpose. Each thread increments only its own counter, but the two counters live next to each other in one struct.

#include <inttypes.h>
#include <pthread.h>
#include <stdint.h>
#include <stdio.h>

#define ITERS UINT64_C(100000000)

struct counters {
    volatile uint64_t a;   /* written only by the bump_a thread */
    volatile uint64_t b;   /* written only by the bump_b thread */
};

static _Alignas(64) struct counters c;

static void *bump_a(void *arg) {
    (void)arg;
    for (uint64_t i = 0; i < ITERS; i++) c.a++;
    return NULL;
}

static void *bump_b(void *arg) {
    (void)arg;
    for (uint64_t i = 0; i < ITERS; i++) c.b++;
    return NULL;
}

int main(void) {
    pthread_t t1, t2;
    pthread_create(&t1, NULL, bump_a, NULL);
    pthread_create(&t2, NULL, bump_b, NULL);
    pthread_join(t1, NULL);
    pthread_join(t2, NULL);
    printf("%" PRIu64 " %" PRIu64 "\n", c.a, c.b);
    return 0;
}

This demonstration assumes a 64-byte cache line. The counters are exact-width uint64_t, so they are 8 bytes each on every platform: a is at offset 0x0 and b at offset 0x8. _Alignas(64) puts the struct on a 64-byte boundary, so with 64-byte lines both counters sit in the same line. volatile stops the compiler from finishing each loop inside a register. Build as C11 with debug info so source lines show up in the report:

gcc -std=c11 -O2 -g -pthread fs.c -o fs

Recording

The manual’s basic form is perf c2c record [<options>] <command>. To look at just this program:

perf c2c record ./fs

To limit the events to user space use -u (--all-user); for kernel space only, -k (--all-kernel).

perf c2c record calls the regular perf record under the hood and turns on -W,-d,--phys-data,--sample-cpu by default. To pass any other perf record option, put it after --. The manual’s example enables call chains and system-wide collection:

perf c2c record -- -g -a

The kernel’s False Sharing page uses a short form that records the whole system with call graphs for three seconds:

perf c2c record -ag sleep 3

When you’re hunting in a service that’s already running, a short system-wide recording like this is easier to work with. When you can build a separate reproduction, recording just the target command is simpler.

Reading the report

perf c2c report --stdio

The report uses a TUI by default; --stdio sends it to standard output. -i (--input) picks a different perf.data file, and --stats shows only the statistics tables. The stdio output has four tables:

1. Trace Event Information: overall statistics of memory accesses
2. Global Shared Cache Line Event Information: overall statistics on shared cache lines
3. Shared Data Cache Line Table: the most expensive cache lines
4. Shared Cache Line Distribution Pareto: every accessed offset for each line

Tables 3 and 4 are where you will actually spend your time.

The cache line list

Each row of table 3 is one cache line. You get a zero-based Index, the line address in Cacheline, this line’s share of all HITMs (Rmt/Lcl Hitm), and HITM counts (LLC Load Hitm: Total, LclHitm, RmtHitm). Alongside are total loads and stores, whether stores hit L1 (Store Reference: L1Hit, L1Miss, N/A), and where loads hit (fill buffer, L1, L2, LLC, remote, DRAM).

The starting point is picking lines from the top with the highest HITM share. The default sort is total HITMs (tot); -d (--display) switches it to rmt, lcl or peer. On Arm64 the default is peer, and in that mode you get Load Peer and Peer Snoop columns, counting loads from a peer cache or DRAM, instead of HITM columns. The same problem can show up under differently named columns depending on the architecture.

Note that the report applies a default HITM % limit of 0.0005, so lines below it don’t appear in the list. To see every captured HITM line regardless of that limit, use --show-all.

The offset distribution

Table 4 is where you decide whether it’s false sharing, because it breaks a line down by offset. Each row has:

• HITM and store percentages for that offset
• the Offset within Data address
• the Pid and Tid that made the accesses
• the Code address, with its Symbol, Shared Object and Source:Line
• cycle sums for remote/local HITMs and generic loads
• how many CPUs took part (cpu cnt) and the Node

What you look for here is simple: within one line, are different offsets being written by different code? If so, that’s a false sharing candidate. If instead many writers pile onto a single offset, that is genuine contention on one variable. Moving fields apart won’t help there; the number of writes itself has to come down.

For the reproduction above, assuming 64-byte lines, the shape you’d expect is one top line with offsets 0x0 and 0x8, their code addresses split between bump_a and bump_b. Because this is sampling, which rows appear and how many samples they get varies between runs and between machines. No measured output is included in this post.

The grouping used for the offset rows is set with -c (--coalesce). The available fields are tid, pid, iaddr and dso, with pid,iaddr as the default. To split things per thread:

perf c2c report --stdio -c tid,iaddr

-N (--node-info) switches the Node column from bare node IDs to per-node CPU counts with HITM/store percentages, or to a CPU list. In the TUI the n key does the same.

A real example from the kernel docs

The kernel documentation shows part of a report taken while running the will-it-scale tlb_flush1 test with:

perf c2c record -ag sleep 3
perf c2c report --call-graph none -k vmlinux

The summary shows Load Local HITM at 92117 and Load Remote HITM at 139. In the offset table for one cache line below it, offset 0x8 is hit by __mod_lruvec_page_state, folio_lruvec_lock_irqsave, lru_add_fn and release_pages, and offset 0x10 by __mod_memcg_lruvec_state, with source locations memcontrol.h:752, mm_inline.h:41 and memcontrol.c:736. You can see at a glance that different functions are hammering two adjacent offsets in one line. The summary also shows local HITMs vastly outnumbering remote ones. When you’re looking at kernel data, point -k at the vmlinux path like this.

From offset to field name

The report gives you a line address and an offset. Matching that to a member of a particular struct is a separate step. The kernel documentation suggests these tools:

• pahole decodes structure layouts at cache line granularity. Line up the perf c2c offset with pahole’s output to find the exact member.
• System.map: for global kernel data, search for the data address here.
• addr2line helps decode an instruction address when there are several layers of inlined functions.

Sometimes lines seem to move in 128-byte units rather than 64. The manual explains that on some architectures an Adjacent Cacheline Prefetch feature makes cache line sharing behave as if the line size were doubled, and offers --double-cl to group detection at double-line granularity. If the offset table seems tangled up with fields from a neighbouring line, take another look with this option.

What to do once you’ve found it

The kernel documentation lists four mitigations, each with real kernel commits as examples:

1. Give hot global data its own cache line, even if it’s just a short. The cost is more memory, cache lines and TLB entries. (e.g. commit 91b6d3256356, “net: cache align tcp_memory_allocated, tcp_sockets_allocated”)
2. Reorganize the structure so interfering members end up in different lines. This can introduce new false sharing between other members. (e.g. commit 802f1d522d5f, “mm: page_counter: re-layout structure to reduce false sharing”)
3. Replace writes with reads where possible, especially in loops: read first and write only when needed instead of writing unconditionally. The documentation’s examples are if (!test_bit(XXX)) set_bit(XXX); instead of a bare set_bit(XXX), and for atomics if (atomic_read(XXX) == AAA) atomic_set(XXX, BBB);. (e.g. commits 7b1002f7cfe5, 292648ac5cf1)
4. Split hot global data into “per-CPU data + global data”, or reasonably raise the threshold for syncing per-CPU values into the global one, to reduce or postpone writes to it. (e.g. commits 520f897a3554, 56f3547bfa4d)

Applying 1 or 2 to the reproduction means putting padding between the fields:

struct counters {
    volatile uint64_t a;
    char pad[64 - sizeof(uint64_t)];
    volatile uint64_t b;
};

The 64 is only the line size assumed in this example. On a system with larger lines or adjacent-line prefetch it may not be enough. After the change, record again the same way and check that the line has dropped down the list.

The kernel documentation also suggests habits for new code: be aware of cache line boundaries, group mostly-read fields together, group fields written at the same time together, keep frequently read and frequently written fields on different lines, and leave a comment saying why the layout looks the way it does.

Before you fix anything

The kernel documentation is clear that false sharing doesn’t always need mitigating. Weigh the gain against complexity and space; there’s no need to hyper-optimize rarely used structures or cold paths. Every mitigation needs its own check for side effects.

Also, after finding and fixing a severe case, performance may still not improve noticeably, because the hotspot moves somewhere else. Measure before and after under the same conditions, and look at how the top lines in perf c2c changed too.

The documentation also lists an open issue: the kernel has an optional structure randomization mechanism, which also randomizes which members end up sharing a cache line. The same kernel source can give different results depending on configuration.

Versions and scope

• The options here follow the perf-c2c manual as checked on 2026-10-15 (the man7.org rendering, taken from upstream on 2026-08-04) and the kernel source tree documentation from the same day. If your installed perf is older, its options may differ; compare against the perf-c2c manual installed on your system.
• The latest upstream documentation adds a TUI view, reached with the TAB key, that groups by function instead of by cache line. The man7.org rendering doesn’t have it yet. If you don’t see it, that’s a version difference.
• Which events are used by default, whether you get HITM or peer columns, and whether the tool works at all depend on the CPU, PMU, kernel and perf build. The manual states that AMD Zen3 is not supported.
• The numbers in the report come from sampling. For Arm SPE in particular, the manual says outright that not every memory operation is sampled. It’s safer to use the report for the relative ranking of lines and offsets than for the counts themselves.

Sources

Checked on 2026-10-15.

• False Sharing (Linux kernel documentation)
• perf-c2c(1) (man7.org)
• tools/perf/Documentation/perf-c2c.txt (Linux mainline source)