Four Threads, Four Different Ints, 120 Cycles Each: False Sharing, and Why Folly Pads to 128 Bytes While GCC Ships 64

Four threads on a four-socket NUMA box, each hammering its own integer. No locks, no shared variable, no data race. Yet in Joe Mario's Red Hat Performance Engineering demo, the average time for each thread to execute its loop was 120 machine cycles — and after one change that alters zero program semantics, it dropped to 35. The change was padding: giving each int its own 64-byte-aligned cache line.

struct foo { int w; int x; int y; int z; };   /* 120 cycles */

typedef int __attribute__((aligned(64))) aligned_int;
struct foo { aligned_int w; aligned_int x;
             aligned_int y; aligned_int z; };  /* 35 cycles */

The threads never shared data. They shared a cache line — and the coherence protocol can't tell the difference.

Coherence has one unit of truth: the line

Your CPU promises that every core sees a consistent view of memory, and it keeps that promise at cache-line granularity — 64 bytes on x86. In a MESI-family protocol, a core that wants to write must first get the line in an exclusive state, which means every other core's copy gets invalidated. Those cores then stall and re-fetch the line — from the writer's cache, not from memory — the next time they touch any byte in it.

The Linux kernel's own false-sharing documentation boils the failure mode down to two conditions: a datum cached by multiple CPUs, and at least one writer among the concurrent accesses. Its canonical example is a struct where a frequently-modified refcount_t sits next to a char name[16] that is written once at creation and then only read. Every refcount bump forces every reader of name to throw away its copy and queue up for the line again. The name readers pay for writes to a field they never touch.

On a single socket this is a slowdown. Across NUMA nodes it becomes grotesque: the line ping-pongs over the interconnect, and Mario's deck notes that on busy systems with four or more sockets, false-sharing load latencies peaking over 60,000 machine cycles are not uncommon. That is a load instruction — nominally 4 cycles from L1 — costing as much as a small syscall.

HITM: the smoking gun

How do you prove this is happening, rather than guess? The signal has a name: HITM — a load that hit in a modified cache line. Your core asked for data, and the answer came from another core's cache where the line was dirty. One HITM is normal. Millions of them on the same line, from loads at different offsets than the stores, is false sharing by definition.

perf c2c ("cache to cache"), which Mario's team upstreamed into Linux in 2016, is built around exactly this event:

$ perf c2c record -ag sleep 3
$ perf c2c report --call-graph none -k vmlinux

The report ranks cache lines by HITM count and splits them into local HITM (modified copy on the same NUMA node) and remote HITM (modified copy on another node — the expensive one). For each hot line you get the offsets being read and written, the PID, the function, and the source line. You then feed the struct through pahole to map offsets back to member names. In the kernel doc's worked example — will-it-scale's tlb_flush1 case — a single run surfaces 92,117 local-HITM loads concentrated on a handful of lines.

This workflow produces real fixes, not just slides. The kernel doc links the commits: 91b6d3256356 cache-aligned tcp_memory_allocated and tcp_sockets_allocated; 802f1d522d5f re-laid-out struct page_counter; 520f897a3554 converted ext4's extent-status cache hit/miss counters to percpu_counters because two global counters bumped on every lookup were a line-contention machine.

So pad to 64 bytes. Right?

Here's where it gets contentious. C++17 added a pair of constants for exactly this — proposed in P0154R1 by JF Bastien and Olivier Giroux:

std::hardware_destructive_interference_size — "the minimum recommended offset between two concurrently-accessed objects" (pad to this to avoid false sharing);
std::hardware_constructive_interference_size — "the maximum recommended size of contiguous memory occupied by two objects" (pack within this to get true sharing for free).

Both are hints, not guarantees, and both must be at least alignof(max_align_t). Simple enough. Then look at what the major implementations actually chose for x86-64.

Folly says 128. Facebook's folly/lang/Align.h defines its mimic of the constant as:

cpp

//  Microbenchmarks indicate that pairs of cache lines also see destructive
//  interference under heavy use of atomic operations, as observed for atomic
//  increment on Sandy Bridge.
//  We assume a cache line size of 64, so we use a cache line pair size of 128
//  to avoid destructive interference.
constexpr std::size_t hardware_destructive_interference_size =
    (kIsArchArm || kIsArchS390X) ? 64 : 128;

The culprit behind the "pair of cache lines" behavior is usually attributed to Intel's L2 spatial prefetcher, which tries to complete a fetched line's 128-byte-aligned pair. In the GCC libstdc++ thread implementing the constants, Noah Goldstein relayed the claim that on Intel parts with a private L2, the spatial prefetcher "can cause destructive interference along 128 byte aligned boundaries" — and Matthias Kretz pushed back in the same thread, noting Intel documents the behavior for Sandy Bridge and questioning whether a prefetched pair line causes ongoing invalidation traffic at all.

GCC says 64. The constants shipped only in GCC 12 — five years after C++17 — precisely because nobody wanted to be wrong in an ABI-relevant way. I checked the shipping behavior directly: GCC 13.3.0 on x86-64 Ubuntu 24.04 returns destructive=64 constructive=64, and it returns 64 even under -mtune=sandybridge or -mtune=znver3. GCC also added escape hatches: --param destructive-interference-size=, --param constructive-interference-size=, and a -Winterference-size warning, per the GCC 12 release notes.

Clang says nothing. On the same machine, Ubuntu's Clang 18.1.3 fails to compile a reference to std::hardware_destructive_interference_size at all: no member named 'hardware_destructive_interference_size' in namespace 'std'.

Why so much cold feet over one integer? Because the constant leaks into ABI. The obvious use is alignas(std::hardware_destructive_interference_size) on a struct member — which bakes the value into object layout. If libstdc++ ships 64 today and decides 128 tomorrow, every binary that used the constant across a library boundary silently disagrees about struct size. In the GCC thread, Jason Merrill's position was blunt: users who prioritize ABI stability probably shouldn't use these variables at all. Folly dodges the problem by defining its own constant and never changing it; the standard library doesn't have that luxury.

The tooling already voted

Whatever you think of the Sandy Bridge evidence, the adjacent-line effect is real enough that the kernel's tooling grew a flag for it: in 2023, Feng Tang's perf c2c --double-cl patch added reporting at 128-byte granularity, specifically to catch false sharing between adjacent cache lines that hardware prefetching couples together. If 64 bytes were always the whole story, that option wouldn't exist.

What to actually do

The kernel documentation's guidance is more interesting than "pad everything":

Group by access pattern, not by meaning. Read-mostly fields together; fields written together, together; frequently-read away from frequently-written.
Read before write. if (!test_bit(X)) set_bit(X); — a read of a shared line is cheap; it's the store that invalidates everyone else. Same for atomic_read() before atomic_set().
Go per-CPU when a counter is hot enough — that's what the ext4 and TCP fixes above did, rather than just padding.
Don't over-mitigate. Padding costs memory and TLB reach; a fixed hotspot sometimes just relocates the contention; and a cold-path struct doesn't deserve 128-byte members. The doc is explicit that false sharing "does not always need to be mitigated."

And measure first. The whole point of HITM counters is that this is one of the few performance pathologies the hardware will name for you — a 3.4x slowdown with no shared data sounds like voodoo until perf c2c prints the exact line, the exact offsets, and the exact threads fighting over it.

References

false-sharingcache-coherenceperf-c2ccppnuma