Your monitoring says the service is using 2 GB. Your allocation profiler says you freed almost everything. Both are right, because free() mostly doesn't return memory to the operating system — it returns it to glibc's allocator, which keeps it around for the next malloc(). Understanding exactly when memory does go back to the kernel explains a whole family of "memory leak" reports that aren't leaks at all, and why MALLOC_ARENA_MAX=2 shows up in so many production configs.
Free 200 MB, get 1.5 MB back
Here's the effect in about 40 lines of C, run on Ubuntu 24.04 with glibc 2.39. Allocate 200,000 chunks of 1 KiB, put one more small live allocation near the top of the heap, then free everything except that pin:
char **p = malloc(N * sizeof *p);
for (int i = 0; i < N; i++) { p[i] = malloc(1024); memset(p[i], 1, 1024); }
char *pin = malloc(1024); /* small live chunk near the top */
for (int i = 0; i < N; i++) free(p[i]);
free(p);
/* ... read VmRSS from /proc/self/status at each stage ... */The output:
start: RSS = 1424 kB
after 200k x 1KiB: RSS = 206440 kB
after free() of all: RSS = 204876 kB (pin still live)
after malloc_trim(0): RSS = 1760 kB (returned 1)Freeing 200 MB of allocations lowered RSS by about 1.5 MB. One explicit malloc_trim(0) call then released essentially all of it. Nothing here is a bug — every byte was tracked, coalesced, and sitting on glibc's free lists. The kernel just never heard about it.
Why free() can't give it back
glibc's malloc serves small allocations from a contiguous region — the classic heap, grown and shrunk with sbrk(2). A contiguous region can only shrink from the end. Per mallopt(3), glibc trims automatically only when the block of contiguous free memory at the top of the heap reaches M_TRIM_THRESHOLD — 128 KiB by default — and then releases it by moving the program break back down.
That "top of the heap" condition is the whole story of the demo above. The single live 1 KiB pin sits above 200 MB of freed chunks, so the contiguous-free-at-top region never gets large and automatic trimming never fires. One long-lived allocation in an unlucky position can hold megabytes of freed memory hostage. This is heap fragmentation in its most literal form: not wasted space between live objects, but freed space that can't be returned because of where the live objects are.
Large allocations escape this trap. Per malloc(3), requests above M_MMAP_THRESHOLD (default 128 KiB) are served as private anonymous mappings via mmap(2) instead of the heap. Each mapping is independent, so free() becomes munmap() and the pages go straight back to the kernel, immediately and unconditionally.
The threshold moves while you're not looking
So why not lower the mmap threshold and get eager release for everything? Because glibc actively fights you. The mallopt(3) man page documents that the mmap threshold is dynamic: whenever you free an mmap'd block larger than the current threshold, glibc raises the threshold to that block's size, up to DEFAULT_MMAP_THRESHOLD_MAX — 512 KiB on 32-bit systems and 4 MiB × sizeof(long), i.e. 32 MiB, on 64-bit ones.
The reasoning is performance: an mmap'd allocation costs a syscall, a page-zeroing pass, and TLB pressure on every alloc/free cycle, while a heap allocation is often just a free-list pop. A program that repeatedly allocates and frees 1 MiB buffers teaches glibc to stop mmap-ing them. The side effect is that the same program's RSS behavior changes over its lifetime: early 1 MiB frees return memory instantly, later ones don't, and the transition point depends on the allocation history.
There's an escape hatch with a sharp edge: setting any of M_TRIM_THRESHOLD, M_TOP_PAD, M_MMAP_THRESHOLD, or M_MMAP_MAX explicitly disables the dynamic adjustment entirely. MALLOC_MMAP_THRESHOLD_=131072 (the trailing underscore is real, and required) pins the threshold at 128 KiB forever. That trades throughput for release eagerness — a reasonable trade for a memory-constrained container, a bad one for an allocation-heavy inner loop. Note that M_MMAP_MAX caps how many mmap'd allocations can exist at once (default 65,536), so "just mmap everything" has a ceiling too.
Threads multiply everything by eight
The single-heap picture above is already optimistic. Per malloc(3), when glibc detects contention on the allocator's mutexes in a multithreaded program, it creates additional arenas — independent heaps, each with its own lock and its own free lists. The hard limit is, in the man page's words, "usually calculated as a multiple of the number of available CPUs"; Heroku's engineering docs state it plainly for 64-bit systems: 8 × the number of cores.
Each arena hoards its own freed memory, and freed chunks in one arena can't satisfy allocations bound to another. On an 8-core box, a thread-happy workload can end up with 64 arenas, each fragmented, each holding its own retained pages. This is the mechanism behind the multithreaded-Ruby memory-bloat saga: Mike Perham (Sidekiq's author) identified the large number of glibc arenas in heavily multithreaded programs as the major cause of fragmentation — Sidekiq defaults to 25 threads — and reported one fleet's Sidekiq footprint dropping from 40 GB to 9 GB after switching the allocator to jemalloc. Heroku went further and has set MALLOC_ARENA_MAX=2 by default for all new applications since September 24, 2019, accepting some lock contention in exchange for a bounded arena count.
That's the actual tradeoff being made when you copy MALLOC_ARENA_MAX=2 into a Dockerfile: fewer arenas means fewer places for freed memory to hide, and more threads serialized on the same allocator lock.
malloc_trim is stronger than its man page's reputation
The demo's punchline — RSS dropping from 205 MB to 1.7 MB — works because of a quiet change in glibc 2.8. Before then, malloc_trim(3) could only shrink the main heap from the top, exactly like automatic trimming, so a pinned heap stayed pinned. Since 2.8 it walks all arenas and releases every whole free page anywhere in the heap (via madvise-style page-granular release), not just the tail. The pad argument — how much slack to leave for future growth — is honored only by the main heap; the function returns 1 if any memory was actually handed back.
The limitation is granularity: a page with even one live 16-byte allocation on it cannot be released. malloc_trim defragments your RSS, not your address space. But for the common shape of the problem — a burst of allocation, a burst of freeing, a long quiet period — calling it at the end of the burst is a legitimate tool, and it's what some long-running daemons do after cache flushes.
What to actually do
- Don't diagnose a leak from RSS alone. Rising RSS with stable live-heap size (from your language runtime's stats, or
malloc_info(3)) is retention and fragmentation, not a leak. RSS is a high-water mark with hysteresis, not a gauge of live data. - Thread-heavy and memory-constrained? Cap arenas (
MALLOC_ARENA_MAX=2) or switch allocators — jemalloc and tcmalloc manage fragmentation very differently. Measure both memory and throughput; the arena cap exists because the default optimizes the opposite thing. - Bursty allocation pattern? An explicit
malloc_trim(0)after the burst can return what automatic trimming structurally can't. - Watch the dynamic mmap threshold if your workload cycles buffers in the 128 KiB–32 MiB range: the allocator's release behavior for those sizes depends on history, and pinning the threshold changes both RSS and performance.
- One more container gotcha: since Linux 4.7,
RLIMIT_DATAcounts mmap'd allocations too, not just the brk heap — so limits tuned for the old accounting can start biting after a kernel upgrade.
The allocator isn't hiding memory from you out of malice. It's betting you'll allocate again soon, and for most programs that bet is right. The trouble starts when your memory limit enforcer — the kernel OOM killer, your container runtime, your autoscaler — grades the bet before it pays off.