The pitch for mmap as a database I/O layer is seductive. Call mmap() on your database file and the kernel hands you a pointer; dereference it and pages fault in from disk on demand. No read()/write() syscalls, no copying between the kernel's page cache and a user-space buffer, no buffer pool to write, no eviction policy to tune — the OS already has one. You get larger-than-memory databases almost for free.
Andrew Crotty, Viktor Leis, and Andrew Pavlo's CIDR 2022 paper, "Are You Sure You Want to Use MMAP in Your Database Management System?", is a catalog of how that pitch fails — with a benchmark where mmap keeps pace with direct I/O for exactly as long as the page cache has room, then falls off a cliff.
The hiring-and-firing pattern
The paper's table of mmap-based systems reads like a list of regretted decisions:
- MongoDB built its original MMAPv1 storage engine on
mmap(2009), deprecated it when WiredTiger became the default in 2015, and removed it entirely in 2019. - InfluxDB used
mmapfor file I/O, then replaced it after observing I/O spikes on writes once a database grew beyond a few GB — plus problems in containerized deployments without direct-attached storage. - SingleStore removed
mmap-based file I/O after finding itsmmapcalls were taking 10–20 ms per query — nearly half of total query runtime for simple sequential scans — due to contention on a sharedmmapwrite lock. Switching toread()made the same queries CPU-bound. - RocksDB exists as a fork of LevelDB partly because of read-path bottlenecks caused by LevelDB's use of
mmap.
And yet LMDB relies on mmap exclusively and credits it for the system's performance, QuestDB and RavenDB ship mmap-based engines, and SQLite offers it as an option. Both camps can't be right unconditionally — which is what makes the mechanism worth understanding.
Where the cliff comes from
The paper's headline experiment: random 4 KB reads over a 2 TB range on a 64-core AMD EPYC 7713 with the page cache capped at 100 GB, so ~95% of accesses miss. The baseline is fio with O_DIRECT, which sustains close to 900K reads/s with 100 threads — saturating the NVMe SSD.
mmap with MADV_RANDOM — the correct hint for this workload — tracks the baseline for about the first 27 seconds. Then the page cache fills, eviction begins, and throughput collapses to near zero for roughly five seconds before recovering to about half of what fio sustains. On sequential scans the gap is worse: striping across 10 SSDs took the fio baseline roughly 20× past mmap, which gained essentially nothing over a single SSD.
Three mechanisms produce this, and none of them is fixable from user space:
- TLB shootdowns. Evicting a mapped page means removing its page-table entry and invalidating it in every core's TLB. Flushing the local TLB is cheap; invalidating remote TLBs requires an inter-processor interrupt — thousands of cycles per shootdown. The paper measured shootdown rates via
/proc/interruptsclimbing into the millions per second once eviction started. - Single-threaded eviction. One
kswapdprocess handles page reclamation, and in the experiments it was fully CPU-bound — one kernel thread trying to free pages fast enough for 100 reader threads. - Page table contention. The shared page table becomes a synchronization hotspot with many concurrent threads, which is precisely the shared-write-lock contention SingleStore diagnosed.
Classic buffer pools avoid all three because a buffer pool is just malloc'd memory: eviction is a pointer swap under the DBMS's own latches, with no page-table mutation and no TLB traffic.
The correctness problems are quieter but worse
The performance cliff at least shows up in benchmarks. The correctness issues hide until a crash.
You cannot control when writes happen. With a writable shared mapping, the OS may flush a dirty page to the backing file at any time — mid-transaction, before commit. And mlock doesn't save you: per POSIX (and Linux's implementation), the OS is permitted to write dirty pages back even when they're pinned. The only thing mlock prevents is eviction, not writeback. Without that control, write-ahead logging's core invariant — log record reaches disk before the page it describes — can't be enforced by the straightforward design.
Every pointer dereference is a potential disk read, and a potential crash. There's no async interface to mmap, so a B+tree leaf scan can't issue overlapped reads to hide latency; any access may stall on a page fault. Worse, an I/O error that a buffer pool would see as a failed read() return code arrives instead as a SIGBUS delivered wherever the faulting instruction happened to be. SQLite's documentation states this plainly: an I/O error on a memory-mapped file results in a signal that kills the process unless the application catches it.
Stray writes become durable. A buffer pool can checksum a page once on read and validate before writeback. With a writable mapping, a wild pointer in a memory-unsafe codebase scribbles directly onto a page the OS will silently persist.
The workarounds eat the simplicity
Systems that stayed on mmap all ended up building the machinery mmap was supposed to eliminate. The paper sorts the update-safety protocols into three families:
- OS copy-on-write (MongoDB's MMAPv1): a second
MAP_PRIVATEmapping as a transaction workspace, plus a WAL, plus bookkeeping to propagate committed pages and periodicmremapto stop the private copy from growing into a full second database image. - User-space copy-on-write (SQLite, MonetDB, RavenDB): copy affected pages out to heap buffers, apply changes there, WAL the changes, copy back. SQLite in
mmapmode maps the file read-only and always copies pages to heap before modifying them — at which point you've reinvented half a buffer pool. - Shadow paging (LMDB): primary and shadow copies of the B+tree, flipped via
msyncand a pointer update — at the cost of allowing exactly one writer at a time.
The rebuttal worth hearing
Howard Chu's Symas response argues the paper benchmarks a strawman: fio is a filesystem benchmark, not a DBMS, and the comparison ignores the engineering cost on the other side of the ledger. LMDB's design dodges most of the listed hazards structurally — the mapping is read-only by default, so there are no dirty mapped pages to flush at the wrong time, eviction never needs writeback, and stray pointers can't corrupt the file. LMDB does all this in under 64 KB of object code, and Chu reframes the question as: are you sure you want to reimplement what the OS already does?
Note what the rebuttal concedes by design: read-only mapping plus a single writer. That's a real sweet spot — read-heavy, working set near memory — not a general answer.
The actual decision rule
The paper's closing prescription is blunt. Skip mmap if you need transactionally safe updates, can't tolerate unpredictable stalls on page faults, need to handle I/O errors and validate checksums correctly, or need high throughput on fast NVMe storage. Consider it when the working set (or whole database) fits in memory and the workload is read-only — or when you're rushing to market and accept the long-term ledger. SQLite's defaults encode the same judgment: mmap support exists, maps read-only, and ships off (PRAGMA mmap_size=0) unless you opt in.
The deeper lesson generalizes past databases: the kernel's page cache optimizes for a machine full of competing processes, with no knowledge of your transactions, your checksums, or your commit protocol. Transparency is the feature — and for a DBMS, transparency is exactly the problem.