Every serious storage system is built on one contract: write() hands data to the kernel, fsync() forces it to disk, and if fsync() returns success, the data is durable. In March 2018, PostgreSQL's developers discovered they had been misreading the fine print of that contract for twenty years — and that almost every other database had misread it the same way. The episode became known as fsyncgate, and it ended with PostgreSQL deliberately crashing itself as the fix.

The assumption everyone made

PostgreSQL's checkpointer works roughly like this: backends write dirty pages into the OS page cache with plain buffered writes; periodically, a checkpoint calls fsync() on the data files, and once every fsync succeeds, it trims the write-ahead log and advances the redo position — the point from which crash recovery would replay.

If an fsync() failed (disk error, dying SAN volume, thin-provisioned storage running out of space), PostgreSQL did the reasonable-looking thing: log the failure, keep the WAL, and retry the fsync at the next checkpoint. If the retry succeeded, the checkpoint completed and the WAL was trimmed.

Craig Ringer's report to the pgsql-hackers mailing list on March 28, 2018 — titled "PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS" — showed why that retry is a trap. He put the misunderstanding in one sentence:

When fsync() returns success it means "all writes since the last fsync have hit disk" but we assume it means "all writes since the last SUCCESSFUL fsync have hit disk".

What Linux actually does when writeback fails

When buffered writeback hits an I/O error on Linux, the kernel does not keep the dirty page around and try again. It typically marks the page clean anyway and reports the error on the next fsync() — once. After that, the error state is cleared. The page still sits in the page cache, still holding the new bytes that never reached the platter, and it is no longer scheduled for writeback.

So the failure sequence looks like this:

  1. Checkpoint calls fsync(); the kernel reports EIO from an earlier writeback failure.
  2. PostgreSQL logs the error and keeps the WAL, planning to retry.
  3. The kernel has already marked the failed pages clean and forgotten the error.
  4. The next checkpoint calls fsync() again. There is nothing dirty and no pending error, so it returns success.
  5. PostgreSQL concludes the data is durable, trims the WAL, and advances the redo position.

The data on disk is now whatever was there before the failed write. The only copy of the new version lives in the page cache, where a reboot, eviction, or memory pressure will erase it. Crash recovery can't save you, because the WAL that could have replayed those changes was truncated on the strength of that second, lying fsync(). Reads make it worse: until eviction, reading the file returns the page-cache copy, so the database happily verifies data that does not exist on disk.

When this surfaced on the list, Tom Lane's reaction captured the mood: "we need to push back on this kernel brain damage, because as you're describing it fsync would be completely useless."

It got worse: the error you never see

Losing the error on retry was only half the problem. The other half is which file descriptor an error is reported to.

PostgreSQL's checkpointer doesn't keep every data file open forever. It often opens a file just to fsync it. On Linux kernels in the 4.13–4.15 range, fsync() reported writeback errors that occurred after you opened the file descriptor. An error that happened before the checkpointer's open() was simply invisible — that fsync returned success on a file whose writeback had already failed. Error state could also be consumed by someone else entirely: another process, or even the kernel's own sync machinery, could observe the error first, leaving nothing for the fd that actually cared.

The kernel side of this story improved over time. Kernel 4.13 introduced the errseq_t infrastructure to make error reporting less lossy, and later kernels (4.14/4.16-era fixes) initialize a newly opened descriptor's error counter so that an unseen recent error still gets reported through it. But "the kernel reports the error once, to someone" is still the model, and older enterprise kernels stayed in production for years.

And this is not just a Linux quirk. The PostgreSQL wiki's survey of platforms found real divergence: Darwin/macOS and NetBSD invalidate the failed buffers after an fsync error; FreeBSD historically kept them dirty so retries were genuinely meaningful (with behavior changing around FreeBSD 11.1 for device-level failures); OpenBSD behaved like the invalidating group. Identical application code has different durability semantics on each.

The fix: crash on purpose

The PostgreSQL community's conclusion was blunt. If the kernel may have thrown away your dirty pages, there is exactly one copy of the truth left: the WAL. And the only way to rebuild from WAL is crash recovery. So starting with PostgreSQL 12 — and backpatched to 11, 10, 9.6, 9.5, and 9.4 — PostgreSQL now PANICs on fsync() failure. No retry, no error-and-continue: the server aborts and recovers by replaying WAL from the last known-good checkpoint.

The escape hatch, a GUC named data_sync_retry (default off), restores the old retry behavior, and its documentation doubles as the moral of the whole saga:

On some operating systems, the status of data in the kernel's page cache is unknown after a write-back failure. In some cases it might have been entirely forgotten, making it unsafe to retry; the second attempt may be reported as successful, when in fact the data has been lost.

The long-term fix discussed on the list — bypassing the page cache entirely with direct I/O, the way Oracle and friends do — was described at the time as "a metric ton of work." (Years later, that work materialized as PostgreSQL's debug_io_direct and the async I/O infrastructure, but that's another article.)

Everyone else was wrong too

In 2020, researchers at the University of Wisconsin–Madison put the broader ecosystem under the microscope in the USENIX ATC paper "Can Applications Recover from fsync Failures?" They injected targeted block failures under ext4, XFS, and Btrfs, then tested five widely used systems — PostgreSQL, LMDB, LevelDB, SQLite, and Redis — using CuttleFS, a FUSE filesystem that emulates each real filesystem's failure behavior.

The filesystem findings: all three filesystems mark pages clean after an fsync failure, so application-level retry is ineffective everywhere. But the details diverge — ext4 and XFS keep the latest (never-persisted) content in the page cache, while Btrfs reverts to the previous consistent state; and ext4 in data mode sometimes doesn't report the failure on the fsync that hit it, instead failing the subsequent call.

The application findings were grim across the board. Redis didn't even check fsync()'s return code before acknowledging writes, yielding databases with old, corrupt, or missing keys. LMDB could report failure to the user when the on-disk state was actually correct. SQLite could fail to write its rollback journal and then corrupt in-memory state by reading that journal back during rollback. And PostgreSQL — post-fsyncgate PANIC fix included in the study — could still lose both new and old data on updates in certain emulated failure modes. The paper's conclusion: applications "use many failure-handling strategies, but none are sufficient."

What to take away

  • A successful fsync() tells you nothing about writes that failed before it. On most systems, the error was reported once, earlier, to whoever asked first — durability is not cumulative.
  • Retry-on-fsync-failure is almost always a bug. The kernel has usually marked the pages clean; your retry fsyncs nothing and reports success. Treat the first failure as "page cache state unknown" and rebuild from a log you know reached disk.
  • Don't trust reads after a write error. The page cache will serve you bytes that were never persisted.
  • If durability is your product, buffered I/O is a liability. The page cache's failure semantics are subtle, kernel-version-dependent, and platform-divergent — which is why databases keep migrating toward direct I/O, where a failed write stays failed and stays yours to handle.

fsyncgate is a rare kind of bug: not a coding error but a two-decade misunderstanding between applications and kernels about what a syscall's return value means. The uncomfortable part is how it was resolved — not by making fsync retryable, but by admitting it never was.

References

fsyncpostgresqllinuxdurabilitypage-cache