In March 2018, Craig Ringer opened a thread on pgsql-hackers with a subject line that undersold the blast radius: "PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS." The bug it described had been in PostgreSQL for about as long as PostgreSQL had existed, and it wasn't really a PostgreSQL bug. It was a disagreement between database developers and kernel developers about what fsync() means — one that, two years later, a USENIX study would show also bites Redis, LevelDB, SQLite, and LMDB.

The retry that could never work

PostgreSQL's write path leans on the OS page cache. A backend modifies a page, writes it with a plain buffered write(), and moves on; the change is durable because it's also in the write-ahead log, which is fsynced at commit. Periodically, a checkpoint walks every dirty data file and calls fsync() on each one. Once every file reports success, the WAL up to that point is no longer needed and can be recycled.

The failure handling looked perfectly reasonable:

c
/* conceptually, the pre-2018 checkpointer */
if (fsync(fd) != 0) {
    ereport(ERROR, ...);   /* log it, fail this checkpoint */
    /* ... and retry at the next checkpoint */
}
/* fsync returned 0: everything written since the
   last successful fsync is on disk. Truncate WAL. */

The assumption, as Jonathan Corbet put it in LWN's writeup, was that "a successful call to fsync() indicates that all data written since the last successful call made it safely to persistent storage." So a failed fsync() was treated as retryable: leave the WAL alone, try again next checkpoint, and if the retry succeeds, proceed as if the first failure never happened.

Here's what Linux actually does when writeback hits an I/O error: it marks the affected pages clean and moves on. The data in those pages never reached disk, but as far as the kernel is concerned there is nothing left to write. So PostgreSQL's retry fsync() found no dirty pages, wrote nothing, and returned 0. PostgreSQL took that 0 as "everything is durable," truncated the WAL — which held the only surviving copy of those changes — and carried on with a database that was now silently missing or corrupting data.

The retry wasn't just ineffective. It was the step that converted a transient storage error into permanent data loss.

"A bad and undocumented convention"

The pgsql-hackers thread got heated because both sides were locally right. Anthony Iliopoulos, arguing the kernel's position, explained the design plainly: "the kernel opts for marking those pages clean (since there is no other recovery strategy), and reporting once to the caller who can potentially deal with it in some manner. It is sadly a bad and undocumented convention." He also pointed out that "even if the kernel retained the dirty bits on the failed pages, retrying persisting them on the same disk location would simply fail" — a dead sector doesn't heal because you ask twice.

Ted Ts'o supplied the practical rationale: the most common I/O error is someone yanking a USB drive mid-copy. If the kernel kept those pages dirty forever, unwritable pages would accumulate until they exhausted memory. Dropping them is self-defense.

The PostgreSQL side was less philosophical. Tom Lane called it "kernel brain damage"; Robert Haas called the behavior "100% unreasonable"; XFS's own Dave Chinner called it "a recipe for disaster." But the kernel behavior wasn't going to change, and it wasn't even Linux-specific — Thomas Munro reported that OpenBSD and NetBSD can also fail to report write errors to user space. Andres Freund noted that dpkg was affected by the same class of problem.

And the error-reporting story was worse than the clean-pages story. Before Linux 4.13, a writeback error was reported roughly once, to whichever file descriptor asked first — a sync() call elsewhere on the system could consume it. Worse for PostgreSQL specifically: the checkpointer often opens a file after the backend that wrote to it has closed it. On those older kernels, an error that occurred before the checkpointer's open() was invisible to it; its fsync() succeeded despite the lost write. Jeff Layton's errseq_t work in 4.13 fixed the single-witness problem — every descriptor open at the time of the error now sees it — but an error can still vanish if the inode is evicted from cache before the file is reopened.

The fix: stop trusting, start panicking

The fix that shipped — committed by Thomas Munro, authored by Craig Ringer, and back-patched to every supported release — is brutally simple. From the PostgreSQL 11.2 release notes:

By default, panic instead of retrying after fsync() failure, to avoid possible data corruption. [...] If we reissue the fsync() request it will succeed, but in fact the data has been lost, so continuing risks database corruption. By raising a panic condition instead, we can replay from WAL, which may contain the only remaining copy of the data in such a situation.

A PANIC crashes the whole server. On restart, recovery replays the WAL from the last completed checkpoint, regenerating the lost page writes from the log — which is correct precisely because the failed checkpoint never got far enough to truncate it. The notes concede the approach is "surely ugly and inefficient," but with buffered I/O there is no alternative: once the kernel has dropped the pages, the WAL is the only copy, and crash recovery is the only code path that reads it.

The escape hatch is the data_sync_retry GUC: default off, settable only at server start. The documentation's warning is a compact summary of the whole saga — on some operating systems a retry "may be reported as successful, when in fact the data has been lost." Set it to on only if you've verified your kernel keeps dirty pages after writeback failure. Linux does not. PostgreSQL 12 added further machinery to keep files open so the checkpointer reliably inherits errors, judged too invasive to back-patch. Operationally, the takeaway is to alert on PANIC in your logs: each one means your storage failed an fsync(), and the database chose a crash over corruption.

Everyone else fails too, just differently

In 2020, Rebello et al. at UW–Madison published "Can Applications Recover from fsync Failures?" at USENIX ATC. They built CuttleFS, a FUSE file system with its own user-space page cache, to emulate each real file system's failure behavior, and ran Redis 5.0.7, LMDB 0.9.24, LevelDB 1.22, SQLite 3.30.1, and PostgreSQL 12.0 against it. Findings:

  • All three file systems tested (ext4, XFS, Btrfs) mark pages clean after a failed fsync, so application-level retry is ineffective everywhere. But the content of those clean pages differs: ext4 and XFS keep the new, never-persisted data in memory, while Btrfs reverts to the previous consistent state.
  • Reporting differs too. ext4 in data-journaling mode sometimes fails not the fsync that lost the data but the next one. On a journal-block failure, ext4 remounts read-only while XFS shuts the file system down entirely.
  • Redis doesn't check fsync's return code at all — it acknowledges writes that were never durable, yielding old, corrupt, or missing keys.
  • SQLite in rollback mode can fail to write its rollback journal, then corrupt in-memory state by reading the bad journal back during rollback. LMDB reports failures for transactions that actually persisted.
  • The subtlest class: LevelDB, SQLite, and PostgreSQL can appear fine while the lost data lingers in the page cache, then silently serve stale data once eviction or a reboot purges it. Recovery code that reads through the page cache is validating against memory, not disk.

The paper's conclusion is the durable lesson: even after fsyncgate, "no application yet handles its failure perfectly," and recovery protocols must rely only on persistent on-disk state. fsync() is not a lever you can pull repeatedly until it works. Each call demarcates an epoch: the first failure is your only notification, the pages are already clean, and anything you haven't kept a copy of — in a WAL, in application memory, anywhere the kernel can't discard it — is gone. Keep data you can re-persist until fsync succeeds (write → fsync → free, never write → free → fsync), treat any failure as a signal to rebuild from durable state, and if that sounds expensive — that's why the kernel developers keep telling database authors to switch to direct I/O.

References

fsyncpostgresqlpage-cachedurabilitylinux-kernel