At midnight UTC on January 1, 2017, a leap second was inserted into civil time, and a value inside Cloudflare's custom DNS server went negative. RRDNS — Cloudflare's Go-based DNS software that picks upstream resolvers for CNAME lookups — measured round-trip times with one of the most ordinary lines of Go ever written:

go
rtt := time.Now().Sub(start)

When the system clock stepped backward over the leap second, that subtraction produced a negative duration. At peak, about 0.2% of DNS queries failed, and fewer than 1% of HTTP requests through Cloudflare saw errors. Impact started at 00:00 UTC and did not fully end until 06:45, even though the most affected machines were patched within 90 minutes.

The interesting part is not that a clock went backward. It's how far a single negative millisecond-scale measurement traveled through a production system before anything noticed — and what Go's maintainers decided to do about it, which is one of the more instructive API designs in any standard library.

The failure path: smoothing, then a panic in a random number generator

Cloudflare's upstream resolvers were tuned to answer in a few milliseconds. Against that baseline, a one-second backward step produced an enormous negative RTT sample — not a slightly-off number, but one three orders of magnitude outside the normal range, with the wrong sign.

RRDNS smoothed RTT samples over time, so one bad reading wasn't fatal. But the outlier was large enough that the smoothed average itself eventually went negative. That negative value flowed into weighted resolver selection, which passed it to rand.Int63n() — a function that panics on a negative argument.

Go's recover caught the panic, so the process didn't crash. The affected DNS lookups simply failed. The fix Cloudflare deployed was telling: don't try to make the time math correct — just discard all current upstream performance data whenever a negative RTT shows up, and let the system remeasure once time moves forward again.

Three layers each behaved reasonably and still composed into an outage: the clock did what POSIX wall clocks are allowed to do, the smoothing did what smoothing does (propagate an outlier rather than reject it), and rand.Int63n enforced its documented contract. The bug lived in an assumption nobody had written down: time does not go backwards.

Your kernel has been offering the right clock all along

Linux exposes several clocks through clock_gettime(2), and the man page is blunt about the one everybody uses by default:

CLOCK_REALTIME — A settable system-wide clock that measures real (i.e., wall-clock) time. ... This clock is affected by discontinuous jumps in the system time and by frequency adjustments performed by NTP and similar applications.

"Settable" and "discontinuous jumps" are load-bearing. Anything that subtracts two CLOCK_REALTIME readings is computing a quantity the clock never promised to make meaningful.

The clock for measuring elapsed time is CLOCK_MONOTONIC: on Linux it counts seconds since boot, and it is "not affected by discontinuous jumps in the system time." Two details are routinely missed:

  • It is affected by NTP frequency adjustments. NTP can speed it up or slow it down slightly to correct drift — it just can't make it jump. If you need a tick rate untouched by NTP, that's CLOCK_MONOTONIC_RAW.
  • It does not count time the system is suspended. If your laptop sleeps for an hour between two readings, the elapsed time is a lie. CLOCK_BOOTTIME is "identical to CLOCK_MONOTONIC, except that it also includes any time that the system is suspended."

So the primitive existed the whole time. The question was how a language should expose it.

Go's answer: don't add an API, hide a second clock inside time.Time

The obvious fix is what most languages do: a separate monotonic type or function (std::chrono::steady_clock, time.monotonic()), and documentation telling you to use it for durations. Russ Cox's design document for Go issue #12914 rejects that path with an argument about failure correlation: two parallel APIs would behave identically in the vast majority of executions, so programs that picked the wrong one would fail rarely — and, during an event like a leap second, simultaneously across your entire distributed system. An API whose misuse is invisible for years and then fires everywhere at once is the worst kind.

Instead, Go 1.9 changed what time.Now() returns with no new API at all. Per the release notes:

The time package now transparently tracks monotonic time in each Time value, making computing durations between two Time values a safe operation in the presence of wall clock adjustments.

Each time.Time from time.Now() now carries both readings, and operations pick the right one, following the design doc's governing rule: "the wall clock is for telling time and the monotonic clock is for measuring time." t.Sub(u), After, Before, Equal, and Compare use the monotonic readings alone when both operands have them, and fall back to wall-clock readings otherwise. Cloudflare's exact line of code became correct without being edited. The proposal estimated that about 29% of surveyed time.Now call sites were elapsed-time measurements that would be fixed with zero code changes.

The representation is a nice piece of bit-packing. time.Time used to be sec/nsec/loc; it became wall uint64, ext int64, loc. The wall field holds a 1-bit flag, a 33-bit seconds count since 1885, and a 30-bit nanoseconds field. When the flag is set, ext holds the monotonic reading as nanoseconds since process start; when clear, ext holds the full seconds since year 1, as before. The struct stays 24 bytes on 64-bit platforms (it grew from 16 to 20 bytes on 32-bit). The 33-bit seconds field means the dual-clock trick only works for wall times between 1885 and 2157 — outside that range, time.Time silently degrades to wall-clock-only.

The sharp edges the hybrid design bought

Transparency has a price, and Go's time docs enumerate it carefully:

  • == is a trap. The operator compares the location and the monotonic reading along with the instant, so two Time values for the same instant can compare unequal. Use t.Equal(u), and strip monotonic readings before using times as map keys.
  • t = t.Round(0) is the canonical strip. Round, Truncate, and AddDate are wall-time computations, so they drop the monotonic reading — and so do t.In, t.Local, and t.UTC.
  • Serialization strips it. Marshaling omits the monotonic reading and unmarshaling produces times without one. Send a timestamp over the wire and subtract it from a local time.Now() and you're back to pure wall-clock arithmetic, with every hazard that implies. The monotonic clock is meaningless outside the process that read it — it's nanoseconds since that process started.
  • Debuggability was handled. t.String() appends an m=±value field when a monotonic reading is present, so two times that differ only monotonically print differently instead of mysteriously comparing unequal.

The portable lesson doesn't require Go. Audit every duration you compute: if it comes from subtracting two wall-clock reads — System.currentTimeMillis(), time.time(), new Date() — it can go negative or jump by seconds, and anything downstream (smoothing, timeouts, rate limiters, TTL math, weighted selection) inherits that. Cloudflare's incident shows the blast radius of getting it wrong; Go's fix shows the cost of getting it right is a page of sharp-edge documentation and three bits of cleverness in a struct. And the defensive fix Cloudflare actually shipped — treat a negative duration as "discard and remeasure" — remains good advice in any language, because your process will eventually run on a machine where some code path still reads the wrong clock.

References

monotonic-clockleap-secondgolangclock-gettimecloudflare