Subtracting Two Timestamps Gave −990 ms: The Leap Second That Panicked Cloudflare DNS and Put a Second Clock Inside time.Time

Every language makes it one line to write, and almost every codebase contains it somewhere:

start := time.Now()
doSomething()
elapsed := time.Now().Sub(start)

The hidden assumption is that the second reading is never earlier than the first. On the night of December 31, 2016, that assumption failed inside Cloudflare's DNS servers, elapsed came out negative, and a chunk of their fleet started panicking at midnight UTC. The root cause wasn't exotic: the code was asking a calendar to do a stopwatch's job.

Your operating system has (at least) four clocks

The word "clock" hides two different instruments. A wall clock answers "what time is it?" — it's synchronized to UTC, which means it must be adjustable: NTP slews it, administrators set it, and leap seconds occasionally deform it. A monotonic clock answers "how much time has passed?" — it starts at an arbitrary point and only moves forward, precisely because nothing is allowed to adjust it backwards.

Linux exposes both, plus variants, through clock_gettime(2):

Clock	Jumps if time is set?	NTP frequency slewing?	Counts suspend?	Epoch
`CLOCK_REALTIME`	Yes	Yes	Yes	1970-01-01 UTC
`CLOCK_MONOTONIC`	No	Yes	No	Unspecified (boot, on Linux)
`CLOCK_MONOTONIC_RAW`	No	No	No	Unspecified
`CLOCK_BOOTTIME`	No	(as `CLOCK_MONOTONIC`)	Yes	Boot

A few details from the man page are worth internalizing:

CLOCK_REALTIME "is affected by discontinuous jumps in the system time (e.g., if the system administrator manually changes the clock)." Every timestamp you subtract from another CLOCK_REALTIME timestamp inherits that hazard.
CLOCK_MONOTONIC never jumps, but it is affected by NTP frequency adjustments — the kernel may run it slightly fast or slow to discipline it. CLOCK_MONOTONIC_RAW gives you "a raw hardware-based time that is not subject to frequency adjustments" if you're doing fine-grained benchmarking.
CLOCK_MONOTONIC "does not count time that the system is suspended." CLOCK_BOOTTIME is identical except it includes suspend time — the right choice for "has this lease expired?" on a laptop or phone that sleeps.

So the primitive you need for timeouts, rate limiters, retry backoff, and latency measurement has existed in the kernel for decades. The trap is that most standard libraries made the wall clock the ergonomic default, and the monotonic clock an expert option.

The −990 millisecond round-trip time

A leap second is the wall clock at its most honest about being a calendar, not a stopwatch. UTC occasionally inserts a 61st second (23:59:60) to stay aligned with the Earth's rotation. Most systems can't represent 23:59:60, so in a common handling, entering the leap second the clock effectively replays a second: as Russ Cox's Go proposal describes it, the clock might read 23:59:59.995 at one instant and 23:59:59.005 ten milliseconds later. Subtract those two readings and you get −990 ms instead of +10 ms.

That is exactly what happened to Cloudflare on the leap second at the end of 2016. Their custom DNS server, RRDNS, measured upstream resolver performance for CNAME lookups with the pattern above:

rtt := time.Now().Sub(start)

Go's time.Now() at the time carried no monotonicity guarantee. One negative RTT sample wasn't fatal on its own — smoothing initially masked it — but the negative values accumulated until a smoothed RTT went negative. That value was eventually fed into weighted server selection, which called rand.Int63n(), which panics on a negative argument. In Cloudflare's postmortem: at peak roughly 0.2% of DNS queries were affected, fewer than 1% of HTTP requests saw errors, and only customers using CNAME records were impacted. The outage ran from midnight UTC until 06:45.

Two things make this incident a great teaching case. First, the bug was latent for years and only fireable during a leap second — a test suite will never catch it by accident. Second, the fix was trivial (clamp negative values, reset the metrics) but the class of bug was everywhere: any subtraction of two wall-clock readings in any process on any machine whose clock steps backward. NTP corrections do this routinely on machines with drifting clocks or misconfigured time sync; the leap second just did it to everyone simultaneously.

Go's answer: smuggle the stopwatch inside the timestamp

Java (System.nanoTime()), Python (time.monotonic()), Rust (std::time::Instant) and C (clock_gettime) all answer this with a second API, and make correctness the programmer's problem: use the wrong one and nothing warns you.

Go 1.9 (August 2017) did something more opinionated, following Cox's proposal written in January 2017, directly motivated by the Cloudflare outage. Instead of a new type or function, time.Now() now returns a time.Time that contains both readings: the wall clock reading, and a monotonic clock reading recorded at the same instant. The layout is tightly packed — 33 bits of wall-clock seconds and 30 bits of nanoseconds, a flag bit signaling whether a monotonic reading is present, and the monotonic value stored as nanoseconds since process start.

The semantics, straight from the time package documentation: if Times t and u both contain monotonic clock readings, then t.After(u), t.Before(u), t.Equal(u), t.Compare(u), and t.Sub(u) "are carried out using the monotonic clock readings alone, ignoring the wall clock readings." time.Since(start) and time.Until(deadline) are robust against wall-clock resets for the same reason.

So the exact line that took down Cloudflare's DNS — time.Now().Sub(start) — is now correct by construction. No code change, no education campaign, no lint rule. This is the rare API design where the safe thing and the ergonomic thing became the same thing.

The seams you can still fall through

The dual-reading design has edges, and they're all documented rather than hidden:

The monotonic reading silently evaporates. Wall-time computations — t.AddDate(...), t.Round(d), t.Truncate(d), and per the current docs also t.In, t.Local, and t.UTC — strip the monotonic reading from their results. So does serialization: MarshalJSON, MarshalBinary, MarshalText, and GobEncode all omit it, and time.Parse, time.Date, and unmarshaling always produce times with no monotonic reading. Round-trip a timestamp through JSON, Redis, or a database and your subsequent Sub is back on the wall clock — with the pre-2017 failure mode. This is unavoidable and correct: a monotonic reading is nanoseconds since this process started, meaningless to any other process or machine.
== is a trap. The Go docs are blunt: the == operator "compares not just the time instant but also the Location and the monotonic clock reading." Two time.Time values for the same instant can compare unequal. Use Equal, and don't use time.Time as a map key without stripping.
Stripping is idiomatic but obscure. "The canonical way to strip a monotonic clock reading is to use t = t.Round(0)." If you've ever seen Round(0) in review and wondered why anyone rounds to zero — that's why.
Suspend still bites. A monotonic reading based on a clock that stops during suspend means "elapsed time" across a laptop lid-close isn't what a human would say elapsed. If your expiry logic must survive sleep, you want CLOCK_BOOTTIME semantics, and you should check what your runtime actually uses on your platform.

What to do in your own code

Never subtract two wall-clock timestamps to measure a duration — not in cron-adjacent shell scripts (date +%s twice), not in SQL (now() - started_at for an in-flight operation), not in any language. If the value came from "what time is it?", it can step.
Assume clock steps are routine, not exotic. You don't need a leap second; an NTP correction on one bad host is enough. Code that panics, divides, or sorts on a negative duration will eventually receive one.
Know when your monotonic reading is real. In Go, any time.Time that crossed a serialization boundary, a Round/Truncate, or a timezone conversion is wall-clock-only. For cross-process or cross-machine ordering, monotonic clocks can't help you at all — that's a distributed systems problem, not a clock API problem.
Pick the clock by the question. "What time did this happen?" → wall clock, stored as UTC. "How long did this take?" / "Has this timed out?" → monotonic. If a single value needs to answer both, you need two readings — which is, in the end, exactly what Go decided to store.

References

monotonic-clocktimekeepinggolanglinuxleap-seconds