Every retry policy is a bet that the failure you just saw was bad luck. When that bet is wrong — when the service on the other end is failing because it's overloaded — retries stop being resilience and become the second wave of the attack. The uncomfortable part is that the standard fix, exponential backoff, doesn't actually solve this. It defers load; it doesn't shed it. Getting that distinction right is the difference between a blip and a cascading failure.
The multiplication nobody budgeted for
Retries compound across layers. Google's SRE book works through the arithmetic in its chapter on cascading failures: suppose a database starts returning errors, and each layer above it — backend, frontend, client — retries a failed call up to three times, so four attempts per layer. The book's warning is blunt: "a single user action may create 64 attempts (4^3) on the database. This behavior is undesirable when the database is returning those errors because it's overloaded."
One click becomes 64 queries, delivered at exactly the moment the database can least afford them. The load amplification is invisible in any single service's code review, because each layer's policy looks reasonable in isolation. That's why the SRE book's core recommendation is structural, not numeric: "avoid amplifying retries by issuing retries at multiple levels." Pick one layer of the stack — usually the one closest to the failure, where the error is best understood — and let every other layer fail fast and propagate the error upward.
Backoff alone leaves the herd intact
Even at a single layer, naive retries synchronize clients. The canonical demonstration is Marc Brooker's 2015 post on the AWS Architecture Blog, which simulates optimistic concurrency control: many clients trying to update the same database row over a network with 10ms mean delay and 4ms variance. Since one client wins each round, completing all updates takes N rounds — and the total work grows with N², because N clients compete in round one, N−1 in round two, and so on.
The classic remedy is capped exponential backoff:
sleep = min(cap, base * 2 ** attempt)In Brooker's simulation, this helps less than you'd expect. His diagnosis: "there are still clusters of calls. Instead of reducing the number of clients competing in every round, we've just introduced times when no client is competing." All clients that failed together back off by the same amount and come back together. The herd is slower, but it's still a herd.
Jitter is the actual fix — and Full Jitter wins
The counter-intuitive move is to add randomness. "Full Jitter" replaces the deterministic sleep with a uniform draw over the whole backoff window:
sleep = random_between(0, min(cap, base * 2 ** attempt))The post compares two alternatives. "Equal Jitter" keeps half the backoff and randomizes the rest, on the intuition that very short sleeps are wasteful:
temp = min(cap, base * 2 ** attempt)
sleep = temp / 2 + random_between(0, temp / 2)"Decorrelated Jitter" grows the window from the previous sleep instead of the attempt count:
sleep = min(cap, random_between(base, sleep * 3))The simulation's verdict: "Of the jittered approaches, 'Equal Jitter' is the loser. It does slightly more work than 'Full Jitter', and takes much longer." Between Full and Decorrelated the call is closer — Full Jitter "uses less work, but slightly more time" — but both substantially cut client work and server load versus jitterless backoff. That intuition about avoiding short sleeps turns out to be exactly wrong: the short sleeps are what keep the server busy during the gaps the herd would otherwise leave empty.
This isn't just simulation trivia. AWS SDKs' standard retry mode computes delays as random(0, 1) × min(20,000 ms, base_delay × 2^retry) — Full Jitter, verbatim, with a 20-second cap, a 50ms base for transient errors and a 1,000ms base for throttling errors (per the updated retry behavior AWS documents in its SDK reference guide, currently opt-in via AWS_NEW_RETRIES_2026).
Jitter spreads load. It still doesn't shed it.
Here's the misconception worth killing: even perfectly jittered backoff does not fix sustained overload. Brooker returned to this in 2022 with "What is Backoff For?", and the conclusion is stark. With a bounded set of clients in a closed loop — like his OCC simulation, or TCP — a client that backs off genuinely removes work from the system. But most services face an open population of independent clients, and there, "each client backing off in this case does not help, because it does not reduce the work being sent to the system." New requests keep arriving from clients that never saw a failure. Deferred retries stack on top of them. His summary: "The only way to deal with long-term overload is to reduce load, deferring load does not work."
Backoff and jitter smooth spikes and buy the server breathing room measured in seconds. If the overload lasts minutes, something has to decide to not send requests. That mechanism is a retry budget.
Token buckets: retries as a paid resource
The SRE book's version is a per-process budget: "only allow 60 retries per minute in a process, and if the retry budget is exceeded, don't retry; just fail the request." That single rule, the book notes, "can contain the retry effect and be the difference between a capacity planning failure that leads to some dropped queries and a global cascading failure."
AWS's SDKs implement the same idea as a token bucket, and the tuning is instructive. In standard mode each client starts with a budget of 500 tokens. A retry after a transient error (a 500, a connection reset) costs 14 tokens; a retry after a throttling error costs only 5. A retry that succeeds refunds its own cost, and every clean first-try success trickles 1 token back. When the bucket is empty, the SDK returns the error immediately — no retry, no delay.
Two design choices stand out:
- Transient retries cost nearly 3× more than throttling retries. A throttling response is the service saying "slow down, then try again" — a retry is likely to work. A wave of 500s and timeouts often means the service is down for everyone, and retrying mostly adds load and latency.
- The bucket only bites under sustained failure. With the default of 3 max attempts, the budget starts draining when roughly 22% of requests hit sustained transient failures (about 32% for throttling). Below that, successes refill it faster than failures drain it, so normal operation never notices the mechanism exists.
That's the property jitter can't give you: during a real outage, the client's retry traffic goes to approximately zero on its own, and refills gradually as the service recovers. AWS's adaptive mode goes one step further with a client-side rate limiter that can delay even initial requests when throttling is detected — with the documented caveat that the limiter is per client instance, so one throttled resource slows every request that client sends.
The policy that survives an outage
Assembled from the sources above, a retry policy that helps during failures instead of piling on:
- Retry at one layer of the stack. Everywhere else, fail fast and propagate.
- Cap attempts per request. AWS SDKs default to 3 total attempts; the SRE book warns against retrying "a given request indefinitely."
- Always use Full Jitter.
random_between(0, min(cap, base * 2^attempt))— and per the SRE book, "always use randomized exponential backoff when scheduling retries," because even a small network blip can re-synchronize unjittered clients. - Give retries a budget. A token bucket or a per-process cap turns "retry forever, slower" into "stop retrying when it clearly isn't working" — the only behavior that actually reduces load during sustained overload.
- Let the server overrule you. When a response carries explicit retry timing (AWS services use an
x-amz-retry-afterheader with a delay in milliseconds), prefer it — the service knows when it expects capacity back.
Backoff and jitter are the polite part of a retry policy. The budget is the part that keeps your clients from becoming the outage.