Every turn of an agent loop starts from zero. The model is stateless: to take step 50, your harness sends the system prompt, the tool definitions, and all 49 previous steps — again. The API bills you for all of it — again. Do the arithmetic and agent loops look economically absurd: with a 10,000-token system-plus-tools preamble and ~2,000 tokens of tool call and result per step, step 50's request alone is 110,000 input tokens, and the cumulative input across 50 steps is over 3 million tokens. The cost of a conversation grows quadratically with its length.
The reason agents are viable anyway is prefix caching — and prefix caching has one unforgiving rule: the cached prefix must match exactly, byte for byte, from position zero. Understanding why that rule exists, and what it forces on your harness design, is worth real money.
Why the cache is a prefix cache, not a content cache
What actually gets cached is not text — it's the transformer's KV cache: the key and value tensors computed for every token during prefill. In causal attention, the keys and values for token i depend only on tokens 0 through i. That's the whole trick. It means the KV state for a prefix is a pure function of that prefix, so a server can keep it around and splice it under any future request that starts with the same tokens — skipping the prefill compute entirely.
It also dictates the failure mode. Change one token at position 100, and every KV entry from position 100 onward is computed against a different history and is now wrong. Nothing after the edit is reusable. Prefix caches don't degrade gracefully under edits; they cliff.
You can see this contract enforced in the open-source serving stacks:
- vLLM's automatic prefix caching hashes each fixed-size KV block by its own token IDs plus the hash of its parent block — the docs describe it as hashing "the tokens in the block and the tokens in the prefix before the block" (with LoRA IDs and multimodal hashes mixed in). The parent-hash chaining is the exact-prefix rule made concrete: an identical block of tokens at a different position, or after a different history, hashes differently and misses. Only full blocks are cached, eviction is LRU, and the win is confined to prefill — decode speed doesn't change.
- SGLang's RadixAttention organizes the same idea as a radix tree mapping token sequences to KV tensors, stored in a paged layout where "the size of each page is equivalent to one token." Eviction is LRU on leaves first, so shared ancestors — the common prefixes — survive longest. SGLang also schedules cache-aware: it sorts waiting requests by matched prefix length instead of first-come-first-served, which the paper measures at 96% of the optimal hit rate on average, with hit rates from 50% to 99% across benchmarks. KV reuse is a major part of why the paper reports up to 6.4× higher throughput than baseline systems.
What the API vendors put on top
Hosted APIs wrap this same mechanism in billing and (in one case) explicit control.
Anthropic exposes explicit cache_control breakpoints — up to 4 per request — and defines the prefix order as tools → system → messages. A change at any level invalidates that level and everything after it: touch a tool definition and your system and message caches are gone too. Writes cost 1.25× the base input rate (2× for the 1-hour TTL variant), reads cost 0.1×. The default TTL is 5 minutes, refreshed at no extra cost on each hit. Two sharp edges from the docs: the TTL clock starts at the start of the request that writes or reads the entry (generation time counts against it), and "a cache entry only becomes available after the first response begins" — so parallel fan-out requests sharing a prefix all miss unless you let the first one start responding before sending the rest. There's also a minimum cacheable prefix — 512 to 4,096 tokens depending on model — below which cache_control is silently ignored.
OpenAI made caching automatic: a breakpoint is placed "at the end of the latest eligible message," with an explicit-breakpoint mode available. The minimum is 1,024 tokens on GPT-5.6 and later; older models round cached-token counts "down to the nearest multiple of 128." Reads are 0.1× the uncached rate and writes on GPT-5.6+ are 1.25×, which gives the docs a tidy break-even fact: writing a prefix once and reusing it once in full costs 1.35× its ordinary input price — cache-positive on the second hit. Entries live 30 minutes after the most recent write or reuse on GPT-5.6+ (roughly 5–10 minutes, up to an hour, on earlier models), and a prompt_cache_key parameter partitions cache accounting per end user. The requirement is stated flatly: "Cache reuse requires the entire rendered prefix to match."
Same 90% read discount, same exact-prefix contract, on both.
The rules this forces on agent harnesses
An agent loop is the ideal caching customer if it behaves: each request is the previous request plus a small suffix, so every turn should re-read the whole history at 0.1× and pay full price only for the delta. Rerun the 50-step example with caching and the ~3M cumulative input tokens bill at roughly the price of ~370k uncached ones. That only happens if nothing upstream ever changes. In practice:
- Be append-only. Never rewrite, reorder, or delete earlier turns mid-conversation. Summarizing or compacting history is sometimes necessary — but recognize it as a deliberate cache-flush you pay full prefill for, not a free tidy-up.
- Nothing volatile before the breakpoint. A timestamp in the system prompt, a request ID, a "current date" header — one changed byte at the front invalidates everything behind it, every single request.
- Freeze the tool list. On Anthropic's API the tool definitions are the first segment of the prefix; a tool added, removed, or reserialized in a different order invalidates the entire cache. Dynamic tool selection per turn is a cache-destruction machine.
- Watch the counters, not the vibes. Both APIs report it:
cache_read_input_tokens(Anthropic) andusage.input_tokens_details.cached_tokens(OpenAI). If that number is ~0 on turn after turn of a long agent run, you have a silent invalidator — the API returns no error for a missed cache, it just bills you 10× more. - Mind the TTL against your loop's tempo. A tool call that takes six minutes of wall-clock — a long build, a human approval — outlives Anthropic's default 5-minute entry. That's what the 1-hour TTL (at 2× write cost) and
max_tokens: 0pre-warm requests are for.
The quiet architectural conclusion
Prefix caching started as an infrastructure optimization, but the incentives now flow backward into agent design. The 10× price gap between cached and uncached tokens is large enough that "does this preserve the prefix?" has become a first-order design question — arguably the design question for a harness. Append-only transcripts, frozen tool rosters, volatile data pushed to the end of the prompt, edits treated as expensive events: none of these are model-quality decisions. They're cache-hit-rate decisions, and they routinely dominate an agent's unit economics before anyone tunes a single prompt.