An agent loop has a brutal cost structure: every turn re-sends the entire conversation. The system prompt, every tool definition, every tool result since turn one — all of it goes back through the model on every single step. Turn 30 of a coding agent isn't billed for turn 30's new tokens; it's billed for the whole transcript, again. Prompt caching is what makes this economically survivable — reads of cached tokens cost 0.1x the normal input price on both Anthropic's and OpenAI's current models.
But the cache is not a key-value store of "things the model has seen." It is a hash of an exact, byte-identical prefix, and both vendors' implementations have sharp edges that agent builders hit constantly: a one-word edit to a tool description rebills your entire context, a timestamp in the wrong block means you never get a hit at all, and on Anthropic's API a long-running conversation can silently walk out of the cache's lookback window.
What a cache entry actually is
On the serving side, the thing being cached is not your text. OpenAI's docs state it plainly: the cache stores the key-value (KV) attention tensors computed from the prefix, not the tokens themselves. A cache hit means the server skips the prefill computation for that prefix and resumes from the stored KV state. That is why the matching rule is so strict — the KV tensors are a function of every token in order, so there is no such thing as a "close enough" hit.
Anthropic's documentation spells out the matching rule: cache hits require 100 percent identical prompt segments, including all text and images, up to and including the block marked with cache_control. Each breakpoint you place writes one cache entry — a cumulative hash of the prefix ending at that block. You get up to 4 breakpoints, and the breakpoints themselves cost nothing.
{
"system": [
{
"type": "text",
"text": "...4,000 tokens of system prompt and tool-use guidance...",
"cache_control": {"type": "ephemeral"}
}
]
}OpenAI historically did all of this implicitly — caching is enabled by default for supported models, with a 1,024-token minimum — but its newest models now also support explicit prompt_cache_breakpoint markers, likewise capped at four cache writes per request. The two APIs have converged on the same design and nearly the same price card: cache reads at 0.1x input price, cache writes at 1.25x for the short-lived tier on both, and a premium long-lived tier (2x for Anthropic's 1-hour TTL, 2.5x for OpenAI's 12-hour TTL).
That pricing asymmetry contains the core bet: a cache write costs 25% extra, so it pays for itself the moment the prefix is reused even once. An agent loop reuses its prefix on every turn, which is why agent-shaped workloads are the best case for caching and also the most expensive place to get it wrong.
The invalidation cascade: tools, then system, then messages
Anthropic's docs describe the prefix as being constructed in a fixed order: tools, then system, then messages. Your cache entries are hashes over that concatenation — which means invalidation cascades forward. Change something in the tools section and every entry downstream of it is dead, no matter how carefully you structured your system prompt and conversation breakpoints.
The documented invalidation table is worth internalizing:
| What changed | Tools cache | System cache | Messages cache |
|---|---|---|---|
| Tool definitions (names, descriptions, parameters) | ✘ broken | ✘ broken | ✘ broken |
tool_choice | ✓ survives | ✘ broken | ✘ broken |
| Adding/removing images anywhere | ✓ survives | ✘ broken | ✘ broken |
| Toggling web search | ✓ survives | ✘ broken | ✘ broken |
The first row is the one that bites agent frameworks. Tool definitions sit at the very front of the prefix, so they are the single most destructive thing you can touch. A framework that "helpfully" injects dynamic content into a tool description — a current date, a feature flag, a per-session ID — pays full input price for the whole context on every request while reporting no errors whatsoever. Caching fails silent: Anthropic's API just returns cache_creation_input_tokens and cache_read_input_tokens both at 0, and if you aren't watching those usage fields you will never know.
Note what the second and third rows imply, too: tool_choice and images live between the tools and system segments in hash terms. You can flip tool_choice per-request and keep your (re-usable) tools cache, but your system prompt and conversation re-bill. OpenAI's version of the same constraint is phrased as a single rule — cache reuse requires the entire rendered prefix to match, and the rendered prefix includes tools, parallel_tool_calls, reasoning.effort, and the output format settings.
The 20-position lookback
Breakpoint reads on Anthropic's API are not a simple "is this exact hash present" check. On each request, the system checks the hash at your breakpoint, and if there's no match it walks backward one content block at a time looking for a previous write — checking at most 20 positions, counting the breakpoint itself as the first. Consecutive tool_use blocks count as one position, and consecutive tool_result blocks likewise.
This is what makes the common agent pattern work: put one breakpoint at the end of the conversation, and each turn the new request's breakpoint sits a few blocks past last turn's write, inside the lookback window. You get a read of the old entry plus an incremental write of the new tail.
But the window is finite. Anthropic's docs call out the failure mode directly: if a growing conversation pushes your breakpoint 20 or more blocks past the last write, the lookback misses it and you pay a full-price re-prefill of the entire transcript. An agent that fires many small single-block messages between breakpoints — subagent results, status updates, injected reminders — burns through 20 positions fast. The documented fix is a second breakpoint placed closer to the previous write, which is why real agent harnesses spend their 4 breakpoints as a moving ladder over the conversation rather than putting them all at the front.
The clock starts before your response finishes
Two timing rules are easy to miss. First, Anthropic's TTL — 5 minutes by default, refreshed at no extra cost every time the entry is read — is measured from the start of the request that writes or reads the entry, not from the end of its response. The docs give the example: if a response streams for 4 minutes, a follow-up reusing that prefix has about 1 minute after the stream completes before the entry expires. Long tool executions between turns eat the TTL the same way.
Second, a cache entry only becomes available after the first response starts. Fire 10 parallel requests sharing a prefix and all 10 miss — each one pays the full write. For fan-out patterns, the documented approach is to wait for the first response before sending the rest; both vendors also expose an explicit pre-warm (Anthropic via a max_tokens: 0 request, OpenAI via prompt_cache_options.prewarm).
OpenAI adds a routing wrinkle Anthropic doesn't document: cached state lives on individual machines, requests are routed by a hash of the prompt's initial tokens, and traffic above roughly 15 requests per minute on one prefix can overflow to other machines that don't have the entry. The prompt_cache_key parameter influences routing but, per the docs, does not pin requests to a machine or guarantee a hit.
What this means in practice
- Freeze the front of the prompt. Tool definitions and system prompt must be byte-identical across turns. Dynamic values (time, user state) belong at the tail of
messages, after your last breakpoint. - Watch the usage fields, not the latency. Both
cache_creation_input_tokensandcache_read_input_tokensat 0 means no caching happened. Alert on cache-read ratio in production; a silent regression here is a 10x input-cost increase. - Budget breakpoints as a ladder. One on tools/system, the rest trailing the conversation so no gap exceeds the 20-position lookback.
- Mind the write/read break-even. At 1.25x writes and 0.1x reads, caching a prefix that's reused even once is already cheaper than not caching. The 2x–2.5x long-TTL tiers only pay off when sessions genuinely go idle longer than the short TTL — for a continuously active agent loop, the free refresh-on-read keeps the 5-minute tier alive indefinitely.
The mental model that survives contact with production: a prompt cache is a hash chain over an append-only log. Appends are nearly free; edits anywhere else rewrite the world.