Prompt caching, and what it doesn't fix

You wired up cache_control, checked the console, saw a cache write — and then every subsequent call still bills full price. Or it worked for a week and quietly stopped. No error, no warning, just a metrics dashboard that never shows a cache read. One developer debugging exactly this: “Despite using the same inputs, every request is billed as a new generation — no cache hits are being recorded.” A GitLab team hit the same wall in production and had to ship a fix for it: “fix: prompt caching not working on production.” Nothing crashed in either case. The requests all succeeded. The cache just never engaged.

Prompt caching, per Anthropic’s current documentation, is prefix-based: it caches your prompt up to a marked breakpoint, in order — tools, then system, then messages. The match is exact, not semantic: “cache hits require 100% identical prompt segments, including all text and images up to and including the block marked with cache control.” The system hashes your prefix and checks for that hash; if a request reorders a tool list, swaps a retrieved document, or changes one character upstream of the breakpoint, the hash changes and the cache silently misses — no error, just a full-price write.

There is also a floor. Each model has a minimum cacheable prompt length — documented in the roughly 1,024–4,096 token range depending on model, with some newer, smaller models as low as 512 — and prompts under that minimum are, per the docs, “processed without caching, and no error is returned.” And the cache itself is not permanent: it defaults to a 5-minute lifetime, refreshed on each hit, with an opt-in 1-hour TTL for lower-frequency traffic. Miss that window and the next call is a fresh write, at a markup, not a read.

Assume it “just works” once enabled.A cache breakpoint is not a promise, it’s a hash lookup. As one write-up on the failure mode puts it: “Every request changes. Which means the cumulative hash changes. Which means the cache key changes. Which means every request becomes a cache miss. No errors occur. Caching is simply bypassed.” (source) Anything dynamic ahead of your breakpoint — timestamps, shuffled results, a per-request system-prompt tweak — quietly disables the cache for that call.

Assume a lower bill means a lower context load.It doesn’t. Caching changes the price of tokens the model reads; it does not change how many tokens it reads, or how reliably it attends to the ones buried in the middle. See what re-reading context costs for the tax caching removes, and why your AI coding bill spikes for what happens when the TTL trap above resets that discount mid-session.

Assume a cache miss means something is broken. Often nothing is — the request was simply too short to meet the per-model minimum, or the prefix was genuinely new. The docs are explicit that this path is silent by design: no error, just a full-price call. Debugging it means reading the usage fields (cache_read_input_tokens, cache_creation_input_tokens), not watching for an exception that will never fire.

For the caching layer itself: put static content first and volatile content last, so the breakpoint sits after everything that doesn’t change between calls. Keep tool definitions and system prompts stable across requests rather than regenerating them per call. Check cache_read_input_tokens and cache_creation_input_tokens on every response instead of assuming the breakpoint is doing its job. If your traffic has gaps longer than five minutes but shorter than an hour, the 1-hour TTL is a real lever, not a micro-optimization.

None of that is Vinculum-specific — it’s just how to use the mechanism correctly, and it’s worth doing regardless of what else you run. What it doesn’t touch is the ceiling underneath: a perfectly-cached prompt is still the full prompt, still inside the window, still subject to the same attention drop-off in its middle. Fixing the bill and fixing the memory are two different problems, and caching only ever solves the first one.

From our own production telemetry: a single session ran 11,232 turns with 99.97% of what it reasoned against served from cache — each turn sent roughly 116 tokens of genuinely new input against about 445,000 tokens of already-computed context. Caching did exactly its job: the re-read tax on that context was near zero. It is also the clearest demonstration of the limit. That context still had to be tracked, still counted against the window on every turn, and the session stayed coherent only because the record of what mattered lived in an addressable store outside the transcript — not because caching kept it fresh. Caching made replaying the record cheap. It never made replaying the record unnecessary.

Get the caching mechanics right — stable prefixes, monitored usage fields, the right TTL for your traffic — and re-sending the same context stops being expensive. It is still context: still inside the window, still degrading the same way a long prompt always degrades, still gone the moment a fresh session starts with no transcript to cache in the first place. A substrateaddresses the part caching can’t: decisions and findings live as typed, addressable records outside the transcript, so the next turn — or the next session — reads a handful of relevant entries instead of replaying a growing prompt, cached or not.

That is a narrower claim than it might sound. Caching is not a mistake, and nothing here is telling you to turn it off — it is correctly solving the problem it was built for. It was just never the problem of a session outgrowing its own memory.