Prompt caching and prefix stability

A system prompt is re-sent verbatim on every call, which makes it the largest block of repeated work a model API is ever handed. Providers offer to keep the work they already did on such a block and reuse it on the next request that starts with the same bytes, billing the reused span at a fraction of the normal input rate instead of processing it again. That discount is prompt caching, and on an agent that makes a dozen calls per task against a six-thousand-token prompt it is the difference between a five-figure monthly bill and a six-figure one.

The discount has one condition, and it is stricter than it looks. What the provider matches on is a prefix: it renders your request in a fixed order — tool definitions first, then the system prompt, then the messages array — hashes that byte stream, and looks for a previously seen request whose stream is identical from the very first token up to some marked point. This chapter is about what that condition costs when it is broken by something that looks harmless, how to price the loss rather than assert it, and the cases where a prefix that varies per user is nonetheless the right design.

Where the cache key comes from#

The match starts at token zero and stops at the first byte that differs, and everything behind that byte is uncacheable.

That is why an opening line is the most expensive real estate in the request. You are speaking with Dana. The current time is 2026-04-14T09:17:03Z. differs from every other rendering of the same template, for every user and every second, so the walk stops at token three and the six thousand tokens of policy behind it — which have not changed in a month — are processed as if the provider had never seen them.

What the provider keeps is worth being precise about, because the word cache invites the wrong picture. What it stores is not your text; it already has your text, you just sent it. What it stores is the intermediate numbers the model computed while reading that text, position by position, during the pass that has to finish before any output token can be produced. Those numbers are what the rest of the model consumes. Producing them is nearly all of the work billed as input, and skipping their recomputation is the entire saving.

The reason the rule is a prefix match rather than something more forgiving comes straight from how a transformer processes a request. The internal state computed for token n depends on every token before it, so the work done on your policy text is only reusable if the text in front of it was identical; change one token at position three and the state for tokens four onward is a different set of numbers that happens to be derived from mostly the same words. There is no partial credit and no set-membership check, because the thing being cached is not your text but the computation over it. Read that way, "the cache matches from the first token" stops being an arbitrary API constraint and becomes the only thing it could have been.

Two things follow from that dependence and both are worth spelling out. The first is why reuse is legitimate rather than merely cheap: the computation is a fixed function of the tokens, so identical tokens in identical positions produce identical numbers every time, and a stored result from an earlier request is not an approximation of what this one needs. It is the same quantity, which is what makes serving it correct rather than a shortcut with a quality cost attached. The second is why the loss runs to the end of the block rather than stopping where the difference stopped. Position five was computed from position four's state, position six from position five's, and one changed token at position three therefore spoils every position after it by transitivity. No provider policy is choosing that.

The picture that gets this wrong is the ordinary one, which is why it is worth saying out loud: nearly everything else you cache behaves the other way. A key-value cache hashes the whole of a key and either finds an entry or does not, so a miss costs one lookup, the other entries are untouched, and the price of a change scales with the size of what changed. Neither property holds here. There is no whole-key hash to miss, because the match is a walk from the front that stops where the two streams diverge. And the price of a change scales with how many tokens sit behind it rather than with how many bytes moved, so twenty volatile tokens at the front cost the same as rewriting the entire policy document, and far more than the same twenty tokens would cost at the end of the block.

The same mechanism means a timestamp is not the only way to lose the prefix, and the other ways are less visible. A tool list assembled from a dictionary iteration order, or built per user so that one customer's request carries an extra tool, changes bytes at position zero — tools render before the system prompt, so a varying tool set invalidates everything. A JSON blob serialised without sorted keys can come out differently between processes. A system prompt assembled with conditional sections gives you one distinct prefix per combination of flags, which is fine at four combinations and useless at four hundred. Each of these produces the same zero in cache_read_input_tokens and none of them looks like a caching bug in review; they look like ordinary code.

The confirming signal is in the response, and it takes one line of logging to watch:

resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=1024,
    system=[{"type": "text", "text": SYSTEM, "cache_control": {"type": "ephemeral"}}],
    messages=messages,
)
u = resp.usage
print(u.cache_read_input_tokens, u.cache_creation_input_tokens, u.input_tokens)
# healthy:   5964  0     412
# broken:    0     5984  412

A cache_read_input_tokens of zero across repeated calls that ought to share a prefix means something in front is varying, and the first place to look is the top of the system prompt. This failure is silent in both directions: nothing errors, no header warns you, and a cache breakpoint — the cache_control marker in the request that says how far the cacheable span reaches — placed on a prefix shorter than the model's minimum (512 tokens on Claude Opus 5, 1024 on several others) simply never caches at all and reports zeros forever.

Those zeros therefore have two causes that look identical in a log and want opposite repairs. Either the span in front is varying between calls, in which case the fix is to move whatever varies, or the span is stable and too short to qualify, in which case moving things around changes nothing and the fix is to give the breakpoint more to hold, or to accept that this prompt is not worth caching. One question settles it without involving the provider: hash the rendered prefix on two consecutive calls and compare. Matching hashes alongside zero reads is a length problem, and differing hashes are the problem the rest of this chapter is about.

Pricing a volatile first line#

"It is only twenty tokens, so it is free" is the answer to give if you want the change shipped, and the way to beat it is arithmetic rather than assertion. Take an agent with a 6,000-token system prompt, twelve model calls per task on average, and roughly 200,000 tasks a day. That is 2.4 million calls a day, and the prefix alone accounts for 2.4M × 6,000 = 14.4 billion tokens per day — before a single token of task content.

Each step of that line is allowed for a reason worth stating, since a total this large is easy to wave away as a trick of multiplication. Calls per task times tasks per day gives calls per day because both figures describe the same day, and an average is legitimate for a daily total even though it conceals the tail. Multiplying the call count by the length of the prefix is the step that looks like double counting and is not: the API keeps nothing between calls, so every one of those calls carries the whole system prompt again, and the prefix is therefore charged once per call rather than once per task or once per deploy. The multiplication itself is six thousand by two point four million, which is six times two point four in the leading digits, giving fourteen point four, and a thousand times a million in the magnitudes, giving a billion. Hence fourteen point four billion tokens a day, all of it text that has not changed in a month.

Every one of those tokens is either read from cache or processed fresh, and the gap between those two prices is what a volatile opening line spends. At Claude Opus 5's published rate of $5.00 per million input tokens — one MTok, which is the unit the table below is priced in — with cache reads billed at roughly a tenth of that:

Every figure in that table is one volume priced three ways, and each rate is the base rate with a single multiplier on it. A billion is a thousand millions, so fourteen point four billion tokens is fourteen point four thousand of the units the rate is quoted in, and the uncached row is that count times five dollars: seventy-two thousand dollars a day. The cached row divides the rate by ten and leaves the volume alone, which is why it is the same figure with the decimal point moved — seven thousand two hundred. The write rate is the base rate plus a quarter of itself, six dollars and twenty-five cents, and the same count against that gives ninety thousand. Subtracting the cached row from the uncached one, seventy-two thousand less seven thousand two hundred is sixty-four thousand eight hundred, which is where the figure quoted below as roughly sixty-five thousand a day comes from.

The middle row is the honest floor for the damage and the third row is the reason an observed jump can be worse than 10×: a request that misses the cache and still carries a cache_control breakpoint pays the write premium as well, so you are billed above the plain uncached rate for the privilege of populating an entry that nothing will ever read. The loss is the cached discount on 14.4 billion tokens a day, which is around $65,000 a day at list rates, and it is not twenty tokens.

A jump somebody reports off an invoice will not land exactly on the ratios in that table, and reconciling the two means allowing for two forces that pull in opposite directions. The write premium pulls the observed multiple up, since a call that misses while still carrying a breakpoint is billed above the plain uncached rate. The task content pulls it down, because the table prices the prefix on its own and an invoice does not: every call also carries its turn's own tokens, which were never cacheable and cost the same before and after the change. An observed multiple therefore lands near the ratio rather than on it, and one that comes in slightly under ten is not evidence that something other than the cache was responsible.

Three assumptions hold that estimate up, and you should replace each with your own number before quoting it in a design review. The rate is a published list price, and an organisation running 2.4 million calls a day is probably not paying list. The hit rate in the first row assumes traffic arrives often enough to keep entries alive within their time-to-live, or TTL, the window after a write during which the provider will still serve an entry before discarding it, which is true for continuous load and false for a workload that runs in bursts every twenty minutes. And the twelve-steps-per-task average hides a long tail, since one task that runs ninety steps contributes ninety prefixes. What survives all three substitutions is the ratio: the loss is the gap between the cached and uncached rate, multiplied by prefix length, steps per task, and tasks per day. Change any input you like and the shape of the answer does not move.

The hit rate is the assumption most worth turning into arithmetic, since it is the one people quote as a success. A miss costs ten times what a hit costs, so at ninety-nine percent the single call in a hundred that misses is priced like ten of its neighbours, and it adds about a tenth again to that row's daily total. A realistic cached bill therefore sits above the first row rather than on it, by an amount that does not disturb the comparison — a tenth added to seven thousand two hundred is still nowhere near seventy-two thousand. Run the same sum at a hit rate several points lower and it degrades quickly, which is the argument for watching the ratio itself rather than watching whether caching is switched on.

The burst case is worth working out, because it produces a bill that behaves like a broken design in a service where nothing is broken. The window after a write is five minutes wide by default, and a workload that wakes every twenty minutes arrives a quarter of an hour after its entries have already expired. So the entry written by the first call of one burst is gone before the first call of the next burst asks for it, and that call pays a write rather than a read, every time, forever. Within a burst the arithmetic works normally and between bursts it never gets to start, which is how a hit rate averaged over a day can look respectable while every burst begins at full price.

Latency moves with it. A cache read skips the prefill work for the matched span — prefill being the pass over the input tokens that has to finish before the first output token can be produced — so on a six-thousand-token prefix the time to first token typically drops by a few hundred milliseconds; across twelve sequential steps that is a couple of seconds per task, which users experience as the agent feeling sluggish rather than as a billing event. Both effects arrive together and neither shows up in any test that sends one request.

Moving the values, and what that costs#

The fix is placement. The instruction "address the user by their first name" is byte-identical on every call and stays up top; the name itself and the timestamp go into the user turn, behind the cache breakpoint.

SYSTEM = """You are Meridian's operations assistant.
Address the user by the first name given in <session>. Use the current
time from <session> for anything date-dependent; never assume today's date.
… 6,000 tokens of stable policy …"""

def build_turn(first_name: str, now: datetime, task: str) -> str:
    return (
        f"<session>\nfirst_name: {first_name}\ncurrent_time: {now.isoformat()}\n</session>\n\n"
        f"<task>\n{task}\n</task>"
    )

Two separable things make that split work. The values move behind the cache breakpoint, the marker that says how far the reusable span reaches, so everything after it is expected to differ from call to call and nothing is invalidated by its differing. The instruction stays in front of it because a sentence about how to behave is the same bytes on every call the service will ever make. What the instruction has to do now, which it did not have to do while the value was interpolated into it, is name the place the value will arrive — hence the line telling the model to take the current time from <session> and never to assume today's date. A standing instruction that refers to a per-request value has to point at where that value lives, or the model supplies one from whatever its training made plausible, and a confidently wrong date is harder to notice than a missing one.

What you lose by moving them is small and worth naming rather than waving away. The values now arrive in the user channel, so they carry request authority instead of operator authority — which matters if the name comes from somewhere a user controls, and matters not at all for a first name read out of your own account table. You also re-send about twenty tokens per call that a cached prefix would have served at a tenth of the price, which at 2.4 million calls a day is on the order of $0.20. And the instruction is now split across two places, so a reviewer has to read both halves to see the whole behaviour, which is a real maintenance cost paid in exchange for roughly $65,000 a day.

When a per-user prefix still pays#

The two interpolated values are not equivalent, and treating them as one thing gets the design wrong in the other direction. A timestamp rendered to the second is unique per request by construction: no second call will ever match it, so its prefix can never be read, and there is no traffic pattern that rescues it. A first name is unique per user, which means the prefix is shared by every call that user makes while the entry is alive.

Both are readings of one quantity: how many calls share each distinct prefix, which is your traffic divided by the number of distinct renderings it produces. For the timestamp that quantity is one, which is the whole of why nothing rescues it. For the name it is however many calls that user makes inside the window, which is a figure you can look up rather than guess. Volatility as such decides nothing; it matters only through what it does to that count, and that is why two values which both look like personalisation price out completely differently.

That turns the question into arithmetic about reuse within the cache's time-to-live. With the default five-minute TTL, a write costs about 1.25× the base input rate and a read about 0.1×, so a prefix that is written once and read once costs 1.35× against 2× for two uncached calls, and the break-even sits at two calls inside the window. The agent above makes twelve calls per task, back to back, so a per-user prefix is written on step one and read on steps two through twelve — eleven reads per write, which recovers most of the benefit even though nothing is shared between users. A per-user prefix pays when a single user's calls cluster; it fails for the long tail of users who make one call and leave, and each of those pays the write premium for an entry nobody reads.

Those two multipliers price any reuse pattern, and the sums are short enough to do once and then carry around. Written once and read once is one and a quarter plus a tenth, which is where the one and thirty-five hundredths comes from, and it beats two calls at full price because adding a tenth is a far smaller thing than adding a whole call. Written once and read eleven times is one and a quarter plus eleven tenths, a little over two and a third in total, against the twelve full-price calls it replaces — roughly a fifth of the cost, from a prefix no other user ever touches. Written once and never read is one and a quarter against one, which is the quarter extra the long tail pays, and it is the only one of the three shapes where caching costs more than not caching.

So there are cases where a genuinely per-user system prompt earns its place. A tenant with its own policy text, glossary, or tool descriptions, running thousands of calls a day, gets a prefix that is stable for that tenant and reused heavily within it — the byte-identical test that sorts standing behaviour from per-request data passes at tenant scope even though it fails at fleet scope. A long agent session that loads a user's persistent memory file into the prefix is the same shape: written once, read for the rest of the session. What does not work is a prefix keyed to something with more cardinality than your traffic can amortise, and a wall-clock timestamp is the extreme case of that, with one cache entry per second per user and no reads at all.

Cache economics are therefore a question about position and about reuse, and neither is visible in the diff that breaks them. Render volatile values at the point where they stop being reused, which for anything below tenant granularity means the user turn. If you want the check to be mechanical, hash your rendered prefix per request and compare the count of distinct hashes against the count of requests: one hash per deploy is healthy, one hash per tenant is a deliberate trade with arithmetic behind it, and one hash per request is a bill.

Everything so far has been about keeping the top of the request stable and keeping it small. There is a third property it needs, and it is the one that a retrieval pipeline will quietly break, because a retrieved passage is per-request — so it fails the caching test too — and teams put it up top anyway on the grounds that it is context. The next chapter is about what else arrives with that passage.