Pricing a cached prefix

A token counter reports quantity. A cost model reports quantity multiplied by a unit price, and in a prompt with a cached prefix there are two unit prices differing by roughly a factor of six. Proposals to shrink a prompt are almost always written in the counter's currency, which is why a 6,000-token block can look like the largest item in the request and turn out to be one of the smallest.

Reading the count in the right currency is a five-minute calculation from two numbers you already have: the list rate and the cache hit rate on the prefix. The worked figures below come from a support agent on Claude Sonnet 5 at $3.00 per million input tokens and $15.00 per million output, written $3.00 per MTok in the pricing tables, handling 40,000 tickets a month.

Pricing the same 6,000 tokens twice#

Prompt caching is the provider keeping the internal state it computed for the opening stretch of a prompt, so that a later request beginning with exactly the same bytes can reuse that state instead of running the prefill pass over those tokens again — prefill being the pass in which the model reads the whole input before it can emit its first output token. You mark where the reusable stretch ends with a cache_control breakpoint; a request whose prompt matches up to that point is a hit, and one that does not is a miss and writes a fresh entry. This agent's system prompt is a stable prefix with cache_control on its last block, and caching reports a 94% hit rate against it. Cache reads bill at 0.1× the base input rate — $0.30 per million — and cache writes at 1.25× for the default five-minute TTL, the window in which a stored entry can still be matched before it is discarded, so $3.75 per million.

A token in that prefix therefore does not cost $3.00 per million. It costs 0.94 × $0.30 + 0.06 × $3.75 = $0.507 per million, which is 17% of the list rate. Converted, 6,000 prefix tokens are worth about 1,010 uncached ones.

Every part of that line is doing something, and it is worth saying what before using the answer. A token in the prefix is billed at one of two prices depending on an event that has not happened yet — whether the request it arrives in matches a live cache entry — so its price is an expected value: each possible price multiplied by how often it applies, added together. Ninety-four requests in every hundred hit, and on those the prefix bills at the read rate of $0.30 per million. The other six miss, and on those it bills at the write rate of $3.75. Weighting each rate by its share and adding is the operation the line performs, and the reason it is the right operation is that both branches are charged on the same quantity of tokens. If the two branches billed different numbers of tokens, the average would have to be taken over cost per request rather than over price per token.

Converting into uncached tokens is then a division that turns the result into a unit anybody can compare. If a prefix token costs 17% of what an ordinary input token costs, then a prefix token buys about a sixth of an ordinary token's worth of budget, and 6,000 of them are worth about 1,010 of the ordinary kind. That conversion is the whole trick of this chapter. Both sides of any prompt-shrinking proposal are quoted in tokens, and tokens on the two sides are not the same unit unless somebody has checked that they are billed the same way.

Here is a whole request, both ways: as it runs today, and as a proposal to move 6,000 tokens of product and policy notes into a retrieval index would run it. The proposed arm retrieves an average of 800 tokens of policy notes per ticket and places them correctly — after the cached prefix, so the prefix survives.

The saving is $0.000642 per ticket, or 3.6%. At 40,000 tickets a month it is $25.68. The retrieval index for the policy library, the embedding pipeline that keeps it current, the chunking decisions, the on-call surface of a new dependency in the request path, and the engineering week to build it are all being bought for about three hundred dollars a year of saved inference.

Reproducing one row makes the rest of the table readable rather than authoritative. The system prefix is 9,400 tokens today, and on the 94 requests in every hundred that hit the cache those tokens bill at $0.30 per million. Per request that is 9,400 multiplied by 0.94, to get the expected billed quantity, then multiplied by $0.30 and divided by a million because the rate is quoted per million tokens — $0.002651. The write row is the identical calculation with 0.06 in place of 0.94 and $3.75 in place of $0.30, giving $0.002115. The proposed column repeats both with 3,400 in place of 9,400, which is why those two cells shrink in exactly the ratio the prefix shrank by and in no other way. The three rows underneath are simpler, because nothing about them is cached: tokens multiplied by rate, divided by a million.

Adding the cells down each column gives the two totals, and subtracting one from the other gives $0.000642. Dividing that by the $0.017816 a request costs today is where 3.6% comes from, and which denominator that is matters. The saving is a share of the whole request rather than of the prefix, because the whole request is what gets paid for, and a proposal quoting its saving against the block it is removing will report a number several times larger for the same change.

The reason the arithmetic comes out this way is worth stating directly rather than leaving as an inference from the table. The 6,000 tokens that were removed were being charged $0.51 per million. The 800 tokens that replaced them are charged $3.00 per million, because a passage that differs per ticket cannot be part of a shared prefix. Seven and a half times fewer tokens at six times the unit price is a small win, and it is small for a reason that will not change no matter how well the retrieval is engineered.

Both of those ratios come straight out of the figures above. Six thousand tokens replaced by 800 is a reduction of seven and a half times, which is the number the proposal was written around and is genuinely large. The unit price moves the other way by nearly as much: $3.00 against the $0.507 the prefix tokens were actually costing is a factor of about six. Cost is quantity multiplied by price, so the two factors partly cancel and what survives is their ratio — a saving of something like a quarter on the material being moved, not a saving of seven-eighths. That is what "small for a structural reason" means here. Better chunking, a better embedding model or a tighter retriever all act on the quantity; none of them touches the fact that the tokens arriving are per-request tokens and the tokens leaving were prefix tokens.

When the same move raises the bill#

Change one detail — where the retrieved passage is placed — and the proposal stops being a small win and becomes a loss. Caching is a prefix match: the cache key is derived from the exact bytes of the rendered prompt up to each breakpoint, and the render order is tools, then system, then messages. Anything unique that sits early invalidates everything behind it.

A team on a neighbouring service made exactly this change. Their prompt was an 8,000-token block of product documentation with a 900-token instruction block and a conversation history, cached at a 90% hit rate. They replaced the documentation block with a 600-token retrieved passage and, reasoning that the evidence should be the first thing the model reads, put it at the top of the system array, ahead of the instructions. The documentation block shrank by a factor of thirteen. Input tokens per request fell 59%. The bill went up 89%.

The write row is the one that catches people. The cache_control marker was still on the system block, so every request dutifully wrote a fresh cache entry covering the passage and the instructions behind it — an entry that no subsequent request could ever match, because the passage was different. They paid the 1.25× write premium on every request and collected zero reads. Meanwhile the conversation history, which had been sitting inside the cached prefix at $0.30 per million, moved to full price because everything after a changed byte is uncacheable.

Tracing where each token in that table went is what turns a surprising result into an obvious one. Before the change, the 12,200-token cached prefix was the 8,000-token documentation block, the 900-token instruction block, and the conversation history sitting behind them; 250 tokens of genuinely per-request material sat outside the prefix, which is why input per request is 12,450. After the change, the prefix is gone, and its contents redistribute. The 600-token passage and the 900 tokens of instructions behind it are the 1,500 tokens written to the cache on every request — written because the marker is still there, never read because the passage differs each time. The conversation history joins the same 250 tokens at list price, and 3,550 is what those come to. Add the written 1,500 to the full-price 3,550 and you have the 5,050 in the last row.

That decomposition is why input fell 59% and the bill rose 89% at the same time, and the two movements are not in tension once the tokens are priced rather than counted. The tokens that left were the cheapest ones in the request: 8,000 tokens of documentation billing at roughly a tenth of list on nine requests in ten. The tokens that stayed got more expensive — the history moved from the read rate to the full rate, which is a tenfold increase on the largest surviving block — and the tokens that arrived are worse than uncached, because a written-and-never-read token bills at 1.25 times list rather than at list. Every individual charge in the after column moved in the same direction. Only the token count moved the other way.

The stumble here is not an arithmetic slip and it is worth stating as a belief somebody would hold confidently: fewer input tokens means a smaller input bill. That is true whenever every token in the request has the same unit price, which is the situation most people's intuition was formed in and is still the situation for a service with no caching at all. Introduce a cache and the request has two unit prices in it, and the count and the bill come apart — far enough apart that they can move in opposite directions on the same change. The counter is not lying. It is answering a question about quantity that nobody asked.

The diagnostic is one field, and it is worth wiring into a dashboard rather than checking by hand after an incident:

const response = await client.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 1024,
  system: [
    // Stable content first, with the breakpoint on its last block.
    { type: "text", text: SYSTEM_RULES, cache_control: { type: "ephemeral" } },
  ],
  messages: [
    ...history,
    {
      role: "user",
      content: [
        // Volatile content lives AFTER the last breakpoint.
        { type: "text", text: `<policy>\n${retrievedPassage}\n</policy>` },
        { type: "text", text: ticket.body },
      ],
    },
  ],
});

metrics.gauge("prompt.cache_read_tokens",  response.usage.cache_read_input_tokens);
metrics.gauge("prompt.cache_write_tokens", response.usage.cache_creation_input_tokens);
metrics.gauge("prompt.uncached_tokens",    response.usage.input_tokens);

If cache_read_input_tokens is zero across repeated requests that share a prefix, something ahead of the breakpoint is changing. The usual culprits are a rendered timestamp, a session or user ID interpolated into the system prompt, a JSON blob serialised without sorted keys, or a tool list assembled per user — and each of them is a one-line fix worth far more than a 6,000-token migration.

Those four look unrelated and are one fault. A cache entry is found by matching the exact bytes of the rendered prompt from the start up to a breakpoint, so the lookup key is the whole prefix and not a summary of it. One character different anywhere in that stretch produces a different key, which finds nothing, which is a miss, which writes a fresh entry that the next request will also fail to match for the same reason. Nothing about the size of the change matters: a timestamp that ticks once a second and a system prompt rewritten from scratch are the same event as far as the lookup is concerned. That is also why the ordering rule is strict rather than advisory. Tools render before system, and system before messages, so a tool list assembled per user sits ahead of everything and takes the whole prefix with it however stable the rest is.

The hit rate is the whole argument#

The reason to compute the effective price rather than argue from intuition is that the answer flips, and it flips at a level that is easy to reach.

Every row in that table is the same weighted average computed with a different hit rate, so the table can be rebuilt from two rates and one variable rather than memorised. What makes the last two rows possible is that the two branches sit on opposite sides of the list price: a read costs a tenth of list, so a hit saves nine tenths of it, while a write costs 1.25 times list, so a miss costs a quarter of it extra. Caching is worth having exactly while the savings on the hits outweigh the surcharges on the misses.

Setting those two against each other gives the break-even without any table at all. Nine tenths saved on each hit has to cancel a quarter added on each miss, so one miss is paid for by rather less than a third of a hit, and break-even arrives once hits are a little over a fifth of requests. That is the 22% row, where the effective price equals the list price to the cent. Below it the misses dominate and the prefix is billing above list. This is also why a low hit rate is worse than no cache rather than merely less good than a high one, which is not what "the cache is not helping" sounds like it should mean.

Below a 22% hit rate, a cached prefix costs more per token than not caching at all, because the write premium outweighs everything the reads save. A team looking at a 9,400-token prefix with a 25% hit rate is looking at a genuinely expensive block, and moving 6,000 tokens out of it would save something like 25% of the request rather than 4%. The same proposal, the same tokens, a tenfold difference in value — decided entirely by a number the original document did not contain.

That is what to do first. Before touching the prompt, read the cache counters in production and compute the effective rate. If the hit rate is high, the block is cheap and the proposal should be judged on something other than cost. If the hit rate is low, find out why before you accept it as a fact about the workload, because a low hit rate on a prefix that never changes is a bug rather than a property.

Cheap is not free, and cached is not stable#

Cheap and free are different, and the distinction matters because the arguments from the first two chapters survive caching entirely. A cached token is still in the context window. It still occupies the model's attention, still sits between the instruction and the evidence, and still competes with the retrieved passage for salience. Prompt caching is a billing and prefill-latency mechanism; it does not reach into the forward pass. A 9,400-token prefix that measurably degrades answers on the frozen eval set should be cut whether it costs $0.51 or $3.00 per million, and the cost argument is simply the wrong argument to have about it.

The reason follows from what a cache hit actually reuses. The stored entry is the internal state the prefill pass computed for those tokens, and reusing it gives the model the same state it would have arrived at by computing it again. The request the model answers is therefore identical either way, token for token and state for state — which is the point, because a cache that changed the answer would be a bug. What was skipped is the recomputation, not the reading. A 9,400-token prefix competes with the retrieved passage for the model's attention in precisely the way the earlier chapters measured, and it does so at the same strength whether the state behind it was computed a moment ago or an hour ago. Cheap and absent are different, and only one of them is a fix for dilution.

Caching also buys nothing across a change. Switch models, add or remove a tool, or edit a single character of the system prompt, and the next request writes rather than reads. A prompt that a team edits three times a week does not have a stable prefix in any meaningful sense, whatever the weekly average hit rate says. And cache entries expire — five minutes by default, an hour at double the write premium — so a service with bursty traffic and long idle gaps pays for writes it never amortises. The 94% figure in this chapter is a real number from a service handling a request every few seconds. A nightly batch job with the same prompt would report something closer to 0%.

The rule that survives all of that is narrow and worth carrying: convert before you compare. A token count is not a cost, a percentage saving computed at list rates is not a saving, and the same 6,000 tokens are worth $25 a month or ten times that depending on a hit rate that takes one query to read. The proposal that arrives with tokens on both sides of the comparison and no hit rate in it has not been costed.

There is one more thing the cost table cannot see, and it is why cost is rarely the deciding factor in this decision at all. Both arms in the first table run one model call. The proposed arm assumed the retrieval was a lookup the service performs before assembling the prompt, which is cheap and predictable. The moment retrieval becomes something the model decides to do — a tool it calls when it notices it needs a policy — the request stops being one model call and becomes at least two, and the number that moves is not the bill.