The cache write premium

Under the explicit-breakpoint regime, the first request to fill a cache entry does not pay the ordinary input rate for the tokens it caches. It pays a write premium — a multiplier above base input, 1.25× on the pricing shape used throughout this topic — in exchange for the provider retaining the computed state, and later requests that match the entry read it at around 0.1×. Both numbers are per token of the prefix, which means caching is a trade: a known extra cost now against an unknown number of discounted reads later.

That framing decides which endpoints in a fleet are worth the work, and it routinely reverses the ranking people arrive with. The instinct is to sort candidates by prefix size, because a large prefix is where the tokens visibly are. Size is one of the two terms in the product and usually the less variable one.

A cache write is the most expensive way to send a token#

The premium buys work that an ordinary request does not do, and it is worth seeing what that work is before pricing it. Every request prefills: the prompt is read through the model to produce, for each token at each layer, the key and value vectors attention needs, and that computation is most of what the ordinary input rate pays for. On an ordinary request the result is thrown away when the response ends, and the memory holding it goes straight back to serving the next caller. A write does the same prefill and then does not throw the result away. The state has to be copied out of the working memory the accelerator needs for live requests, put somewhere it can outlive the request that made it, indexed under the hash of the prefix, and held there until it expires.

Two of those steps are the expensive ones, and neither of them shrinks. The copy is bandwidth on the busiest memory in the machine. The holding is capacity — the same capacity that decides how many requests the hardware can run at once — reserved for the entry's whole lifetime whether or not anybody ever reads it. So a write is an ordinary send plus a copy plus a reservation, and the multiplier above base input is where those two land on the bill. The read rate is low for the mirror-image reason: a hit skips the prefill entirely, which was the expensive part of the input rate, and leaves only the work of finding the entry and wiring it into the request.

Charging that to the writer rather than spreading it across the readers is a choice, and the reason for it is that nobody knows the reuse count yet. At the moment an entry is created, the request creating it may be the first of two million matching requests or the only one there will ever be, and the provider has no more information about which than you do. Spreading the cost would mean one of two things. Either the provider bills retroactively once the count is known, which a per-request API cannot do — the request that paid has completed and been invoiced. Or the read rate carries an average write cost, which raises the price of every hit in order to subsidise entries that were never read, and the depth of the read discount is the entire product.

So the cost lands on the request that caused it, and that has a consequence worth stating plainly before any arithmetic. An endpoint that writes entries nobody reads pays for them itself, and its bill is worse than if it had never cached at all. Nothing in the mechanism notices this, and nothing corrects it: there is no refund for an unread entry, no warning that one was never touched, and no field in a response that means "this entry was a waste". That is what turns caching from a default into a decision, and it is why the three conditions at the end of this chapter cost money rather than merely failing to save it.

Start with the smallest version of the question of whether a write pays for itself, in units of the prefix itself. Call one send of the prefix at the base input rate one prefix-equivalent. Sending it uncached n times costs n. Caching it costs 1.25 for the write plus 0.1 for each of the n − 1 later reads.

Working in prefix-equivalents is legitimate because the two worlds being compared are identical everywhere else. The same n requests go out, each carrying the same suffix, each producing the same output, and both sides are billed for those at the same rates — so the suffix and output terms are the same quantity on both sides of the comparison and drop out of it. What is left differs only in the rate applied to the prefix. Dividing through by the prefix's token count times the base input rate then removes the last two quantities the two sides share, and leaves a comparison between bare multipliers. Nothing has been assumed about how long the prefix is or what a token costs, which is why the answer holds for any prefix on any price list of this shape.

The split into one write and n − 1 reads is not an assumption either. An entry cannot be read before it exists, so the first matching request always misses and always pays the write, and every later one finds the entry already sitting there. That holds exactly when three things are true: the entry is still alive when the next matching request arrives, that request's bytes are identical, and no two requests are in flight against an entry that does not exist yet. The first is chapter 6, on how long an entry lives; the second is chapter 8, on a prefix that changes without anyone meaning it to; the third is the cold-start burst chapter 1 described, where several requests fill the same entry at once. Where any of the three fails, the number of writes rises, and every figure below moves with it.

The crossover is the second matching request, and it is worth writing out because it is the number worth carrying: two uncached sends cost 2.0 prefix-equivalents while a write plus one read costs 1.35, so a single reuse clears the premium with room to spare. Nothing in that calculation mentions the prefix's size, which means the break-even in requests is identical for a 1,024-token prefix and a 200,000-token one. Size does not move the break-even; it scales what sits on either side of it.

"With room to spare" can be made exact, and the exact crossing is more use than the table row. Caching costs the same as not caching when the write plus the later reads equals the number of plain sends, which is 1.25 + 0.1(n − 1) = n. Multiplying out the bracket gives 1.25 + 0.1n − 0.1 = n. Collecting the two constants on the left leaves 1.15 + 0.1n = n, and taking 0.1n from both sides leaves 1.15 = 0.9n. Dividing by 0.9 puts the crossing at a little over one and a quarter sends. Requests do not arrive in fractions, so the first whole number past it is two — but the useful reading is that the second request does not merely reach break-even, it clears it by nearly three quarters of a send.

The claim that size cancels survives the same treatment, and it is worth doing rather than believing. With a prefix of P tokens the uncached side is P × n and the cached side is P × (1.25 + 0.1(n − 1)); every term on both sides carries a factor of P, and dividing an inequality through by a positive quantity leaves which side is larger unchanged. So P disappears and the crossing sits wherever it sat. A 200,000-token prefix and a 1,024-token one break even at the same request, and the only difference is that one of them is doing it with nearly two hundred times as much money on the table.

The general form falls out of the same two lines. With a prefix of P tokens read n times from one entry, the saving against sending it uncached is

def savings_tokens(prefix_tokens: int, reuses: int,
                   write_rate: float = 1.25, read_rate: float = 0.1) -> float:
    """Effective input tokens saved by caching one entry read `reuses` times."""
    uncached = prefix_tokens * reuses
    cached = prefix_tokens * (write_rate + read_rate * (reuses - 1))
    return uncached - cached

savings_tokens(30_000, 1)      # -7,500  — the premium, unrecovered
savings_tokens(30_000, 2)      #  19,500
savings_tokens(30_000, 100)    #  2,665,500

which is P × (0.9n − 1.15). For any n worth caring about the 1.15 is noise and the expression is close enough to 0.9 × P × n to do in your head: nine tenths of the prefix size times the number of times it is read. Both terms are multiplicative, so a factor of ten on either side moves the answer by the same factor of ten, and an endpoint that wins on one term by a factor of thirty and loses on the other by a factor of twenty-seven is roughly a draw. The size of a prefix, on its own, says nothing about what caching it is worth.

That expression is worth deriving rather than accepting, because each of its two constants says something. Uncached costs P × n. Cached costs P × (1.25 + 0.1(n − 1)), and multiplying out the bracket gives P × (1.25 + 0.1n − 0.1), which collects to P × (1.15 + 0.1n). Subtracting the second from the first gives P × n − P × 0.1n − P × 1.15, and taking the common factor of P out front leaves P × (0.9n − 1.15). Subtracting term by term is allowed here because both expressions are already carried in the same unit, effective input tokens, which is what the prefix-equivalent was for.

Read in plain terms, the expression starts optimistic and then corrects itself. The 0.9 is what a read saves — a token that would have cost 1.0 costs 0.1 instead, so nine tenths of it comes back — and multiplying by n treats all n requests as reads. The 1.15 is the correction for the one that was not: the writer paid 1.25 where a reader would have paid 0.1, a gap of 1.15 prefixes, charged once for every entry. So the whole of caching's arithmetic is nine tenths off every request, less a bit over one prefix for each entry you had to buy in order to get it. The rest of this topic is about how many entries that turns out to be.

Two endpoints, one budget#

Take two candidates properly, with the reuse counts made explicit and expiry set aside for the moment.

Endpoint A is a document question-answering service. A document of 40,000 tokens sits in the prefix, users ask an average of six questions against it, and about 400 documents are processed a day. Each document is one cache entry, written once and read five times, so per document the saving is 40,000 × (0.9 × 6 − 1.15) = 170,000 effective tokens, and across 400 documents a day that is 68 million tokens.

Endpoint B is a ticket classifier. Its prefix is 1,500 tokens — framing, output schema and twenty few-shot examples — and it is identical on every one of two million daily requests, with the ticket text appended at the end. Ignoring expiry entirely and assuming one entry serves the day, the saving is 1,500 × (0.9 × 2,000,000 − 1.15) ≈ 2.7 billion effective tokens.

Both of those are worth doing slowly, since they are the only two multiplications in the chapter that decide anything. For endpoint A, nine tenths of six reads is five and two fifths of a prefix; taking off the write's 1.15 leaves four and a quarter; four and a quarter prefixes of 40,000 tokens is 170,000 effective tokens saved per document, and 400 documents a day makes 68 million. For endpoint B, nine tenths of two million is one and four fifths million prefixes, against which the 1.15 is invisible; one and four fifths million prefixes of 1,500 tokens is about 2.7 billion effective tokens. It is the same multiplication run twice. What separates the two answers is which of the two terms each endpoint happens to be large in.

The classifier is worth roughly forty times what the document endpoint is worth. Its prefix is 27 times smaller and its reuse count is 833 times larger, and those two ratios are most of the answer.

They do not quite multiply to forty, and the gap between their product and the real ratio is the write premium doing its work. Endpoint A buys an entry for every six requests and pays 1.15 prefixes for each one, which takes better than a fifth off its gross saving before anything else happens. Endpoint B buys an entry for every few thousand requests and pays close to nothing. It is also worth being exact about what the 833 counts: it is the ratio of the two endpoints' daily request totals rather than of their reads per entry, and requests are the right thing to count, because a saving accrues once per request that reads rather than once per entry that exists.

The last row is where the two endpoints look most similar, and it is the reason percentages are a poor basis for allocating effort. Endpoint A sends 2,400 × 40,200 ≈ 96.5 million input tokens a day and saves 68 million of them, which is 71%. Endpoint B sends 2,000,000 × 1,700 = 3.4 billion and saves 2.7 billion, about 79% of its input and rising toward 90% if you count only the prefix and ignore the ticket text that has to be sent either way. Reported as percentages these are the same kind of win. Reported in dollars one of them buys a contractor for a month and the other buys lunch, and the percentage hides that entirely because it is normalised by the very quantity — total traffic — that the decision turns on.

There is a second difference in that table which matters as much as the ratio. Endpoint B's hit is close to certain: two million requests a day is roughly 23 a second, so a matching request arrives long before any plausible expiry, and the only thing that can break the hit is a change in the prefix's content. Endpoint A's hit depends on a reader asking their second question soon enough, which is a behavioural assumption about people working through a contract. One of those is an engineering property and the other is a product analytics question, and a plan that treats them as equally reliable is not comparing like with like.

Latency reverses the ranking#

If the budget being allocated is time rather than money, the two endpoints swap places. A cache hit removes prefill work for the matched tokens, and prefill time is roughly linear in token count, so the time saved per request is proportional to prefix size alone — the reuse count does not enter into it. At a prefill throughput in the region of 30,000 tokens a second, endpoint A's 40,000-token document is about 1.3 seconds of prefill that a hit turns into perhaps 0.1 seconds of lookup, while endpoint B's 1,500 tokens are about 50 milliseconds that nobody was going to notice.

The reason the reuse count drops out is that money and time are accumulated by different parties. A day's spend is a sum over every request the endpoint served, so anything that happens more often contributes more to it. Latency is summed by nobody. Each caller waits for their own request and no one waits for the total, so a hit that removes a second of prefill removes a second for the person who made that request, and repeating it a million times does not make it a million seconds of anything. The reuse count multiplies the money and leaves the time where it was, which is why the two rankings come apart the moment two candidates differ in prefix size.

So the honest recommendation depends on which problem the team actually has. If a document reader's p95 time to first token — the wait that one request in twenty exceeds — is the complaint arriving in the support channel, endpoint A is the right place to spend the effort even though it is worth a fortieth of the money, because 1.3 seconds is the difference between an interface that feels responsive and one that does not. If the goal is the invoice, it is endpoint B and it is not close. What cannot be done is ranking candidates by prefix size and claiming to have optimised either one.

Three conditions that would move the recommendation#

Three conditions would change that answer, and each is worth checking before writing the plan rather than after.

The first is the minimum length from chapter 4. Endpoint B's prefix is 1,500 tokens, which clears a 1,024-token threshold with 476 tokens to spare and falls 548 tokens short of a 2,048-token one. On a model family where the small model requires 2,048, the entire $8,100 a day evaporates and the recommendation inverts again — a strange sentence to write about a 33% difference in a threshold, and exactly the cliff the previous chapter described.

The second is whether B's prefix is genuinely identical on every request. Twenty few-shot examples selected per request by a retriever, meaning a search step that picks the examples closest to the incoming ticket, or shuffled to avoid ordering bias, produce a different byte sequence every time. The endpoint then writes two million entries a day, reads none of them, and pays 1.25× on 3 billion tokens instead of 1.0×, turning an $8,100 saving into a $2,250 surcharge. The fix is to fix the selection, and the check is the write counter.

The third is where the ticket text goes. The classifier appends it, which is correct. An assembler that renders the ticket into a header line at the top of the prompt — "Ticket #48213 from acme-corp, priority high" — puts volatile content in front of stable content and produces exactly the same zero-hit, premium-priced outcome, with no error and no symptom other than the bill.

Those last two conditions are the same failure reached by two routes, and it is the failure that makes a write premium worth more than a footnote. In both, every request writes and none reads, so the endpoint pays a quarter above base input across its whole prefix volume rather than a tenth of it. Nothing else about the system changes. The completions are identical, the latency is identical, the evaluation suite passes, and the only symptom is an input line on the invoice that went up on the day caching was switched on. The belief that the premium is a rounding error is exactly right in the steady state, where writes are rare and reads are almost everything, and exactly wrong in the state where every request is a write.

The write premium is therefore the thing that makes caching a decision rather than a default. It sets a floor of one reuse before the mechanism breaks even, it makes a cold endpoint measurably worse off than an uncached one, and it puts the value of any candidate at nine tenths of its prefix size times its reuse count — an expression in which either term can dominate and neither can be guessed from the prompt alone.

Every reuse count above was nevertheless handed to you. Endpoint A "reuses five times" because six questions arrive per document; endpoint B "reuses very often" because two million requests arrive per day. Neither statement is a fact about caching — they are facts about traffic, and they only become reuse counts once you know how long an entry survives between requests. Six questions spread over ten minutes against an entry that lives five minutes is not five reads from one entry; it is some smaller number of reads from two or more entries, each paid for at 1.25×. That conversion is the second term of the multiplier, and it is the next chapter.