Prefix cache lifecycle and pricing

Within a single generation, the key and value tensors of a prompt are a local variable: they are allocated when prefill runs, appended to as tokens are emitted, and released when the request ends. A serving layer does something different with them. After the response has been sent, the prefix's tensors stay resident — in device memory, or spilled to host memory, on the node that computed them — indexed by a lookup key, and a request that arrives twenty minutes later from an unrelated user can adopt them. That is what turns reuse into an object with a life of its own.

This chapter is about that object. It has capacity, and the capacity is shared with the sequences currently generating rather than reserved separately. It has a replacement policy, so entries disappear without anyone asking them to. It has an expiry, so a gap in traffic costs you an entry that memory pressure would have left alone. It has an invalidation path, which is the only one of the three you control directly. And on a hosted API it has a price on both sides — reads discounted, writes at a premium — which makes "should this prompt be cached at all" an arithmetic question with an answer that is sometimes no.

An entry's life#

The unit of the store is usually not a whole prompt. vLLM's automatic prefix caching, which is the reference design most self-hosted stacks either use or imitate, chops the sequence into fixed-size blocks — sixteen tokens is the common default — and hashes each block's content together with the hash of the block before it. Block n's identity therefore encodes the entire sequence up to block n, which is what makes a lookup a walk: take the request's token IDs, hash forward block by block, and stop at the first block whose hash is not resident. Everything before that point is a hit and skips prefill; everything after it is computed and written. Two prompts that share their first 4,096 tokens and then diverge share 256 blocks and no more, with no coordination between them and nobody having declared the boundary in advance.

The chaining is not a hashing convention chosen for tidiness. It is forced by what the blocks contain. A block holds the key and value tensors for sixteen positions, and those tensors are a function of every token from position zero up to the position in question. Two blocks whose sixteen token IDs are identical therefore hold different numbers if the tokens ahead of them differed, because the residual stream arriving at those positions was different. Keying a block on its own content alone would let a lookup return tensors computed under some other history, which is the one thing the reuse argument does not license. Folding the previous block's hash into this block's identity makes the key stand for the whole run of tokens from zero, which is exactly the span the tensors depend on. The chain is the key doing what the arithmetic requires of it.

That also makes the block counting straightforward rather than mysterious. Sixteen tokens to a block means a shared region is measured in sixteens, so 4,096 shared tokens is 4,096 divided by 16, which is the 256 blocks above. The first block on which two prompts differ is where the chain breaks, and every block after it carries a different predecessor hash whether or not its own sixteen tokens happen to match. Sharing is a prefix length divided by a block size, and nothing else enters into it.

Each block, once resident, moves through a small number of states.

The three exits are worth separating because they have different causes and different fixes. Eviction is the store telling you it is out of room: blocks whose reference count is zero — nobody is currently generating against them — are dropped in least-recently-used order to make space for a request that needs to allocate. It is not an error condition and it happens constantly under load. Expiry is a time bound. Hosted APIs give the entry a TTL — a time to live, after which the entry is dropped whether or not anything else needed the room — commonly five minutes, refreshed on each hit; a conversation with a two-minute human pause between turns keeps its entry alive, one with a twenty-minute pause does not. Invalidation is deliberate: you changed the model version, or the prompt template, or you decided that one tenant's blocks must never be reachable from another tenant's request, and the key has to reflect that.

Those three, and the difference between them, are what a hit-rate graph is actually made of. A rate that sags during peak traffic and recovers off-peak is eviction. A rate that is fine for interactive sessions and terrible for a batch job with long gaps is TTL. A rate that goes to zero at a deploy boundary and stays there is invalidation, deliberate or accidental.

The store shares one budget with the running batch#

Cached blocks are not free-floating; they sit in the same GPU memory pool that the currently-running sequences allocate from. On an 80 GB A100 running a 70B model in bf16, weights take about 140 GB across two cards, leaving perhaps 8–10 GB per card for everything else once activations and the CUDA context are accounted for. At 320 KiB per token of KV, that is on the order of thirty thousand tokens of total capacity per card — shared between every in-flight sequence's growing cache and every retained prefix.

Which means retention and concurrency are in direct competition, and the eviction policy is the referee. Push the batch size up and resident prefixes get dropped to make room, so the hit rate falls and prefill work goes back up. Reserve more space for retention and fewer sequences fit, so queueing latency rises. Run a workload with wildly mixed sequence lengths and the allocator's behaviour starts to matter as much as the totals, because a 30,000-token capacity does not mean thirty thousand tokens of usable space when the free blocks are scattered. Chapter 7 does that arithmetic and its failure modes; the point to carry forward is that the store has a budget, the budget is contended, and "increase the cache" is never a free instruction.

Keeping entries reachable#

An entry is only useful to a request that reaches the node holding it, which makes the load balancer part of the cache. Round-robin across eight replicas gives a request holding a hot prefix a one-in-eight chance of landing where that prefix is resident, and the other seven times it triggers a fresh prefill and a fresh write — so a fleet ends up storing eight copies of the same blocks and reading each of them one eighth as often as it could.

The usual remedy is to route on a hash of the prompt's leading tokens, so that all traffic sharing a prefix converges on one replica. A production version needs two refinements. Consistent hashing rather than modulo, so that adding a replica moves roughly 1/N of the keys rather than reshuffling all of them; and a load ceiling, so that a single extremely popular prefix does not pin all its traffic to one saturated node while seven sit idle. Engines that ship this — SGLang's cache-aware scheduler is the clearest example — combine the two by preferring the cache-holding replica until its queue exceeds a threshold, then spilling.

The other half of reachability is time. TTLs are refreshed on hit rather than fixed from creation, so a prefix under continuous traffic never expires and one used twice an hour never survives. Where the traffic is bursty — a nightly batch, a business-hours-only tool — the choice is between a longer TTL at a higher write premium and deliberately re-warming. Re-warming is a request that exists only to fill the cache: send the stable prefix with the output capped at zero tokens, and the provider runs prefill, writes the entry, and returns immediately with no output tokens billed. Firing that at worker startup, after a deploy, or a minute before a scheduled window turns the first real request of the day into a hit. It is worth doing exactly when first-request latency is user-visible and the traffic has gaps; under continuous traffic it is a pure extra write, since real requests keep the entry alive on their own.

What a hit is worth on the invoice#

On a hosted API the trade appears directly in pricing rather than in GPU-seconds, and the shape is consistent across providers even where the numbers differ. Anthropic's prompt caching is a representative case: a cache read is billed at 0.1× the model's base input rate, and a cache write carries a premium of 1.25× base for the default five-minute TTL, or 2× base for a one-hour TTL. Output tokens are untouched by any of this.

That shape is not arbitrary, and knowing where it comes from makes it predictable across providers who quote different numbers. A read is cheap because the provider skips prefill for those tokens, and prefill is the arithmetic-heavy phase; what remains is the block lookup and the memory traffic of attending over tensors that are already there, which is real but small. A write is dear because it commits the provider to holding several hundred megabytes of device memory for the length of the TTL, whether or not anything ever reads it, and device memory is the scarcest resource in the system. The discount is a discount on compute and the premium is rent on memory, which is why a longer TTL costs more for the same reason a longer lease does.

The premium on the write is the part people forget, and it sets a break-even.

The last column is one comparison per row, and it is worth performing rather than reading. Serving a prefix twice with no caching costs two full-rate passes, which is 2.0 in the units of that table. Serving it twice with a five-minute entry costs 1.25 for the write on the first request plus 0.1 for the read on the second, which is 1.35, so caching is already ahead at two requests. Serving it twice with a one-hour entry costs 2.0 plus 0.1, which is 2.1 against an uncached 2.0 — behind, narrowly. Add a third request and the one-hour case reaches 2.2 against an uncached 3.0 and has paid for itself. The pattern underneath every row is that the write premium is a fixed cost paid once while the discount is earned per read, so the question is never whether caching is cheaper but how many reads an entry collects before it disappears.

A prefix served exactly once costs 25% more than not caching it at all. The longer TTL keeps entries alive across gaps in bursty traffic, which is worth having, but it needs a third read before it pays for itself.

Now run the case you will be asked about in an interview: an 8,000-token system block — instructions plus tool schemas — followed by a short user message of about 60 tokens, on a five-minute TTL. Uncached, each request bills 8,060 input tokens at 1.0×. Cached, the first request bills 8,000 × 1.25 + 60 = 10,060 effective tokens, and every subsequent hit bills 8,000 × 0.1 + 60 = 860.

Both of those lines are worth reading component by component, because the same two moves generate every figure that follows. On the first request the 8,000 stable tokens are being written, so each is billed at 1.25 and their share of the bill is 8,000 multiplied by 1.25; the 60-token message sits outside the cached region and is billed plainly, and the two together are the 10,060. On every later request those same 8,000 tokens are read rather than written, so each is billed at 0.1, and 8,000 multiplied by 0.1 with the same 60 added on top is the 860. "Effective tokens" here means tokens at the base rate: a token billed at 0.1× counts as a tenth of a token, which is what lets these totals be compared against an uncached count without a price appearing anywhere.

Over twenty requests that is 10,060 + 19 × 860 = 26,400 effective tokens against 161,200 uncached, an 84% reduction in input cost. In the steady state, ignoring the one-off write, it is 860 against 8,060, or 89%. That is the honest ceiling for this prompt shape, and it is high because the shape is favourable — 99% of the input is the stable prefix. Change the mix and the ceiling moves: a request with 8,000 cacheable tokens and 4,000 tokens of freshly retrieved context saves 89% of two thirds of its input and nothing on the rest, which is about 59% overall. And none of it touches output tokens, so an agent that reads 8,000 and writes 2,000 sees a much smaller dent in its total bill than the input-side percentage suggests.

Those percentages come from two different denominators, and both divisions are worth seeing. The uncached figure of 161,200 is 8,060 tokens on each of twenty requests. Dividing 26,400 by 161,200 leaves about a sixth, so 84% of the input cost is gone, and that figure carries the one expensive write inside it. The steady-state figure drops the write and compares a single hit against a single uncached request, so 860 divided by 8,060 leaves about a ninth and 89% is gone. The mixed case adds one step to the same procedure: 8,000 tokens of that request are cacheable and 4,000 are not, so the cacheable share is two thirds, and 89% of two thirds is about 59%. A prompt's ceiling is always the steady-state discount multiplied by the fraction of its input sitting inside the cacheable prefix, which is why the shape of the prompt decides the answer before any provider's rate card does.

You verify all of this from the response rather than from arithmetic, because the arithmetic assumes a hit that may not have happened:

{
  "usage": {
    "input_tokens": 60,
    "cache_creation_input_tokens": 0,
    "cache_read_input_tokens": 8000,
    "output_tokens": 412
  }
}

Three fields, and they partition the prompt: input_tokens is the uncached remainder only, so total prompt size is the sum of all three. A run of requests where cache_read_input_tokens stays at zero and cache_creation_input_tokens keeps climbing is a prompt that is rewriting the cache on every call and paying the write premium for nothing — the most expensive configuration available, worse than not caching at all.

The reason a service can run in that state for months is that every intuition about caches says it cannot exist. A cache that misses is normally free: you pay for a lookup, get nothing back, and do the work you were going to do anyway, so the worst case of adding one is that it does not help. Here the miss is not free, because the request that failed to read also writes, and the write is dearer than the plain request would have been. Nothing about latency, output or error rate distinguishes a deployment in that state from a healthy one. The signal exists in those three fields and on the invoice, which is why an alert on cache_read_input_tokens sitting at zero is worth more than any amount of reading the prompt.

Durability, scope and meaning#

The store is not durable. The blocks live in a process's memory pool, so a pod restart, an OOM kill, or a rolling deploy takes them with it. Some engines can spill to host RAM or NVMe to survive tighter memory pressure, and that helps with eviction, but nothing about a prefix cache is designed to survive a process death. Treat a cold cache after any deploy as the normal case and size your capacity headroom for the recovery window rather than the steady state.

It is not global, either, in the sense a Redis cluster is. There is no shared namespace and no replication; the routing described above is a way of steering requests toward memory that already exists, not a way of making that memory available everywhere. One practical consequence is that a change to the routing policy can take the hit rate to zero without anyone touching a line of cache code, and the change will look like a load-balancing improvement in every review it passes through.

And it is not a semantic cache. Nothing in the block-hash walk asks whether two prompts mean the same thing; it asks whether the bytes agree, block by block, from the beginning. Two prompts that a human would call identical can hash differently on the very first block and share nothing at all.

Treating the prefix cache as a store rather than an optimisation is what makes its behaviour predictable. Blocks are created by a miss, kept alive by hits, and removed by exactly one of eviction, expiry or invalidation; they are reachable only from the node that holds them, which makes routing a cache concern; and they are priced asymmetrically, so the first fill of an entry is the most expensive request that entry will ever serve. Those facts together explain a hit-rate graph, a latency graph after a deploy, and an invoice that went up after caching was enabled, which are otherwise three unrelated mysteries.

The last of the three properties is where the rest of this topic lives. Two prompts share blocks when their bytes agree from the beginning, and the entire practical difficulty of getting hit rates up is that "agree from the beginning" is a statement about integers, not about text.