KV cache

Read these first

• Prefill vs decode

What you will know

• Build a minimal causal-attention implementation that caches K/V for a prefix and verify appended-token logits against full recomputation.

  A mismatch usually exposes causal-mask, positional-encoding, or layer-order bugs rather than a caching problem.

• Measure prefill and decode latency and GPU memory as prefix length grows, with cache reuse and without it.

  Prefill is largely parallel while decode is serial, so end-to-end latency alone hides which phase benefited.

• Reproduce prefix-cache hits and misses by varying tokenization, serialization, system messages, and the shared-prefix boundary, then trace the cache key.

  A single whitespace, tokenizer-version change, or altered tool schema can change token IDs even when the visible prompt looks equivalent.

• Run a batched or continuous-decoding workload with different sequence lengths and identify how KV allocation, padding, and eviction affect throughput and OOMs.

  Memory scales with layers, heads, head dimension, sequence length, and precision; fragmentation or padding can exhaust capacity earlier than nominal estimates.

• Diagnose a stale or mis-keyed cache incident by comparing cached and uncached generations and tracing the entry's creation, reuse, and invalidation events.

  Incorrect reuse can produce plausible text, making a correctness check and an auditable cache-key path more important than an obvious crash.

Chapters

1. 1

   The KV cache

   A transformer's key and value tensors at a given position depend only on the tokens before it and on the model weights, so a prompt that opens the same way as an earlier one can adopt tensors that were already computed. This chapter shows what a decoder actually stores per layer and per head, what a token of cache costs in bytes and saves in floating-point operations, and why appending one token to a cached prefix costs a single forward step instead of a full prefill.

   Read chapter
2. 2

   Verifying cached-prefix equivalence

   Reuse is only worth having if it is also correct. Build a minimal causal-attention loop that caches K and V for a prefix, then compare the logits of an appended token against recomputing the whole sequence from scratch. When the two paths diverge, the cause is almost never the cache: it is an off-by-one causal mask, a positional encoding applied at the wrong offset, or layers written back out of order.

   Read chapter
3. 3

   Prefix cache lifecycle and pricing

   Once key and value tensors outlive the request that produced them they stop being a local variable and become a store: blocks with a lookup key, a capacity shared with the running batch, an eviction policy, a TTL, a deliberate invalidation path, and a price. This chapter walks an entry from first fill to eviction, shows why a hit is worth roughly a tenth of an uncached token while a first fill costs a premium, and does the break-even arithmetic on both TTLs.

   Read chapter
4. 4

   Exact token-ID prefix matching

   Reuse demands an exact token-ID match starting at position zero, which makes this a prefix cache and not a similarity cache. Two prompts that print identically can key differently because of a stray space, a re-serialised tool schema, a tokenizer version bump, or an injected system message. This chapter shows how to compare token IDs rather than rendered text, how to locate the byte that moved the shared boundary, and how to turn a reusable fraction of a prompt into the fraction of input cost it actually saves.

   Read chapter
5. 5

   System prompt layout and cache breakpoints

   A line reading `Current date: 2026-08-05` in front of nine thousand stable tokens caches essentially nothing, because the varying token sits before everything stable. The same arithmetic explains why hoisting retrieved documents above the system prompt can quadruple cost and latency. Position bias is real, so this chapter resolves the conflict by keeping that lever inside the suffix rather than reordering the prefix, and shows where the breakpoint belongs as a conversation grows.

   Read chapter
6. 6

   Measuring TTFT and TPOT separately

   Cache hits shorten prefill, and prefill is the phase that sets time to first token; decode was never the work being skipped, so the per-token generation rate stays where it was. Measure the two phases separately as prefix length grows, with reuse and without it. A single end-to-end latency figure averages them, weights the average by output length, and can hide a large prefill win behind an unchanged decode bottleneck.

   Read chapter
7. 7

   KV cache capacity planning

   Cached prefixes live in GPU memory that scales with layers, heads, head dimension, sequence length and precision, so retained entries compete with the running batch for one pool of capacity. This chapter computes that pool explicitly, shows why padding and block allocation change the usable figure, and runs the mixed-length ramp that finds the concurrency where throughput turns into preemptions and the hit rate collapses.

   Read chapter
8. 8

   Mis-keyed and stale cache entries

   When a prefix entry is keyed wrongly or never invalidated, nothing crashes: the model writes coherent, confident output continued from someone else's prefix or a stale document set. Diagnosis means regenerating the request with reuse disabled and diffing the two, then tracing when the entry was created, reused, expired and invalidated — which is only possible if those events were logged with the key that produced them. This chapter builds a key path that can be audited, and the invalidation lever to use once it has been.

   Read chapter
9. 9

   Cache hit rate and cost forecasting

   Discounted cached input makes a 20,000-token prefix across forty turns look almost free, but only at a hit rate nobody observes. Every task's first turn is a miss by definition, entries expire on a TTL, routing moves requests onto replicas holding nothing, and a one-character template edit takes the rate to zero without raising an error. This chapter works the cost curve for a hit rate below one, names the six mechanisms that hold the rate down, and replaces the assumed rate in a forecast with an effective input multiplier measured from real traffic.

   Read chapter