Prompt cache operations

Read these first

• Prompt caching mechanics
• Metrics and SLOs

What you will know

• Report a real cache hit rate

  Aggregate hits against eligible lookups and segment by model, route, tenant, and prefix class; a global numerator can hide one broken path.

• Calculate the break-even hit rate given write premium and read discount

  The write charge is paid before a future reuse is known, so short-lived prefixes or low reuse can cost more than uncached requests.

• Show a change that raised the hit rate

  Prefix ordering, request affinity, or TTL changes should produce an attributable before-and-after result rather than a coincidental billing fluctuation.

• Compare realized savings across workload shapes

  A workload dominated by long reusable prefixes can look excellent while short, unique prompts produce little benefit and still incur writes.

• Detect a deployment that silently destroyed the hit rate

  Changing serialization, model routing, or worker affinity can make equivalent requests land on different cache keys without causing errors.

Chapters

1. 1

   Hits over eligible lookups, split by route

   A "prompt cache hit rate: 71%" computed as cached input tokens over all input tokens drags requests that had no cacheable prefix into the denominator, so the figure tracks the traffic mix and cannot distinguish a cache that stopped working from a change in the questions. Count hits against eligible lookups instead, segmented by model, route, tenant and prefix class. A blend that never moves is usually one route at zero averaged with one at 95%.

   Read chapter
2. 2

   The write premium is paid before any reuse exists

   With writes at 1.25x base input and reads at 0.1x, a prefix needs roughly 1.28 requests inside its lifetime to pay for itself — about a 22% hit rate. That clears comfortably for a 20,000-token agent prompt called thousands of times an hour and fails for a 50,000-token document asked one question, where a 25% surcharge on every write becomes a permanent line item nobody goes looking for. Per-session prefixes sit in between and have to be measured.

   Read chapter
3. 3

   Ninety percent hit rate, four percent off the bill

   Two teams on the same provider both report 90%; one bill fell 60% and the other 4%, with neither measuring anything wrongly. The discount reaches only the repeated input prefix, while variable input and every output token are billed exactly as before. On a workload of a 400-token prefix, long user documents and long generations, a near-perfect hit rate is arithmetically incapable of moving the total, which is also why pushing 90% to 98% can be worth nothing.

   Read chapter
4. 4

   A new customer can take credit for your prefix reorder

   Moving the variable part of the prompt behind the stable instruction block raised the hit rate from 44% to 81% and cut cost per request 30% — in the same week a large customer with an entirely different usage pattern onboarded. A platform-wide before-and-after cannot separate the two effects. Compare the same route and tenant segments across the boundary with the new tenant excluded from both sides, or the result is a billing coincidence you claimed.

   Read chapter
5. 5

   A unique cache key hint scatters identical prompts

   At 09:40 the hit rate falls from 78% to 12%: no errors, no prompt edits, latency slightly up, cost per request up 40%, and one deploy five minutes earlier carrying a dependency bump, a request-building refactor and a change to how correlation ids are generated. Equivalent requests are now landing on different cache keys. A provider-side change does not start five minutes after your release, and escalating outward before reading your own diff costs a day.

   Read chapter
6. 6

   A saving figure that reconciles against the invoice

   One platform number, stable for two quarters, with a route near zero and a route above 95% hidden inside it. A usable quarterly report carries hit rate and eligible volume per route, write and read token counts, and a saving computed against what that same traffic would have cost uncached — multiplying cached tokens by the read discount ignores the premium and always overstates. The alert that replaces it fires on a per-route collapse, never on the average.

   Read chapter