A unique cache key hint scatters identical prompts

At 09:40 the per-route hit rate on agent-chat falls from 78% to 12% and stays there. The error rate is flat at 0.03%. p50 latency is up 90 milliseconds and p95 up 210. Output quality metrics — the eval suite that runs against sampled production traffic — are unchanged to three decimal places. Cost per request is up 40%. The only event in the change log is a deploy at 09:35 containing three things: a minor version bump on the HTTP and serialization libraries, a refactor of the module that assembles outbound requests, and a change to how correlation ids — the identifiers a service stamps on a request so that every log line it produces can be tied back together — are generated, so that they propagate across the new async worker pool.

The first message in the incident channel proposes opening a ticket with the provider, on the reasoning that cost per request rose and nothing on our side touched the prompt text. That instinct is the expensive one. The route spends about $12,000 a day, so a 40% increase bleeds roughly $200 an hour, and a day of waiting for a support reply costs $4,800 to learn that the provider has no incident.

Three facts rule the provider out before anyone types the ticket. The break begins five minutes after your release and not at any other moment in the preceding month. A provider-side pricing or capacity change would be visible across other accounts and on the status page, and would not respect your deploy schedule. And the observation is a hit rate falling, which is a statement about lookups, while a pricing change would leave the hit rate flat and raise the cost of the same hits. Cost per request is up here because reads at 0.1x were replaced by writes at 1.25x, which is the arithmetic of chapter 2 running in reverse.

Rank the three, then find the one measurement that splits them#

Three candidate causes, ordered by prior probability.

The correlation id change is first. Providers give you a way to hint at request identity — a cache key field, a metadata identifier, a user tag — and use it to steer equivalent requests toward the machine or shard — one of the interchangeable slices their serving fleet is divided into — where the prefix is already resident. If that field was previously derived from something stable and now carries a per-call correlation id, every request asks a different place, and every place has to write the prefix fresh. The symptom set matches exactly: no errors, because the request is perfectly valid; unchanged quality, because the prompt bytes never moved; latency up by the cost of prefill on the uncached span; cost up by the write premium.

# Before the refactor — one hint per tenant per route, stable for weeks.
def cache_key_hint(ctx) -> str:
    return f"{ctx.tenant_id}:{ctx.route}"

# After — request identity got consolidated into one helper, and the helper
# reached for the id that was already in scope.
def request_identity(ctx) -> dict:
    return {
        "cache_key": ctx.correlation_id,   # unique per call
        "trace_id": ctx.correlation_id,
        "idempotency_key": ctx.correlation_id,
    }

The request-building refactor is second. Anything that changes the serialized bytes of the prefix ends reuse at the first differing byte, which the prompt caching mechanics topic covered; a refactor that touches assembly order or whitespace produces the same graph.

The dependency bump is third by likelihood and first by unpleasantness, because it can produce the refactor's symptom with no line of your code having changed — a JSON library altering key ordering or unicode escaping rewrites your tool schema block without appearing in any diff you wrote.

Written as a ranking with the evidence attached:

One measurement separates the first hypothesis from the other two, and it takes about five minutes. Capture the outbound request bodies for two requests that should share a prefix, serialize the cacheable span exactly as it goes on the wire, and hash it.

import hashlib, json

def prefix_digest(request: dict) -> str:
    """SHA-256 of the bytes that make up the cacheable span, in wire order."""
    span = {"tools": request["tools"], "system": request["system"]}
    return hashlib.sha256(
        json.dumps(span, sort_keys=False, separators=(",", ":")).encode()
    ).hexdigest()

# Two requests from the same tenant and route, seconds apart:
#   digests equal      -> the bytes are stable; the lookup failed for another
#                         reason. Inspect the routing/cache-key field.
#   digests differ     -> the bytes moved. Diff the two spans and find the
#                         first differing offset; whose code owns that offset
#                         tells you refactor vs dependency.

Getting those two request bodies is the step that decides how long the incident lasts, and most services cannot do it. Request bodies are large, frequently contain customer text, and are therefore not logged, which leaves an on-call engineer trying to reconstruct a prompt from templates and a database. The cheap standing fix is to log the digest rather than the body, a digest being a short fixed-length fingerprint of the bytes that changes if any of them change and from which the bytes cannot be recovered: compute prefix_digest on every request and emit it as a low-cardinality field alongside the route and model. It reveals nothing about the content, it costs one hash of a few hundred kilobytes, and it turns this diagnosis into a single query — count distinct digests per route per five minutes, and watch it go from 1 to 4,000. A service that logs digests answers the content-versus-routing question before anyone has opened an editor.

Equal digests plus cache_read_input_tokens: 0 on every request is a decisive result: the content the provider would match on is identical, so the failure is in where the request was sent rather than in what it contained. That is the correlation id. Unequal digests sends you to the byte offset, and whether the first difference falls inside your own template or inside a library-serialized tool block separates the refactor from the dependency bump.

What the request path is actually doing#

The entry is not gone. It is sitting on a shard nobody asks any more, expiring quietly on its own schedule while every request writes a fresh copy somewhere else. That is why the hit rate settles at 12% rather than zero — occasional collisions still land two requests in the same place inside the entry's lifetime — and why nothing in the system reports an error. Every individual request is correct, and the only thing wrong is the population statistic.

Roll back, or pin it and ship forward#

Two proposals will arrive within a minute of each other, and both are defensible. Rolling back the whole deploy restores the previous cost immediately and needs no diagnosis. Pinning the specific thing and shipping forward is a one-line change: restore cache_key_hint to the tenant-and-route derivation and leave the correlation id doing tracing.

Ship forward, once the digest measurement is in hand — and it takes five minutes, which is $17 of bleed. A rollback of three changes tells you nothing about which one was responsible, so the next deploy either re-introduces the same fault or gets held up while somebody bisects the three changes, re-shipping them one at a time until the graph moves, in an environment that does not reproduce production's routing behaviour. Shipping the targeted fix also verifies the diagnosis: the hit rate returning to 78% within one entry lifetime is the confirmation, and if it does not return, you have learned something real rather than having merely made the graph go away.

That answer is conditional on the diagnosis being cheap, and it is worth putting a bound on it out loud. If the discriminating measurement is not in hand within about half an hour, the bleed has reached $100 and a rollback is the correct call, with the bisect moved to a quieter afternoon. The rule is that rollback is the right response to uncertainty, and targeted fix-forward is the right response to knowledge; the incident's first job is to convert one into the other quickly.

The bound also has to account for what a rollback costs, which is not zero and is not the same on every deploy. Reverting a dependency bump that carried a security patch means re-opening a vulnerability window and re-running the upgrade later with the same risk. Reverting a request-builder refactor that other in-flight branches have already rebased onto creates a merge problem for three people. Neither of those makes rollback wrong; both mean the on-call engineer should say out loud what is being reverted before reverting it, because "roll back the deploy" sounds atomic and is not.

Confirming the fix takes longer than shipping it, and the delay is the entry lifetime rather than the deploy. Once the hint is restored, the first request on each tenant-route pair still writes, and only requests arriving after that read. On a five-minute entry with steady traffic the hit rate climbs back to its previous level over roughly one lifetime; on a route with sparse traffic it can take an hour to become convincing. Declaring the incident resolved on the first cache read is premature, and declaring it unresolved because the rate is still 40% two minutes later is how a correct fix gets rolled back on top of a rollback.

The gate that would have caught it in CI#

This class of fault ships silently by construction. It produces no exception, no failed test, no changed output, and no schema violation — three properties that make it invisible to every check a normal service has. Catching it requires asserting on something no test would otherwise look at.

Two assertions, both cheap, both belonging in the same test file as the request builder.

FIXTURE = load_fixture("agent_chat_request.json")

def test_prefix_bytes_are_stable():
    """Fails whenever anything changes the cacheable span's bytes — our code,
    a template edit, or a serialization library's behaviour. Updating the
    constant is a deliberate, reviewed act."""
    assert prefix_digest(build_request(FIXTURE)) == (
        "3c1f9a0b6d2e47c8f5a1b93e04d7c26af8e0b415d9c73a26e18f0d5b2c94a771"
    )

def test_cache_hint_ignores_per_request_ids():
    """Two logically identical requests must produce the same hint, whatever
    their correlation ids are."""
    a = build_request(FIXTURE, correlation_id="corr-aaaa")
    b = build_request(FIXTURE, correlation_id="corr-bbbb")
    assert a["cache_key"] == b["cache_key"]

The first test is what makes a dependency bump loud. When a library changes its key ordering, the digest test fails on the upgrade PR, and the reviewer sees a diff of the serialized prefix instead of discovering it six weeks later on an invoice. The second encodes the invariant that the routing hint is a function of tenant and route and of nothing else, which is the property the refactor violated without anyone deciding to.

A third gate belongs in production rather than CI: a post-deploy canary — a check that runs against the first small slice of traffic to reach the new build, before it is promoted to the rest of the fleet — which samples a few dozen requests on each cacheable route a couple of minutes after rollout and asserts that cache_read_input_tokens is above zero on a majority of them. Two minutes is enough for an entry to be written and read, and a canary that fails there blocks promotion to the rest of the fleet before the bleed reaches a hundred dollars.

None of these gates fire on the thing the dashboard actually showed, which is a per-route hit rate falling seventy points in ten minutes. That signal exists, it is unambiguous, and on the day in question nobody was paged by it — because the alert was attached to the platform figure from chapter 1, where a seventy-point fall on one route registers as three points of noise. Making that signal fire correctly, and making the quarterly report say something a finance team can reconcile, are the same problem.