A threshold is chosen from two curves, not one

The documentation assistant is the surface where the harness came back clean, so this one is going ahead and the only open item on the ticket is the constant. Someone suggests 0.95 because it sounds safe. Someone else points out that the vendor example uses 0.85 and their demo works. A third person wants 0.90 because the hit rate at 0.95 looked thin in a quick test. All three numbers are being defended with adjectives, and the sample that would settle it is already sitting in the repository from last week's measurement.

What a candidate threshold buys is two quantities, and they are only meaningful together. The first is the hit rate — the share of eligible requests that find a neighbour above the bar, which converts directly into money and latency. The second is cache precision — of the requests that were served from the cache, the share where the stored answer was in fact correct for the question asked, which converts into wrong answers reaching users. Sweeping the labelled sample gives both at once.

The sweep is the argument#

The pairs from the mining harness already carry a score and a safety label, so producing the curves is a loop over candidate cut points. The only additional input is traffic volume, because a rate without a denominator cannot be argued about.

ELIGIBLE_PER_DAY = 9_400      # requests on this surface that could be cached
SAVING_PER_HIT   = 0.0142     # USD: 900 output tokens + retrieval we skip

def sweep(pairs, thresholds):
    rows = []
    for t in thresholds:
        admitted = [p for p in pairs if p["score"] >= t]
        if not admitted:
            continue
        hit_rate = len(admitted) / len(pairs)
        precision = sum(p["safe"] for p in admitted) / len(admitted)
        hits_per_day = hit_rate * ELIGIBLE_PER_DAY
        rows.append({
            "t": t,
            "hit_rate": hit_rate,
            "precision": precision,
            "saved_per_day": hits_per_day * SAVING_PER_HIT,
            "wrong_per_day": hits_per_day * (1 - precision),
        })
    return rows

Run against the documentation corpus, where the bands separated, the table reads like this.

The last column is where the decision lives. Moving from 0.95 to 0.93 buys $16 a day for 77 additional wrong answers a day, which is 21 cents per wrong answer. Moving from 0.93 to 0.91 buys $12 for 227 more, at five cents each. Nobody has to agree on the exact price of a wrong documentation answer to see that five cents is below it and that the step from 0.91 to 0.88 — two cents per wrong answer, a thousand a day — is not a trade anyone would make written out in those units. The threshold falls out at 0.95 or 0.93 depending on how much the surface costs when it is wrong, and that is a conversation with product rather than a preference about numbers.

Two cautions belong on that table before anyone quotes it. The precision figures at the top rows are computed on small counts — the 0.98 row rests on 54 admitted pairs, so a 99.4% precision has a 95% interval (the range the true rate could plausibly sit in, given a sample that small) running from roughly 92% to 100%, and the difference between the top two rows is not statistically real. Precision estimates need a few hundred admitted hits each to be worth arguing over, which means the sample has to be sized against the strictest threshold under consideration and not against the total. The other caution is that SAVING_PER_HIT is a modelled number rather than a measured one; it assumes the cached request would have generated a response of typical length, and if your long answers are also your least repeated ones, the hits skew short and the real saving lands below the estimate. Reconciling the predicted daily saving against the billing export in the first week is the cheap way to find that out.

Reading either curve alone produces a degenerate answer every time. Maximise hit rate and you arrive at whatever floor your index will return, because the curve is monotonic — hit rate only ever rises as the bar comes down — and never argues back. Maximise precision and you arrive at a threshold of 1.0, which is an exact-match cache with extra infrastructure. The pair of curves is what makes the choice an interior point rather than an endpoint.

The order-support surface from the previous chapter is worth putting through the same sweep, because it shows what the method looks like when the answer is not a number. Precision there runs 67%, 72%, 73%, 73% across the same cut points — flat, because the unsafe pairs live at the top of the range along with the safe ones. A flat precision curve means there is no interior optimum to find, and the sweep has told you that this surface is a run-or-don't-run decision rather than a tuning one. That is a more useful output than a number, and it is invisible unless you plot both curves.

Pricing the second stage instead of arguing about it#

The suggestion that always arrives at this point is a two-stage design: retrieve the nearest cached entry as before, then have a cheap model check whether the cached answer actually addresses the new question, and only serve it if the check passes. The reflexive objections are that it adds a model call to a path whose whole purpose is to avoid model calls, and that using a model to guard a cache is circular. Both dissolve once the numbers are on the table.

The verifier sees the new question and the cached answer, and returns a yes or no. That is around 1,100 input tokens and five output tokens on a small model, which is roughly $0.0003 and 180 milliseconds. Set against a generation you are skipping at $0.0142 and 1.8 seconds, the guard consumes 2% of the money saved and takes the cache-hit path from 40 milliseconds to about 220, which is still an order of magnitude under generation. On a miss you have burned 180 milliseconds and a third of a cent for nothing, and misses are the majority of traffic, so the honest accounting charges the verifier against every eligible request rather than every hit: at a 31% hit rate that is $0.0003 across the board against $0.0044 of average saving, and it still comes out well ahead.

What the second stage buys is a judgement made by something that reads the words. A model asked whether an answer about post-shipment cancellation addresses a question about pre-shipment cancellation has no difficulty at all, because the distinction is trivially available in language and only hard in a pooled vector. Running the verifier on the order-support pairs at a permissive 0.90 retrieval threshold took precision from 72% to 96% while keeping 38% of the hits, which is a materially different system from the one the previous chapter rejected.

It is not free of its own failure mode. The verifier has a false-accept rate, and yes-biased small models will wave through answers that are merely topical, so it needs its own labelled evaluation on the same pairs before you believe the 96%. It has to be shown the cached answer rather than the cached question, because the stored answer may hedge, may have been truncated, or may address only part of what the original question asked, and a check against the cached question would certify a match that the answer text does not support. It also makes the cache non-deterministic, so two identical requests can take different paths, which complicates reproducing a complaint. And there is a version of this analysis where the verifier wins the argument too thoroughly: if a small model at $0.0003 can adjudicate these questions, it is worth asking what fraction of the traffic that small model could simply answer, since the same money might buy correctness rather than a guard on someone else's cached correctness.

Shipping something whose failures throw no exception#

Everything above rests on a labelled sample of a few hundred pairs from one week. Production drifts away from that sample, so the number needs a standing measurement rather than a launch-day one, and the difficulty is that a bad hit produces a well-formed response, a 200 status code and a normal latency profile.

The direct signal comes from a shadow evaluation. Sample 2% of cache hits, call the model anyway in the background, and have a judge — a second model given both texts and asked whether they say materially different things — compare the served answer against the fresh one. At 2,900 hits a day that is 58 shadow generations, costing under a dollar, and it produces a daily divergence rate that is a live estimate of the precision column in the sweep. When divergence starts climbing without a deploy, something upstream has moved — a new product surface sending different phrasings, a change in how the front end suggests questions, or an embedding model version bump.

The indirect signals are only interpretable against a control, which is why the most valuable thing to build on day one is a permanent randomised holdout: 5% of eligible requests bypass the cache entirely, chosen by hashing the request id so the assignment is stable. Everything else then becomes a comparison between two cohorts rather than a number with no baseline.

The rephrase-within-60-seconds rate is usually the earliest mover, because a user who receives an answer to a neighbouring question does not file a complaint, they retype the question with more words. That behaviour is already in your logs and costs nothing to compute, provided the response log records whether the cache served the answer.

The threshold itself has an expiry date, because both curves were measured against one query distribution and one embedding model. A model version bump changes the geometry of the space, so the 0.93 that admitted 31% of traffic last quarter may admit 44% of it after the upgrade with a precision nobody has measured, and the index has to be fully re-embedded in any case since old and new vectors are not comparable. Treat the embedding model version as part of the cache key namespace, so that a bump orphans the old entries rather than mixing them, and re-run the sweep on a fresh week of pairs before trusting the constant again. Quarterly is a reasonable cadence in the absence of a model change; a marketing campaign that changes what people ask, or a new product area landing on the same surface, is reason enough to run it early.

The logging schema is what makes all of this possible after the fact, and it is cheap to get right before launch and expensive to backfill. Every response needs cache_hit, and every hit needs the id of the matched entry, the similarity score, the threshold in force, and the version string of the embedding model. Without the matched entry id, a complaint about one wrong answer can be investigated and a class of wrong answers cannot, because there is no way to ask which other requests were served from the same entry. That query becomes the whole story in the next chapter, where the cached answer is not merely for a different question but for a different customer.