Three percent of hits is a count of customers per day

The slide has three numbers on it. The semantic cache on the customer support assistant is serving 34% of eligible traffic, model spend on that surface is down 28%, and a labelling exercise over 400 admitted hits found that 3% returned an answer that was wrong for the question actually asked. The team has done more measurement than most teams do, they have the labelled sample rather than a vibe, and the ask is to roll the same configuration out to two more product teams next quarter.

Three percent and twenty-eight percent are both percentages, of different denominators, in different units, and no amount of staring at them together produces a decision. One is a share of cache hits and the other is a share of a monthly bill, so the comparison being invited — three is smaller than twenty-eight — is arithmetic with no meaning. Both have to be converted into things that can sit on the same page: money per day, and customers per day.

Put both numbers into units that can be compared#

The surface takes 62,000 eligible requests a day. At a 34% hit rate that is 21,080 cache hits, and 3% of those is 632 answers a day that were correct for some other question. Model spend on the surface was $4,100 a day before the cache, so 28% is $1,148 a day saved. Dividing one by the other gives the exchange rate the proposal is actually offering: $1.82 of infrastructure saving per customer told something wrong.

Nobody in the room defends 19,200 wrong answers a month when it is written that way, and nobody was defending it before either — they were looking at a 3 next to a 28. The conversion is not a rhetorical trick, it is the only form in which the two quantities are commensurable, and it is why the first question to a team presenting cache metrics is what the eligible request volume is.

The 3% also carries more uncertainty than a single decimal suggests. Twelve wrong answers in a sample of 400 gives a 95% interval of roughly 1.6% to 5.2%, so the daily count is somewhere between 340 and 1,100. Sizing that interval down to something you would make a decision on takes a sample in the low thousands, which is a day of one person's labelling, and it is worth asking for before a rollout rather than after.

Several things that would change the answer are not on the slide at all. The severity distribution of the twelve errors is unrecorded, and the difference between an answer that describes an adjacent feature and an answer that states the wrong refund window is the whole decision. The saving is quoted gross, with no deduction for embedding calls on every eligible request, the vector store, or the extra generations caused when a customer who got a bad answer asks again — if 40% of bad hits produce a re-ask, that is 253 full-price generations a day flowing back against the saving. And there is no holdout arm, so the team cannot say whether escalation and rephrase rates on cached traffic differ from uncached traffic, which was the cheapest thing they could have built and the one that would have made the 3% redundant.

The errors land on the questions the most people ask#

The distribution of bad hits is not uniform over the query space, and the reason is mechanical. A cache hit requires a near neighbour, near neighbours are most available in the densest region of the query distribution, and the densest region is your most common questions. Every query near the head has many neighbours within the threshold, some of which are the qualifier-differing pairs the earlier chapters catalogued, so both the hit rate and the wrong-hit rate rise together as you move toward the head.

The aggregate 3% is a volume-weighted average dominated by the top rows, and the 4.4% in the first row applies to the questions that define what the product appears to say. A wrong answer on an obscure question reaches one person who probably knows they are in unusual territory. A wrong answer on the fourth most common question in the queue reaches a few hundred people a week, all of whom are asking the thing the product is most often asked, and all of whom get the identical wrong text.

Added error and absorbed error are not the same thing#

The argument that arrives next is that the model is wrong about 5% of the time anyway, so 3% on a third of traffic sits inside noise the business already accepts. It fails on three separate grounds, and the arithmetic one is the easiest to state.

A cache hit does not replace a model error, it adds to one. The stored answer was itself generated by the model, so it carries the model's own error rate with respect to the question it was generated for, and then the cache adds an independent chance that it is being served to a different question. On cached traffic the error rate is approximately 1 − (0.95 × 0.97), which is 7.85%, against 5% on the traffic that misses. Cached traffic is 57% more error-prone than uncached traffic, and the blended rate across the surface rises from 5% to about 6%. Framing 3% as "inside" 5% treats the two as alternatives when they compose.

The second ground is that the errors differ in kind. A model error on a support question is often a hedge, a partial answer, or a claim phrased with enough uncertainty that a reader treats it with suspicion, because the model that is unsure tends to sound unsure. A cache error is a complete, fluent, internally consistent answer that was written with full confidence — for a different question. It reads better than the average correct answer, since it was produced by an unhurried generation with good retrieval behind it, and there is nothing in it for a reader to catch.

The third ground is correlation, and it is the one that turns a rate into an event. The model's 5% is spread across distinct questions and distinct users, and it looks like background noise in the support queue because that is what it is. A bad cache entry is a single artifact that answers one popular question wrongly for everyone who asks it until it is evicted, so 400 wrong answers can be 400 identical wrong answers, which is a support macro (a canned reply the team writes because the same complaint keeps arriving), a pattern a customer notices across two conversations, and the kind of thing that gets screenshotted. Independent errors at 5% and correlated errors at 3% do not have the same consequences even when the counts match.

What has to be true before the other two teams turn it on#

The specific thing to refuse is the transfer of the configuration. The 0.93 was derived from one team's query distribution, one embedding model, one eligible volume and one cost of being wrong, and each of those inputs is different for the next surface. A team whose questions differ by qualifiers rather than by topic gets the flat precision curve from the second chapter and should not run the cache at all; a team whose surface is internal and read by engineers who will notice a wrong answer can run it far more loosely than a customer-facing one; a team with a tenth of the volume gets a tenth of the wrong answers per day from the same rate, which may move it from unacceptable to fine.

def decision_units(eligible_per_day, hit_rate, wrong_rate, baseline_spend, saving_pct):
    hits = eligible_per_day * hit_rate
    return {
        "wrong_answers_per_day": round(hits * wrong_rate),
        "saved_per_day": round(baseline_spend * saving_pct, 2),
        "usd_per_wrong_answer": round(baseline_spend * saving_pct / (hits * wrong_rate), 2),
    }

decision_units(62_000, 0.34, 0.03, 4_100, 0.28)
# {'wrong_answers_per_day': 632, 'saved_per_day': 1148.0, 'usd_per_wrong_answer': 1.82}

The conditions worth attaching to an approval are the ones that make the same conversation repeatable on each surface rather than the ones that lock in a number.

Approving on those terms costs the two teams a week each and produces three surfaces whose risk is stated in the same units, which is what makes a platform-level decision possible at all. Refusing on the grounds that a semantic cache is dangerous produces neither the saving nor the measurement, and the first team's numbers show a real $34,900 a month that is worth having on a surface where the error profile turns out to be acceptable.

The largest thing still unresolved is severity. Every number in this chapter counts wrong answers as interchangeable, and they are not: within the 632 a day there are answers about a neighbouring feature that cost a customer thirty seconds, and answers about a refund window that cost them a refund. Weighting them requires a severity taxonomy applied during labelling, which is more work than the labelling itself and which no published benchmark supplies. Until that exists, the workable substitute is to make the eligible set narrow enough — through the write gate, the allowlist and the denylist — that a wrong answer inside it cannot be one of the expensive kinds. That, rather than the threshold, is the control with the most leverage, and it is set before a single vector is stored.