The retrieval miss mode

Most selection changes trade quality against tokens along one axis: a block dilutes or it does not, a rule sits near the answer or far from it. Replacing a pre-loaded block with a tool call is different in kind. When the manual was pasted in, the governing rule was in context on 100% of requests by construction. It could be diluted, it could be ignored, it could sit too far from the instruction — every failure the earlier chapters measured was available. What could not happen is the rule not being there.

Just-in-time makes that possible, and the resulting failure is not a small version of the old ones. An answer produced with no governing rule in context states a policy the company does not have, in the same confident register as a correct one, to a customer who will act on it. The figures below come from a support agent whose 12,000-token refunds and cancellations manual was replaced with a lookup_policy tool, scored on a frozen 300-case eval.

What the flat mean is made of#

On that eval the mean answer quality is 3.71 out of 5 with the manual and 3.72 without it, mean input tokens per request fall from 13,400 to 2,700, and p95 latency is flat because the two-call path is offset by the shorter prefill. Printing the per-case deltas takes about a minute and changes what those numbers mean.

The 198 small gains sum to +49.5 and the 47 losses sum to −49.0, which is why the mean moved by two thousandths. The shape is the one to expect from this class of change rather than a coincidence: removing a large block improves most answers a little because most answers were being diluted by material they did not need, and ruins a few completely because those were the answers the material was there for.

Both of those sums are one multiplication each, and doing them is what makes the flat mean legible instead of suspicious. The gains are 198 cases each moving by 0.25, and a quarter of 198 is 49.5. The losses come in two sizes: 32 cases at 0.50 is sixteen points, and 15 cases at 2.20 is thirty-three, which together are the 49.0. Half a point of quality separates the two totals. Spread that half point across all 300 cases and the mean moves by less than two thousandths, which is the bottom row of the table and is indistinguishable from nothing at any precision anybody reports.

The last row is what makes the arithmetic dangerous rather than merely uninformative. Fifteen cases carry two-thirds of the total damage — thirty-three of the forty-nine points — and they do it from 5% of the eval set. A mean divides by the count of cases, so it treats those fifteen as five per cent of the evidence; the business treats them as answers stating a refund policy the company does not have. Nothing is wrong with the division. The problem is that averaging presumes the cases are commensurable, and an answer that is slightly less well supported and an answer that is confidently false are not two amounts of the same thing.

The bottom row is the new failure mode, and its magnitude is what distinguishes it. A −0.50 case is an answer that is correct and less well supported. A −2.20 case is an answer that states a refund policy the company does not have. Averaging those two together produces a number with no operational meaning, because the cost of the second is not five times the cost of the first.

Where the fifteen came from#

Retrieval telemetry answers this, and it has to be logged per case alongside the generation or the join is impossible afterwards. Each eval case carries a gold_rule_id — the policy rule a support lead says governs that question — and each request logs what the tool was asked and what it returned.

Twenty-two of the 300 questions never triggered a tool call at all, which is the failure the pasted manual most obviously could not have: the model read a question about a cancellation, judged it answerable, and answered it. Six of those 22 were wrong. Of the 278 calls, five returned an empty result set and 22 returned a rule that was on the same topic and not the one that governs — the general 30-day pro-rata rule instead of the unconditional 14-day withdrawal right, in nine of them. Retrieval hit rate on this eval is 251/300, or 83.7%.

Every branch of that flow accounts for its own cases, and checking that it closes is what makes it a diagnosis rather than a diagram. Twenty-two no-calls and 278 calls are the 300. Inside the 278, five empty results and 22 wrong rules and 251 correct rules are the 278 again. The hit rate divides the 251 by all 300 rather than by the 278, and that choice is deliberate: a question where the model never searched is a question where the governing rule was not in context, so counting only the searches would score the retriever on the questions it happened to be asked instead of on the questions the eval contained.

One identity in that flow is a finding rather than a bookkeeping check. The wrong-answer counts on the three failing branches are six, two and seven, which add to fifteen — exactly the number of cases in the −2.20 row of the per-case table. Those two artefacts were produced by different instruments, one from the judge's scores and one from the retrieval logs, and they land on the same fifteen cases. That is what closes the argument. The catastrophic cases are not a scattering of bad answers that happen to be bad; they are precisely the cases where the governing rule was absent, and each one has a named reason for the absence.

Two things in that flow are worth separating because teams routinely collapse them. A retrieval miss is not automatically a quality failure: 34 of the 49 misses produced an acceptable answer, either because the question did not really hinge on the rule or because the model's priors happened to match company policy. And a retrieval hit is not a quality success: four cases had the governing rule in context and answered against it, which is the old dilution failure surviving the change. Reporting hit rate alone would have overstated the problem, and reporting quality alone hid it.

The 34 is a subtraction worth doing, because the number that comes out of it is the one that would mislead a team most. Total misses are the 300 questions less the 251 hits, which is 49. Fifteen of those produced a wrong answer, so the other 34 produced an acceptable one — more than two in three. A team that adopted hit rate as its single metric would therefore be treating 49 events as failures when 34 of them cost nothing, and would spend its effort dragging the hit rate from 83.7% towards a figure whose remaining gap is mostly harmless. A team that kept quality as its single metric would see a mean that moved by two thousandths. The two metrics fail in opposite directions and for the same reason: each is answering half of a question about supply and use, and the fifteen cases sit where the halves meet.

The judge could not see it either#

The eval scores came from a model judge on a five-point rubric covering completeness, tone, actionability and "consistent with company policy". It gave several of the fifteen catastrophic cases a 4. That is not a defective judge so much as a judge running the same architecture as the thing it was grading, since its own prompt contained the question, the answer and the rubric, and no policy. Asked whether an answer is consistent with a policy it cannot see, a model scores whether the answer sounds like a policy.

The step to hold on to is that the judge and the system under test were failing in the same way for the same reason, one turn apart. The agent was asked to answer from a governing rule and did not have one, so it answered from what a refunds policy usually says. The judge was asked whether the answer matched a governing rule and did not have one either, so it assessed whether the answer read like something a refunds policy would say — and the answer did, because it had been generated by the same process against the same prior. A judge sharing the system's blind spot does not produce noisy scores. It produces confident scores that are wrong in exactly the cases that matter, which is a strictly worse failure than noise, because noise averages out and a shared blind spot does not.

The fix is to put the governing rule in the judge's context, taken from the eval set's gold label rather than from the retrieval under test, and to score groundedness as a separate field from quality — groundedness being whether the answer is actually supported by the governing rule rather than merely compatible with the tone of one.

def score_case(case, answer, retrieved_ids):
    """Two scores, computed from different inputs, never averaged together."""
    quality = judge(
        rubric=SUPPORT_RUBRIC,
        question=case.question,
        answer=answer,
        governing_rule=RULES[case.gold_rule_id],   # from the label, not the retriever
    )
    return {
        "case_id": case.id,
        "quality": quality,                        # 1-5
        "retrieval_hit": case.gold_rule_id in retrieved_ids,
        "called_tool": len(retrieved_ids) > 0 or case.no_call is False,
        "cites_retrieved_rule": cited_id(answer) in retrieved_ids,
        "contradicts_gold": judge_contradiction(answer, RULES[case.gold_rule_id]),
    }

contradicts_gold is the field that matters most and the one nobody adds until an incident forces it. It is a different question from "is this answer good": it asks whether the answer asserts something the governing rule denies, which is a check a judge can do reliably because it is close to entailment, the narrow question of whether one statement follows from another rather than the open one of whether an answer is good. Across the 300 cases it fires 15 times in the retrieval arm and once in the pasted-manual arm, and that single pair of counts is the sentence the comparison was missing.

The reason a judge is trustworthy on that question and shaky on the other one is a property of the question rather than of the model. Deciding whether an answer contradicts a rule requires only the two texts, and both are in the prompt: the rule comes from the eval set's gold label and the answer comes from the run. The verdict is determined by their content, so two competent readers agree. Deciding whether an answer is good requires a standard the prompt does not contain, so the judge supplies one, and what it supplies is a general sense of what good support answers look like — which is the same prior the failing system was drawing on. Narrowing the question is not a way of asking for less. It is what removes the judge's discretion from the part of the measurement that has to be reliable.

Four numbers, and none of them is a mean#

The comparison that decides this swap is the same eval set, both arms, reported on the axes that can move independently.

The three changes behind the third column are ordinary. A 640-token index of rule titles and one-line summaries is pre-loaded on every request, so the model always knows that a 14-day withdrawal rule exists even when its search fails to surface the text of it; that alone took the no-call count from 22 to 4, because the failure was mostly the model not knowing there was anything to look up. The tool returns the top three rules rather than the top one, which absorbs most of the wrong-rule cases at 310 extra tokens. And the answer is required to cite a rule ID, with a post-generation check that the cited ID appears in what was actually retrieved; on a mismatch the service retries once with the tool result restated immediately before the question, and escalates to a human if the second attempt also fails to cite.

The token column of that third arm is those three changes added to the second arm, and it is worth confirming rather than trusting. The lookup arm sends 2,700 tokens. The pre-loaded rule index adds 640 of them to every request. Returning three rules instead of one adds 310. Two thousand seven hundred plus 640 plus 310 is the 3,650 in the table, and 13,400 less 3,650 is the 9,750 the scenario quotes. Every token of the increase is accounted for by a named change, which is what allows the third arm to be argued about a change at a time rather than as a package.

The index is the change that repays study, because its effect is out of proportion to its size and the reason is a mechanism rather than a quantity. Twenty-two questions produced no tool call at all, and a model does not decline to search because searching is expensive; it declines because nothing in its context indicated there was something to find. Six hundred and forty tokens of rule titles and one-line summaries change that without answering anything: the model can now see that a 14-day withdrawal rule exists, so a cancellation question is visibly a question with a rule behind it. That took no-calls from 22 to 4. Adding more retrieval capacity would not have touched those 22 cases, because the retriever was never invoked on them.

The floor the pasted manual had#

None of this establishes that the pasted manual was safe. It produced one contradiction of its own, it scored worse on 66% of cases, and the earlier crossover work is a catalogue of ways a large block degrades the answers around it. What the pasted manual has is a floor: the material is present, so every failure is a failure of use rather than of supply, and the fixes for that are the ones already covered — better placement, fewer redundant restatements, a tighter selection over what enters the block.

Just-in-time removes that floor and replaces it with a measurable one. Retrieval hit rate is a quantity you can compute offline over a labelled set without generating a single answer, which makes it cheap to monitor and cheap to regression-test when the corpus changes.

What makes that possible is where the quantity's inputs come from. A hit rate compares the set of rule IDs a query returned against the one ID a support lead labelled as governing, and neither side involves the model producing text. Run the queries, collect the IDs, compare against the labels: no generation, no judge, no sampling variance, and the whole thing re-runs against a changed corpus in the time the retriever takes. That is why it can sit in continuous integration where a quality eval cannot. It is also why it is only half a measurement — a number computable without ever asking the model anything is, necessarily, a number that says nothing about what the model did with what it got. A pasted block gives you nothing equivalent, because "was the rule in context" is always true and therefore carries no information. The design with the extra failure mode is in some respects the more observable one, provided somebody spends the effort to observe it.

That effort is the real cost, and it is worth naming plainly because it is where this practice usually stops. The 300 gold rule labels took a support lead two days to produce and they go stale as policy changes; roughly 15% of them needed revision after a pricing change six months later. Without them there is no hit rate, no contradicts_gold, and no way to distinguish a retrieval failure from a reasoning failure, which leaves you back at a mean. The current research direction is to reduce that dependence — the AgenticRAG work on enterprise knowledge bases reports on retrieve-when-needed designs evaluated at this granularity — and no available technique removes the need for somebody who knows the domain to say what the right answer was.

A swap from pre-loaded to just-in-time is therefore two changes reported as one. The first is the token reduction, which is real and easy to measure. The second is a new supply failure whose size is a property of your retriever and your corpus rather than of the design, and which a mean will absorb without moving. Report them separately — per-case deltas, hit rate, contradiction count — and the decision becomes a trade somebody can accept or refuse rather than a number somebody has to believe.

Every figure in this chapter is about one block: the policy manual, which somebody added deliberately, which somebody proposed removing deliberately, and which got an eval arm in both directions. The system prompt it lives inside got none of that. It is 6,200 tokens, it has been edited by nine people, and no version of it has ever been run against the frozen set as an experiment. The next chapter is about reading a prompt like that as evidence of how it was maintained, and about the repair.