Component evals and prompt changes

Most retrieval-augmented systems run two evaluations on every merge. The component one scores retrieval with recall@5 — the fraction of cases whose labelled relevant passage comes back somewhere in the top five results — and is cheap, stable and easy to reason about. The end-to-end one grades the final answer with an LLM judge and is none of those things. When a change lands and the two disagree, the reflex is to read the stable number as the reliable one.

That reflex is a category error whenever the change is a context-engineering change, because the cheap, stable, easy metric is blind to most of what context engineering does. Knowing the difference between "this metric did not move" and "this metric could not have moved" is the whole skill in this chapter, and it is a fact about your pipeline's wiring rather than about the numbers on the screen.

Where each metric's boundary sits#

Retrieval takes the user's question, possibly a rewritten version of it, and searches an index. A glossary block, a few-shot block or a style guide is assembled after that search has already returned, when the prompt is built. Nothing about deleting one of them can reach backwards into a search that already happened.

Stated as a fact about inputs, that sentence becomes the rule at the top of the chapter. A metric is a function of specific values: recall@5 takes the ordered list of chunk IDs the search returned, together with the labelled relevant chunk for that case, and returns a number. The only way a change can move that number is by changing one of those two things. The retrieved list is produced by the search; the search consumes the question and the index and nothing else; and a block concatenated into the prompt after the search has returned has no data dependence running back into either. The metric's inputs are therefore identical across the two arms, and a function of identical inputs returns an identical output. The flat recall number is not weak evidence about the glossary. It is not evidence about the glossary at all, and the distinction worth holding is between a measurement that could have moved and did not — which says something — and one that could not have moved, which says nothing. On a screen the two look exactly alike.

Anything inside ASM is inside the end-to-end scope and outside the component scope. That placement, not the observed numbers, is what tells you which instrument can register a change.

The four "no" rows are the cases where a green component eval carries no information about the change under test, and every one of them is a context-engineering change.

"Component eval" is not a single thing, and the second one most systems of this kind run behaves differently. Groundedness — scoring whether the answer's claims are supported by the context that was actually supplied — sits at the boundary between ASM and G, which puts prompt assembly inside its field of view rather than outside. Removing a glossary can therefore move groundedness, and in the case below it did: the model asserted a deadline that appears nowhere in the supplied passages, which is exactly what an ungrounded claim looks like. The general form of the rule survives; what changes is that you have to ask where each metric's boundary sits rather than sorting metrics into "component" and "end-to-end" and assuming the first kind is blind to prompts.

There is one exception worth holding on to, because it turns a "no" into a "yes" and it is not exotic. If the glossary were an input to the query-rewrite step — expanding "my RMA got stuck" into "my return merchandise authorisation got stuck" before the search — then it would sit upstream of R, and removing it would move recall. Whether a block is upstream or downstream is a fact about your pipeline's wiring, not about the block's contents, and the block_order and retrieval fields from the previous chapter on instrumentation are what let you check the wiring rather than assume it.

One point is not yet a finding#

A one-point drop on a 500-case set is five cases. Against a churn floor of eleven at a single repeat, five cases is genuinely unreadable — it is smaller than what the harness produces against itself. That is a reason to run more repeats, not a reason to conclude nothing happened.

The conversion in that first sentence runs in one step, and it is the same one every part of this topic leans on. A point is one hundredth of the zero-to-one score scale, the set holds five hundred cases, and one hundredth of five hundred is five. So the reported drop is five cases judged good with the glossary and not good without it. Read in the other direction, the smallest difference this set can express at all is a single case, which is a fifth of a point — the score's arithmetic precision is far finer than its reliability, and quoting it to three decimal places quotes two more places than the harness can defend.

Re-running both arms three times each and taking the per-case majority drops the floor to four cases and gives a stable answer. On the glossary removal, four of the five regressions persisted across all three repeats and one was a judge coin-flip that resolved differently each time. Four persistent regressions against a floor of four is right at the boundary, so the honest report is "small, borderline, and specific" — and the specificity is what makes it actionable, because four cases can be read in fifteen minutes.

What the repeats bought is worth naming, because it is not only a smaller number. Taking the per-case majority of three runs discards any case whose verdict was not stable across them, and the chapter on churn baselines measured the judge as the largest single source of that instability. So the four surviving regressions are cases that came back bad in every one of three independent runs of the identical configuration, which is a stronger claim than four cases that came back bad once. The fifth case, which resolved differently each time, was the judge rather than the glossary. That is why a count sitting right on the floor can still be worth reading case by case: the count is at the floor, and the cases behind it are no longer the kind of case the floor is made of.

sup-0207  "why was my RMA rejected under SLA-2?"
  with glossary:    treats SLA-2 as the shipping tier (correct: 2-day shipping SLA)
  without glossary: treats SLA-2 as the support response tier, quotes wrong deadline
sup-0311  "does the K-band adapter void the warranty?"
  without glossary: answers about a radio-frequency band; K-band is our product line code
sup-0348  "CX credit vs goodwill credit for this case"
  without glossary: uses the two terms interchangeably; they have different approval limits
sup-0412  "escalate to T2 or raise an RMA first?"
  without glossary: RMA expanded correctly, T2 mapped to the wrong queue

Every one of the four turns on an internal term that appears in the retrieved passages without being defined there. The passages say "issue an RMA under SLA-2" because internal documentation assumes the reader knows what those mean. Retrieval did its job perfectly — the gold passage was present at rank 1 in all four cases, which the gold_rank field confirms without re-running anything — and the answer was still wrong, because the model was reading a correct passage in a vocabulary it could not resolve.

That is a finding with a shape, and the shape immediately suggests a cheaper fix than the 2,000-token block. The glossary is doing definition lookup for roughly fifteen terms, and any given question needs two of them. A conditional glossary that includes only the entries whose terms appear in the question or in the retrieved passages sends about 180 tokens instead of 2,000, and re-running the ablation with that as a third arm is the obvious next experiment.

What a moved recall number would have meant#

Suppose the CI comment had instead read recall@5 0.913 → 0.887. The first response should not be a theory about the glossary. Recall moving under a prompt-assembly change means one of the arms is not what the arm file says it is, which is a defect in the experiment before it is a fact about the system. The candidates are short and checkable.

The pipeline may be wired differently from the diagram, with the glossary feeding query rewrite or query expansion — the case above where the "no" becomes a "yes". Or the two arms hit a live index and something ingested between them, which is the drift the first chapter of this topic closed by snapshotting or replaying retrieval; the index_snapshot field on the rows answers this in one query. Or the removal was implemented by editing a template that also contains the rewrite prompt, so more than one thing changed and the assert_single_variable check was never run. Or retrieval is genuinely non-deterministic — approximate nearest-neighbour search, which buys speed by probing part of the index instead of scanning all of it, can return different neighbours from one call to the next at the same recall level — in which case the A/A discipline applies to retrieval too and you need a retrieval churn floor before reading retrieval deltas. Only after all four are excluded does "the glossary affects retrieval" become the explanation, and by then you have learned something about your pipeline rather than about the block.

The fourth candidate is the one people find hardest to believe, so it is worth stepping through. An exact nearest-neighbour search compares the query against every vector in the index and is therefore deterministic by construction. Approximate search buys its speed by not doing that: it holds the vectors in a structure — a graph, a set of partitions — and visits only part of it, choosing where to go from runtime state such as which entry point it started from, how many candidates a worker had time to expand, and how concurrent work was scheduled. Two identical queries can therefore traverse different parts of the index and come back with different neighbours, while the fraction of relevant passages recovered stays about the same, because approximate search is built to guarantee a recall level rather than a particular list. Retrieval then has a churn floor of its own, measured exactly the way the model's was: run the same query set twice and count the cases whose returned IDs differ.

None of this makes component evaluation less valuable; it makes it a detector with a known field of view. Retrieval metrics catch exactly the failures that the end-to-end grade reports as a vague quality drop with no address, and they catch them for a tenth of the cost. A chunk-size change that moves recall is the case in point: a team watching only the end-to-end number sees four points disappear with nothing to point at.

The cost asymmetry is worth quantifying, because it explains why teams reach for the component number first and why the habit is hard to break. Scoring recall@5 over 500 cases means 500 retrieval calls and a set comparison: under thirty seconds of wall clock and effectively no model spend. The end-to-end run over the same 500 cases, at three repeats with a three-vote judge, is the $81 and twenty-seven minutes priced in the chapter on churn baselines. Nobody is being irrational when they read the number that returns in thirty seconds; the fix is not to run the cheap eval less, but to know in advance which changes it is entitled to comment on.

The mistake, then, is not running component evals. It is treating a flat component metric as evidence about a change it cannot observe, which is contagious in review threads because "recall is unchanged" sounds like reassurance. The one-line habit that prevents it: before reading either number, say out loud which of the two evals could possibly move under this change, and if the answer is "only end-to-end", do not report the component number as evidence at all.

There is a class of change where even that discipline runs out. The glossary removal was legible because the block carried meaning that specific cases needed and four failures pointed straight at it. A change that adds no meaning at all — the same questions, the same evidence, the same answer budget, with more surrounding material padding the request from 20,000 to 60,000 tokens — has nothing for a per-case reading to grab onto, and the failures it produces are diffuse rather than attributable.