High context recall with low answer accuracy puts the loss after retrieval

The previous six chapters built five metrics. This one is about reading two of them together, because a pair of numbers can locate a fault that neither one locates alone, and locating a fault is what decides where a quarter of engineering time goes.

The pair is context recall and end-to-end answer accuracy. Recall says how often the passages needed to answer the question arrived in the prompt. Answer accuracy says how often the right answer came out. When the first is much larger than the second, the material was in front of the model and the model did not produce the answer, and everything upstream of the prompt is running well enough that improving it further cannot recover most of the loss. That is a claim about which component to work on, and it is worth being careful with, because a component-level claim is exactly what an aggregate is least able to support.

Verifying the 0.94 before spending it#

Take a system reporting context recall of 0.94 and end-to-end answer accuracy of 0.62. Those two numbers describe a pipeline in which the passages needed to answer arrive on 94% of queries and the right answer comes out on 62% of them. At most six points of the thirty-eight-point loss can be blamed on evidence that never showed up. The other thirty-two points happen after the passages are in front of the model.

That inference is only as good as the 0.94, which is why the first move is not to redirect anything. Each of the previous chapter's four pathologies inflates recall in a way that would make the argument collapse, and the check took a day: the gold sets had been pooled across BM25 and two dense retrievers rather than harvested from the live one, 13% of gold IDs fell outside the current retriever's top 50, the eval questions were sampled from six weeks of logs rather than generated from the chunks, and the eval k matched the serving k of 8. The 0.94 held at 0.92 once unanswerable queries were added at 12% of the set, which is within the range the argument tolerates.

Read those four results as a set and they answer the four pathologies one for one, in the same order the previous chapter listed them. Thirteen per cent of gold outside the retriever's reach sits inside the healthy 10–20% band, which is what a non-circular set looks like. Questions sampled from logs cannot have been generated from the chunks they are graded against. An eval k equal to the serving k measures the pipeline that ships. And the drop to 0.92 under added unanswerable queries is small enough that the set is not being flattered by an absence of hard cases. Had any of those failed, the correct next step would have been rebuilding the labelled set, because a quarter redirected on a fabricated number is worse than a quarter spent on the wrong component.

Splitting thirty-two points into named failures#

"The loss is downstream" is a claim about a component, not a diagnosis. Turning it into work means taking the queries that failed end to end and scoring each one on the metrics from the earlier chapters, which is a run over a few hundred samples that costs a few dollars and an hour.

def classify(sample: EvalSample, scores: dict[str, float]) -> str:
    if scores["context_recall"] < 0.5:
        return "retrieval_miss"
    if scores["faithfulness"] < 0.85:
        return "unsupported_claims"
    if scores["answer_relevance"] < 0.70:
        return "grounded_wrong_question"
    return "grounded_relevant_still_wrong"

The order of those tests is the design decision in the function, and it is forced by what each metric can be trusted to mean. Recall is checked first because a query whose evidence never arrived will also score badly on faithfulness, so testing faithfulness first would file retrieval misses as generation failures. Relevance is checked after faithfulness for the same reason in the other direction: an answer full of invented material is often also off-topic, and the invention is the more actionable of the two. Each test therefore claims a query only once the tests above it have declined it, which is what makes the four classes a partition rather than four overlapping counts.

Applied to the 152 failing queries out of 400:

Two things in that table close exactly and are worth checking before anything is argued from it. The four counts add to 152, and 152 of 400 failing leaves 62% succeeding — the end-to-end accuracy the pipeline reports. So the classification covers every failure the accuracy figure is made of, with none unclassified and none double-counted, and the shares in the third column are shares of the whole loss rather than of a sample of it.

The second check is against the recall figure, and it is a bound rather than an identity. Recall of 0.94 means the evidence was missing on about six queries in a hundred, so at most a little under a quarter of the 152 failures could be retrieval misses. The classifier found 16, which is comfortably inside that ceiling, and the gap between the two is expected: the ceiling counts any query with less than complete evidence, while the classifier only files a query as a retrieval miss when more than half its evidence is absent. A count above the ceiling would have meant the recall figure and the per-query scores disagreed and one of them was wrong. A count below it is the ordinary case.

The quarter as approved was aimed at 11% of the observed failures. That is the sentence to bring to the planning conversation, and it is stronger than any argument from the aggregate pair, because each row names a different team's work.

Every branch of that tree was built in an earlier chapter of this topic, and running it query by query is what converts four dashboard numbers into a work breakdown.

The two downstream classes need different teams#

Unsupported claims are a grounding problem, and the reflex fix — rewriting the system prompt to say "only use the provided context" — is the one that disappoints. On this pipeline, three rounds of prompt hardening over two weeks moved faithfulness from 0.71 to 0.74 on the failing slice, roughly one and a half times the run-to-run noise.

What moved it materially were changes to what the system does rather than to what it is told. Giving the model an explicit abstention output and rewarding it in the eval for using it took faithfulness on that slice to 0.83, because a large share of unsupported claims are the model refusing to say "the documentation does not cover this". Swapping the generator for a model with stronger instruction-following added another four points. And a verification pass that decomposes the draft answer, checks each claim against the passages, and regenerates when a claim fails took the slice to 0.91, at 1.9× the generation cost and +1.4 s at p50 — which is chapter 2's procedure moved from the eval harness into the serving path, a common destination for a metric once you trust it.

Those four steps decompose the whole movement, and reading them as gaps rather than as levels is what makes the lesson transferable. From 0.71 to 0.91 is twenty points, and the four changes account for all twenty with nothing left over: three of them from two weeks of prompt wording, and the remaining seventeen from an architectural change, a model change and an added verification stage. The wording round is the cheapest thing on the list and the smallest effect on the list, and its three points sit close enough to the noise floor that a team running only that change could not have told whether it worked.

That three-point yield is worth holding next to chapter 3, where an instruction telling a faithfulness judge to ignore the reference answer recovered three of sixteen points and left thirteen standing. Two different chapters, two different models being instructed, two different problems — and the same measured size for what a wording change buys. The common cause is that an instruction is a constant: the same words reach the model whether the context supports the claim or says nothing about it, so the defence cannot vary with the property it is supposed to police. Both times the rest of the gap came from changing what the model was given or what it was allowed to emit.

Grounded answers to the wrong question are usually not a generation problem at all. In 33 of the 44 cases here the retrieved passages were about a neighbouring topic and the model summarised them faithfully, which means the query was interpreted wrongly before retrieval ran — the refund-window question from chapter 1 that fetched the renewal policy is exactly this shape. Fixes live upstream: rewriting the query into the corpus's vocabulary, decomposing multi-intent questions into separate retrievals, or asking the user a clarifying question when the top passages disagree about what was meant. Prompt engineering the generator cannot recover a question that was never understood, and a team that spends the quarter there will watch the metric not move.

The 31 cases that are grounded, relevant and still wrong are the ones worth escalating outside engineering. Reading twenty of them showed eleven where the corpus contained a superseded policy that had never been unpublished, six where two documents contradicted each other and the retriever surfaced the older one, and three where the eval label was wrong. Those three account for the whole sample with nothing unexplained, which matters because the third group is the one that is not a bug at all: a fifteenth of the class is the labelled set being wrong rather than the pipeline. The other two groups are content operations work and a corpus freshness policy, and no model change fixes them.

Retrieval changes that fix downstream symptoms#

A downstream diagnosis does not license cancelling all retrieval work, because recall is not the only retrieval-side number that affects generation. The previous chapter's reranker moved end-to-end accuracy by seven points without improving recall — it improved precision, and the failures it removed would have been classified as generation failures in exactly the table above, since a model anchoring on a distractor produces claims that are unfaithful or that answer a neighbouring question. A retrieval change can therefore be the correct fix for a downstream symptom, which is why the argument here is against the scope of the quarter rather than against retrieval work as such.

High recall also does not mean the passage was used. Recall says the gold passage was in the list; it says nothing about whether it was at rank 1 or rank 8, and a model reading 8 passages attends unevenly across them. A query can score recall 1.00 and have the supporting passage effectively invisible, which shows up in this taxonomy as a generation failure and is fixed by reranking. The isolation test for that is to rerun the failing queries with only the gold passages in the context: if accuracy on that slice jumps from 0.31 to 0.85, the evidence was present and buried, and the fix is ordering rather than model behaviour. What makes that test decisive is that it holds the generator, the question and the gold passages fixed and removes only the distractors, so whatever moves is attributable to what was sitting alongside the evidence.

And every number in this chapter is an average over the eval set's query mix, which the previous chapter's breakdown showed can hide a class going from usable to broken. The same slicing applies here — on this pipeline, end-to-end accuracy was 0.71 on single-passage queries and 0.44 on multi-passage ones, and the unsupported-claims class was concentrated in the latter, because a model asked to synthesise across three passages has more opportunity to bridge them with something it made up. Notice that the pooled 0.62 sits between those two and much nearer the single-passage figure, which is itself a statement about the traffic mix rather than about quality. A downstream diagnosis that is not sliced by query class will send you after the wrong 40%.

The inverse pattern is worth recognising too, because it appears on smaller corpora and means something quite different. A pipeline reporting context recall of 0.55 and end-to-end accuracy of 0.80 is answering correctly on queries where the evidence never arrived, which happens when the questions are about widely documented subjects the generator knows from pretraining. Users are being served, the corpus is barely load-bearing, and the system will keep producing the same answers after you update the documentation. Faithfulness is the metric that exposes it — on such a pipeline it typically sits between 0.4 and 0.6 while everything else looks healthy — and the decision it forces is whether the product is a retrieval system at all or a general assistant with a citation habit.

Running this on a schedule#

The five metrics this topic built are cheap enough to run continuously, and they earn their place in different loops. Faithfulness, answer relevance and citation accuracy run on every build over a fixed 200-question set, at a few dollars and under ten minutes, and each has a threshold that fails the build: faithfulness below 0.85, answer relevance below 0.70, citation accuracy below 0.70 with coverage above 0.90. Context precision and recall run on the labelled set whenever retrieval, chunking or the index changes, reported at the serving k and broken down by n_gold. The failure classification above runs once per release on the queries that failed, and its output is a table of four counts that goes into the release notes rather than a single score.

Two things happen quarterly rather than per build. The judges get re-calibrated against fresh human labels whenever a judge model or prompt changes, since an uncalibrated verdict is not evidence. And the labelled set gets re-sampled from recent logs, because a set built from last spring's traffic slowly stops describing what users ask, and the first sign of that is a set of metrics that stay flat while the support queue does not.

Two numbers locating a fault, then, and a classifier turning the location into work. The pair is not a diagnosis and cannot become one, because an aggregate knows nothing about which queries it is made of; what it does is tell you which half of the pipeline to spend the diagnosis on. The classifier supplies the rest by scoring every failing query on the metrics the earlier chapters built and filing it under exactly one owner, and the counts it produces are the thing to carry into a planning conversation, since a share of failures is an argument a plan can be checked against and a mean is not.

The starting point on a pipeline that has none of this is smaller than the list suggests. Take fifty answers from last week, decompose each into claims, read each claim against the passages that were retrieved for it, and count the answers containing at least one unsupported claim. That number, produced by hand in an afternoon, is the one that tells you which of these chapters you need first.