Context precision and recall need a per-passage relevance label

The metrics so far in this topic all read the generated answer. Faithfulness compares its claims against the passages, answer relevance compares it against the question, and citation accuracy compares each claim against the passage it names. None of them can say whether the passages themselves were the right ones, because none of them has anything to compare the passages against.

The two that can are context precision, the share of what was retrieved that is relevant, and context recall, the share of what is relevant that was retrieved. Both need a third artifact that no part of a pipeline produces: for each question, a list of the passages a person judged capable of supporting the answer. That labelling job is the one with the four pathologies from the previous chapter, and the numbers in this chapter come from the rebuilt set rather than the flattering one.

The change under review is one config edit and forty lines: retrieve 20 candidates as before, score them with a cross-encoder reranker, a second model that re-scores each candidate against the query and reorders the list, and pass the top 5 to the generator instead of all 20. It moved three numbers.

One engineer read that as an obvious ship: seven points of end-to-end accuracy for a quarter of the context budget and 180 ms. The other read the recall line as a blocker, on the grounds that a missing passage is a wrong answer and five points of recall means the system has stopped being able to answer some questions at all. They are looking at the same four rows and neither is being unreasonable, because the rows do not by themselves say which queries moved.

Both numbers rest on a label somebody wrote#

Context precision and context recall are set comparisons over the retrieved list, and the set they compare against is the relevance labels for that question. For question q with gold set G(q) — the passages a person judged capable of supporting the answer — and retrieved list R(q) truncated at k, precision is the fraction of what you retrieved that is relevant, and recall is the fraction of what is relevant that you retrieved.

def context_scores(retrieved: list[str], gold: set[str], k: int) -> dict[str, float]:
    top = retrieved[:k]
    hit = set(top) & gold
    return {
        "precision": len(hit) / len(top) if top else 0.0,
        "recall": len(hit) / len(gold) if gold else float("nan"),
        "n_gold": len(gold),
    }

Neither line of arithmetic is interesting; the label behind gold is where the metric is made or ruined, and it is a judgement someone has to write down. Nothing in the pipeline can produce it, because a retriever's own similarity score is its opinion about relevance and using it as the label makes the metric a tautology.

The token row is the one place where the table checks itself, and it is worth doing before trusting anything else in it. Divide 9,100 by the 20 passages that produced it and each passage runs a little over four hundred and fifty tokens. Divide 2,280 by 5 and each passage runs a little over four hundred and fifty tokens again. The two rows are the same passage size counted at two different values of k, which says the reranker changed how many passages arrive and not which kind, so nothing in the movement of the other rows is a side-effect of the prompt filling up with longer or shorter material.

The precision jump needs more care than the token row, because the arithmetic that appears to explain it does not survive being checked. Questions in this set have 3.6 relevant passages on average. Reading the precision column as the flat ratio context_scores computes, 0.31 at k=20 means 6.2 relevant passages among the top 20, and 0.78 at k=5 means 3.9 relevant among the 5. Both counts are larger than 3.6, which is the average number of relevant passages that exist, and a retrieved set cannot contain more relevant passages than the labels admit.

The bound is worth deriving rather than asserting, because it is what rules out the obvious rescue. For one question, the flat precision is the number of relevant passages retrieved divided by k, and the number retrieved cannot exceed the number that exist, so that question's precision is at most its own gold-set size over k. Average both sides over the set: the mean precision is at most the mean gold-set size over k, so the maximum achievable value here is 3.6 over 20, or 0.18. The rescue people reach for is skew — a few questions with very large gold sets pulling the average up — and it does not work, because the inequality was averaged before the comparison. Skew changes which questions can reach a high precision; it cannot lift a mean above a ceiling that was computed from the mean.

So the tabulated precision column is not flat precision@k over these labels, and the three definitions in circulation, set out in the next table, say what else it can be. The rank-weighted definition, which is what the Ragas library ships as context_precision, averages the precision measured at each rank where a relevant passage sits, so a single relevant passage at rank 1 scores 1.0 on a question with one gold passage. That definition has no n_gold / k ceiling, and 0.31 and 0.78 are unremarkable values under it. The counts of 6.2 and 3.9 are what those figures would mean if the column were the flat ratio, and they are the signal that it is not.

What survives is the part the decision rests on. Under either definition the column moved sharply upward, and under either definition it is not comparable across different values of k — the flat one because its ceiling is n_gold / k, the rank-weighted one because truncating a list changes which ranks contribute. That non-comparability is the most common way this metric is misreported, and the second most common is quoting it without saying which of the definitions produced it.

The third is the one teams adopt when labelling feels too expensive, and it changes the metric's character: it is now a judge verdict with its own error, calibrated or not, rather than a set operation. It is genuinely useful on production traffic where no labels exist, and it is not comparable to a labelled score computed on the eval set.

What the three movements mean together#

Precision rising sharply while recall slips slightly and end-to-end accuracy rises is the signature of a noise-for-coverage trade that came out favourably. The reranker discarded fifteen of twenty passages; almost all of what it discarded supported nothing, and a small amount of genuine supporting evidence went with it. The generator, given a clean five, produced better answers than it did with a diluted twenty — which tells you the previous context was hurting rather than merely costing tokens.

That effect has a mechanism, and the per-passage size from the token row makes it a quantity rather than an impression. Fourteen irrelevant passages at roughly 450 tokens each are 6,300 tokens of plausible, on-topic, wrong material sitting alongside the evidence — well over half the 9,100-token prompt. Passages retrieved by a dense retriever, which matches on embedding similarity rather than on shared words, are irrelevant in the most dangerous way available: they are about the same subject. A passage describing Enterprise-plan retention next to one describing Business-plan retention is a distractor the model can anchor on, and the chapter-1 quadrant has a name for the result — an answer faithful to the wrong passage, which scores 1.00 on faithfulness and fails the user. Cutting the set to five removes most of those opportunities, and that is the seven points.

The trade holds until the lost recall lands somewhere concentrated. Recall of 0.86 is an average over queries, and the five points that disappeared were not distributed evenly.

Splitting recall by how many passages the question needs#

The single breakdown that settles the disagreement is recall grouped by n_gold, the number of passages a question's answer requires. On this eval set:

The three class sizes add to four hundred, so this table and the aggregate one above describe the same run partitioned three ways rather than two different experiments, and that lets the aggregate be reconstructed as a weighted average. Doing so is how you find out how much of the aggregate is a mixture effect. Weighting the before-recall column by the three class sizes gives 0.91, which is what the aggregate reports, to the last digit. The same weighting on the end-to-end columns lands within a point of both aggregate figures. The after-recall column is the one that does not reconcile: the mixture comes out close to nine tenths against a tabulated 0.86, a residual of about three points that has to live somewhere in the class rows and cannot be located from what is printed. Three of the four reconstructions closing is enough to read the aggregate as a mixture; the fourth is a real discrepancy and is worth saying so rather than rounding past.

Read as a mixture, the aggregate gain of seven points is carried entirely by the 232 single-passage queries, which are the majority and which the reranker helps a great deal. The 64 multi-passage queries went from mediocre to bad, losing eighteen points of recall and eleven points of end-to-end accuracy, and no aggregate would have shown that — a five-point average drop and a concentrated eighteen-point collapse in a sixth of the traffic are the same number once they are averaged, which is precisely why the average could not settle the argument.

The cause is in how a cross-encoder reranker scores. It takes one query and one passage and produces a relevance score for that pair, independently of the other candidates. Follow that independence to its consequence: a passage that is individually a weak match but jointly necessary — the one carrying the second half of a two-part answer, phrased in vocabulary the question does not use — is scored without any knowledge that its partner has already been selected, so it ranks below several individually strong passages that all say the same thing as each other. Reranking to a small k therefore systematically favours redundant coverage of the strongest interpretation of a query over complementary coverage of the whole of it, and the more passages a question needs, the more of its gold set falls off the end. Questions like "what are the retention limits on each plan, and which ones can be extended" are exactly the shape that breaks.

The limits of both numbers#

Neither metric knows anything about position within the k it scores. Flat precision treats a relevant passage at rank 1 and at rank 5 identically, and long-context attention does not, which is why the rank-weighted variant exists and why a reranker can improve answers without moving flat precision at all.

Neither is a substitute for an end-to-end number, and the reverse is also true. A change that improved precision from 0.31 to 0.78 while leaving end-to-end accuracy flat would have told you the distractors were not the binding constraint, which is a useful thing to learn and only visible because both were measured. Topic 3.14's point about component and end-to-end evaluation is doing its work here: the component numbers explain a movement in the end-to-end number, and the end-to-end number decides whether the component movement mattered.

The relevance label itself is binary, and relevance is not. A passage that mentions the retention period in a footnote and a passage that is the retention policy are both relevant under a binary label, so a retriever that finds only footnotes scores the same as one that finds the policy. Graded labels and nDCG — normalised discounted cumulative gain, which rewards a ranking for putting the most relevant passages nearest the top — exist for this and cost roughly twice as much to produce; most teams live with binary labels and accept that the score is coarse, which is defensible as long as nobody reads a two-point movement as meaningful.

Two set comparisons, then, both resting on a label nothing in the pipeline can supply, and both reported as means over a query mix. The means are worth having: they say a trade happened and roughly how large it was. What they cannot say is which questions paid, and since a reranker's failure mode is concentrated in exactly the class of question that needs several passages at once, the breakdown by gold-set size is not an optional refinement. It is the difference between a change you can defend and a change that quietly removed a capability.

After all of this the retriever on this pipeline is in reasonable shape: recall at 0.86 overall, 0.96 on the single-passage majority, and precision no longer diluting the prompt with fourteen distractors. End-to-end answer accuracy is 0.71, and on a different pipeline with recall at 0.94 it sits at 0.62 — which is a gap that no amount of further retrieval work can close, and a sprint scoped as "improve retrieval" is about to be spent on the wrong component.