Component vs end-to-end evaluation

Read these first

• Error analysis

What you will know

• Score retrieval and generation separately in a RAG pipeline

  A fluent answer can conceal weak evidence selection, while good retrieved passages can be discarded by generation; use stage-appropriate metrics and labels.

• Locate which stage caused an end-to-end failure

  Trace the request, retrieved context, intermediate transformations, and final output so the failure can be assigned to a specific boundary rather than guessed from the final score.

• Run an end-to-end eval alongside component evals and show a case where the aggregate score hides a stage failure

  Averages can remain stable when one stage regresses and another improves, masking failures concentrated in particular query or document slices.

• Build a component eval for the weakest stage first

  Start with the cheapest high-signal test, such as retrieval recall against labeled relevant passages, before spending inference budget on full answers.

• Resist fixing the prompt when the retrieval was empty

  No wording change can recover evidence that was never supplied, and prompt workarounds often increase unsupported or hallucinated answers.

Chapters

1. 1

   End-to-end and component metrics

   An LLM-judge helpfulness average reading 4.1 out of 5 over 300 questions, unmoved for a month while support escalates three confidently wrong answers, comes out the same whether retrieval never surfaced the right passage or generation ignored a passage sitting in its context. Covered here: what an end-to-end metric and a component metric each measure, why a score taken at the end of a chain cannot be inverted onto any link in it, what each of the two can and cannot tell you, and the reasoning error of letting a flat composite overrule a component signal.

   Read chapter
2. 2

   The stages of a RAG pipeline

   Stages cannot be scored separately until they have been named. A corpus somebody cut into chunks is searched for a question, a ranked list comes back, a reranker may reorder it, and the top k lands in the generator's context. Covered here: that path artifact by artifact, the two configuration choices that move retrieval most — how the corpus was chunked and whether a reranker sits in the path — the mechanism by which a chunk boundary destroys a retrieval result, and where the labelled relevant passages every retrieval metric is scored against come from.

   Read chapter
3. 3

   Retrieval recall and answer groundedness

   A fluent answer can conceal evidence selection that missed the point, and a correct passage can sit in the context and be ignored. These are the two stage-appropriate metrics that make those cases distinguishable: recall@k, a set comparison against labelled relevant passages that needs no model call, and groundedness, a judge-computed check that an answer stayed inside the context it was given. Covered here: both definitions, how to calibrate the judge that produces the second, the three scores they get confused with, and the four cells a failing question can fall into.

   Read chapter
4. 4

   Measurable stage boundaries

   Query rewrite has an input, an output and a log line, and still cannot carry a score, because nobody can say what the correct rewrite of a question is. Covered here: the three questions a seam has to pass before an eval built on it means anything, the five-stage pipeline scored against them, why the boxes in an architecture diagram are not necessarily the seams, how to measure an unmeasurable stage by its effect on the first downstream boundary that is measurable, and what has to exist before any of it is worth instrumenting.

   Read chapter
5. 5

   Attributing a failure to a stage

   One question fails and the instinct is to open the prompt template. Attribution replaces the guess with a lookup: confirm from the trace whether the gold passage reached the model's context, then re-run with only that passage present — if the answer comes out right, the problem is distraction or ordering rather than wording. Covered here: the four artifacts a trace has to record for any of it to be possible, what the storage costs, how to replay a trace faithfully, how to run the procedure nightly over a whole failing set, and where a boundary assignment stops being a root cause.

   Read chapter
6. 6

   Prompt fixes and missing evidence

   Appending 'if the provided context does not contain the answer, say you don't know' converts a wrong answer into an abstention, which is safer. On the 40% of failures where retrieval returned zero chunks it answers nothing: the question is still unanswered and the retrieval bug now hides behind a polite refusal. Covered here: what the abstention line changed and what it cost, the empty-context branch that does the thing wording cannot, why deployment friction rather than evidence decides where the work goes, what a three-week prompt sprint can buy at recall 0.58, and the failures where prompt work is the correct fix.

   Read chapter
7. 7

   Sequencing component evals

   Recall@k against labelled relevant passages needs no generation call at all, which makes it the first thing to build when retrieval is the stage you suspect. Covered here: the build cost and decision value of five candidate evals side by side, why one credible eval beats five thin ones, how to spend the day and a half that the labelled set costs, how to predict what lifting recall does to the end-to-end score and what would falsify the prediction, and when to argue against component evals altogether.

   Read chapter
8. 8

   Cancellation in an aggregate score

   End-to-end score 78 to 79 across two releases, while retrieval recall@5 falls from 0.81 to 0.66 and groundedness rises from 0.72 to 0.88, and the release notes say 'quality improved slightly'. Covered here: the mechanism that produces exactly that pattern, the single slice that confirms or kills it, why netting two metrics on different scales repeats the composite's error, the four reporting changes that stop the same sentence being written again, the cadences the two kinds of eval run at, and why end-to-end evaluation still earns its cost once component evals exist.

   Read chapter