Measurable stage boundaries

A seam is a place in a pipeline where a nameable artifact crosses from one stage to the next, and a measurable seam is one where that artifact can be called right or wrong. The distinction matters because architecture diagrams present every box as equivalent, and a plan that allocates two days per box will spend some of those days building numbers that cannot move. Such a number is not obviously broken: it renders on a dashboard, it has a plausible value, and it survives review because nobody has a reason to doubt it.

The three questions to put to a seam before instrumenting it assume a prior step of their own — that the boxes in the diagram are the seams — which was false twice on the five-stage retrieval pipeline described here.

A number that cannot move#

The query-rewrite eval took four days to build and produced a number that has never moved. It works the way you would expect: 200 logged questions, the rewriter runs on each, a judge model reads the original and the rewrite and rates how well does this rewrite preserve intent while improving searchability, 1 to 5. It reads 4.4. It read 4.4 before the rewriter's prompt was rewritten, it read 4.4 after, and it read 4.4 on a control run where the rewriter was replaced with a function that returns its input unchanged.

The reason is visible in any single item.

original: "why did my export fail last night"

rewrite A: "export job failure causes"
rewrite B: "troubleshooting failed data export scheduled run"
rewrite C: "why did my export fail last night"

A is tighter and drops the temporal hint. B adds vocabulary that may or may not appear in the indexed documents. C changes nothing, which is the right move if the retriever is a hybrid that already handles the phrasing. There is no fact of the matter about which is correct, and a judge asked to rate them will produce numbers in the low fours for all three, because all three are reasonable and none is checkable. Two human annotators asked to pick the best of the three agreed on 61% of a hundred items, which is barely above what you get from tossing a coin between the two they were most drawn to.

The stage has everything an instrumentable boundary appears to need. It has a clear input, a clear output, a log line, a latency, its own prompt file and its own owner. What it does not have is a definable correct output, and without one, every number computed at that boundary is a number about the judge.

The reason a definable correct output is the load-bearing requirement, rather than one item on a checklist, comes out of what a metric is. A metric compares what the stage produced against what it should have produced and reports the size of the gap. That comparison needs two operands. Retrieval supplies the second one from labels: the correct output for this question is a list containing these chunk IDs, written down by a person before the run. Query rewrite supplies nothing, because no person can write down the correct rewrite of "why did my export fail last night" — A, B and C above are all defensible and no procedure decides between them. A judge asked to score with the second operand missing does not refuse. It substitutes the only comparison available to it, which is against its own sense of what a reasonable rewrite looks like, and it reports how reasonable the rewrite looked. That reading is a genuine measurement of something; the something is the judge.

The control run is what turns that argument from a plausible story into a settled one, and it works the same way the pair of traces in chapter 1, on end-to-end and component metrics, worked. An instrument that returns the same reading for the rewriter and for a function that does not rewrite is returning one value for two very different states of the world. Whatever quantity it is tracking, that quantity does not change when the stage changes, and a number that does not change when its subject changes cannot be used to decide anything about its subject. This is also why the control belongs at the start: replacing the stage with a function that does nothing is the cheapest available test of whether a stage is evaluable at all, and it can be run before the four days as easily as after them.

The three questions to ask of a seam#

Before spending a sprint instrumenting a stage, put it through three questions in order. A stage has to pass all three for a component metric on it to mean anything.

The first is whether the input can be specified independently. A valid input to this stage has to be constructible without running the stages before it, because otherwise the stage cannot be evaluated in isolation and its metric will move whenever anything upstream changes. Retrieval passes easily, since a question is a question. Reranking passes only if you freeze the candidate list you feed it, which is a real constraint discussed below.

The second is whether the output admits a correctness verdict. Given the stage's output, can someone — a person, a script, an entailment judge — say this is right or this is wrong, without appealing to what happens downstream? Retrieval passes because "does the gold chunk appear in this list" is a set membership question. Query rewrite fails, as above.

The third is whether ground truth is obtainable at that point at a cost you will actually pay. Sometimes the correct output is definable in principle and unobtainable in practice: the correct chunking of a 12,000-page corpus is definable if you are willing to have an expert mark every fact boundary in it, which nobody is. Sometimes ground truth is free because the stage's own input supplies it — groundedness needs no labels beyond the context that was passed in, which is why it is the cheapest useful generation metric there is.

The five-stage pipeline, scored against the test#

Take the pipeline as inherited: query rewrite, retrieval, rerank, generation, citation formatting, with one end-to-end pass/fail sitting at 71%.

Three of the five carry a real metric, one carries a conditional metric, and the stage that looks most like the interesting AI work is the one that cannot carry a score at all. That ratio is normal, and it is the answer to the question of what to build: you do not instrument five boundaries because the diagram has five boxes.

The rerank row deserves the extra sentence. Its metric is conditional because its input is the retriever's output, so recall@5 measured after reranking moves when the retriever changes, when the reranker changes, and when the candidate depth changes. Scoring the reranker means pinning the candidate list — snapshot 50 candidates per question from a fixed retriever build, store them, and replay the reranker against that frozen file. Then the number is about the reranker. Without the freeze you have a joint metric of two stages wearing one stage's name, which is the composite problem from chapter 1, on end-to-end and component metrics, reproduced one level down.

Citation formatting is the opposite kind of surprise: it looks like plumbing and it is the easiest thing in the pipeline to check properly, because most of its correctness conditions are structural and need no model at all.

def check_citations(answer_text, cited_ids, context_ids, context_text):
    failures = []
    for cid in cited_ids:
        if cid not in context_ids:
            failures.append(("phantom_citation", cid))       # cites a chunk never supplied
    for sent in split_sentences(answer_text):
        ids = extract_citation_markers(sent)
        if makes_factual_claim(sent) and not ids:
            failures.append(("uncited_claim", sent[:60]))
    if not set(cited_ids) & set(context_ids[:1]):
        failures.append(("top_chunk_uncited", context_ids[0]))
    return failures

Every check in that function is a question about membership or presence, which is why none of them needs a model. A phantom citation is an ID in the answer that is not in the list of IDs supplied — two lists, one comparison. An uncited claim is a sentence that makes a factual assertion and carries no marker, and the only part of that needing judgement is makes_factual_claim, which most teams implement as a crude heuristic on purpose because a crude heuristic that runs on every build beats a careful judgement that runs quarterly. The ground truth here is free because it is the stage's own input: the context that was supplied is the standard the citations are checked against, and it was already in the request.

Running that over 400 answers takes under a second and, on the pipeline this chapter is describing, turned up phantom citations on 2.3% of answers — chunk IDs the model had produced by pattern-matching the format rather than by copying an ID that was present. That is a real defect, found for no inference spend, on the stage nobody thought was worth measuring.

The boxes in the diagram are not necessarily the seams#

There is a prior step that the table above quietly assumes: that the five named stages are the boundaries. Architecture diagrams are drawn along ownership and module lines, and measurement boundaries follow artifacts — the places where a nameable object exists, could be written to disk, and could be swapped for a different one. The two coincide often enough that nobody checks, and the cases where they diverge are where the interesting defects hide.

On this pipeline they diverged twice. The chunker and the embedder are separate modules with separate owners and separate configuration, and they share one boundary, because the only artifact that leaves them is the index; you cannot evaluate chunking without embedding it and searching it, so there is one probe point, not two. In the other direction, prompt assembly is not a stage in the diagram at all — it is a template call inside the generator — and it materialises an artifact that everything in this topic depends on, namely the context as it finally appears in the model's input. Treating "the five reranked chunks" and "the context the model read" as the same object is an assumption, and on this pipeline it was false: a 4,000-token input budget was truncating the assembled prompt, and on questions where the earlier chunks were long, the fifth chunk was being dropped before the call. Recall@5 measured on the reranker's output read 0.79. Recall measured on the text actually sent read 0.74, and the five points of difference were a silent truncation nobody had a metric for.

The mechanism there is worth following because it is entirely mundane and entirely invisible. The generator has a fixed input budget, and the assembled prompt is a system instruction plus five chunks plus the question. Chunks are not all the same length. On a question where the first few chunks run long, the sum of those parts exceeds the budget, and something has to be dropped; what gets dropped is whatever sits at the end, which is the fifth chunk. Every stage involved behaved correctly. The reranker emitted five chunks and its own metric scored those five. The generator received four and answered from four. Recall@5 was being computed on an artifact — the reranker's output list — that had stopped being the same object as the artifact it was supposed to stand for, which is the context the model read. The two had been the same object right up until the day somebody set a budget, and nothing announced the day they stopped.

This is the general form of the trap, and it is why the enumeration below asks about artifacts rather than about stages. A metric names an artifact implicitly and is computed on whatever the code hands it, and those two can drift apart without any stage misbehaving. The question that catches it is whether the object being scored is the object being consumed downstream, asked of each probe separately.

The way to find the real seams is to ignore the boxes and enumerate the artifacts, asking of each one whether it could be serialised, stored, and fed to the next step from a file. On this pipeline that produces the raw question, the rewritten query, the index, the candidate list, the reranked list, the assembled context, and the answer with its citations. Each artifact that can be materialised is a place a probe can sit and a place a replay can start, and every stage boundary that does not produce a materialisable artifact is a boundary you can only measure by ablation, meaning by running the pipeline with the stage switched off and reading what changes downstream.

That enumeration is also what protects two weeks of work from being spent on the wrong five things. A plan that instruments the five named stages will build a query-rewrite metric that returns a constant and will never measure the truncation, because truncation happens inside a box rather than between two of them.

Measuring a stage by its effect#

Failing the test is not a reason to remove the stage or to stop caring about it; query rewriting may well be the reason recall is as high as it is. It is a reason to measure it at the first downstream boundary that does have a definable correct output, with the stage switched on and off.

def rewrite_ablation(questions, retrieve, rewrite, k=5):
    rows = []
    for q in questions:
        raw  = recall_at_k(retrieve(q["text"]),            set(q["gold"]), k)
        rw   = recall_at_k(retrieve(rewrite(q["text"])),   set(q["gold"]), k)
        rows.append((q["qid"], q["intent"], raw, rw))
    return rows

That is not a component metric on the rewriter; it is a contrastive end-to-end metric on the prefix of the pipeline ending at retrieval, and it answers the only question about the rewriter that anyone actually needs answered — whether it helps. The general shape is that unmeasurable stages get evaluated by their effect, and the cost of that is that ablation requires a full re-run of the downstream prefix per variant, so it does not go on a nightly dashboard the way a stage metric does. It is a decision-time tool rather than a monitoring tool.

What licenses the attribution is worth stating, because it is the only thing holding the argument together. The two recall figures in each row come from runs that are identical in every respect except one: the same question, the same index, the same retriever, the same labels, the same k. The rewriter is applied in one and not in the other. Any difference between the two numbers therefore has exactly one place it can have come from. That is not a property of ablation as a technique — it is a property of this particular pair of runs, and it evaporates the moment a second thing changes between them. An ablation run against a rebuilt index, or with a different k on the two sides, measures the sum of two effects and reports it as one.

The intent field in that tuple is not bookkeeping either. An overall difference is an average, and an average over a mixture can be the sum of movements in opposite directions, which is the failure this whole topic keeps circling. A rewriter that helps one kind of question and harms another produces a single positive number and a decision to keep it, and the harm stays in the product unnamed. Carrying the intent through to the rows costs one column and makes the opposite movements separable afterwards, which is what turns "keep it" into "keep it except here".

Two things that have to exist first#

Everything above assumes two conditions that are easy to skip past. The first is labels attached at the boundary being measured, in a form that survives the changes you are going to make — the offset-based gold spans from chapter 2, on the stages of a RAG pipeline, rather than raw chunk IDs. The second is that the stage's input is recorded per request: if your logs contain the question and the final answer but not the fifty candidates or the five chunks that reached the model, every metric in the table above is uncomputable after the fact.

There is also a case for building none of this. A single-stage pipeline has no boundaries worth naming, a prototype whose stages change weekly will invalidate its labels faster than it can collect them, and a system whose end-to-end score already meets the product bar has no attribution problem to solve. Chapter 7, on sequencing which component eval to build first, puts numbers on when the investment pays.

The test is cheap to apply and it changes what gets built. Ask whether the input can be constructed alone, whether the output can be called right or wrong on its own terms, and whether the ground truth for that verdict is affordable; three questions, asked before a sprint is allocated. On a five-stage pipeline the answers came back as three clean metrics, one conditional on a frozen candidate list, and one stage measured by ablation instead — plus two seams that were not boxes at all, one of which was quietly dropping the fifth chunk of every long context.

What the test does not say is which of the measurable seams to score first, or what to do with a single failing question in front of you right now. Both need the request's own record, which is where chapter 5, on attributing a failure to a stage, begins.