Per-step pass rates plus a fault label for every failure

When a pipeline has been sitting at the same number for weeks and each step's owner believes their step is fine, that is not a modelling problem and it will not be fixed by rewriting a prompt, because nobody can say which prompt. It is a missing instrument. The single end-to-end number is a product of everything the pipeline does, and a product tells you nothing about its factors.

The instrument has two parts. The first is a per-step pass rate, meaning: given the input this step actually received in this run, did it produce the correct output for that input. The second is a label on every failure recording whether the step failed on a good input of its own, or faithfully processed a bad one handed to it from upstream. The second part is what makes the first usable — a step that receives garbage and produces garbage has done nothing wrong, and a raw pass rate cannot tell that apart from a step that received a clean input and mangled it.

The equivalent instrument already exists for services#

Backend engineers have this for request paths: a distributed trace where every span carries a status, and an error is attributed to the span where it originated rather than to every span downstream that returned a 500 because of it. Nobody would debug a five-service request path by looking only at the status code the client saw. A prompt pipeline is a request path with the same topology and worse observability, because a bad intermediate is a syntactically valid JSON object rather than a stack trace, and every step downstream of it succeeds in the sense that it returns.

Computing "was the input good" requires gold data at every boundary rather than only at the end, gold being the answer a human has adjudicated as correct and recorded alongside the example. For a labelled sample you need the correct output of each stage for each sampled document, which is more annotation work than most teams have budgeted. It is also the work that unlocks everything else here, and 300 fully annotated documents will tell you more than 30,000 with only a final label.

The cheap way to collect it is to annotate by correction rather than by authoring. Run the pipeline, show an annotator each intermediate next to the source document, and ask them to fix what is wrong rather than to write the correct intermediate from nothing. Most intermediates are already right, so most rows cost a glance, and the ones that take real time are exactly the ones you wanted a human to look at. The by-product is that every correction is a labelled failure with a location already attached to it, which is half of what this chapter is asking you to build.

Four labels, and what each one tells you to do#

{
  "run_id": "r-8841",
  "doc_id": "d-10237",
  "steps": [
    {"name": "extract",  "input_ok": true,  "output_ok": true,  "fault": null},
    {"name": "classify", "input_ok": true,  "output_ok": false, "fault": "own"},
    {"name": "policy",   "input_ok": false, "output_ok": false, "fault": "inherited"}
  ],
  "end_to_end_ok": false,
  "origin_step": "classify"
}

input_ok compares the step's actual input against the gold intermediate for that document; output_ok compares its output against the gold output for the input it received, which is the distinction that makes the labels mean anything. From those two flags the label follows:

amplified and inherited share the same two flags and are separated by a judgment about the output: an inherited failure carries the upstream defect forward in a form that is still visibly wrong, while an amplified one converts a recoverable defect into an unrecoverable one. A policy step that receives a mis-extracted amount of £5,400 instead of £540 and escalates it for manual review has inherited a failure; one that auto-approves it because £5,400 falls inside a different approval band has amplified it. The distinction costs an annotator a few seconds per failure and it is the one that tells you where to put a schema check.

recovered is worth counting even though it is rare, because a step with a meaningful recovery rate is doing work nobody has credited it with, and someone will eventually propose deleting it.

Reading a filled-in table#

Instrumented over a 4,000-document sample of a three-stage expense pipeline — extract the line items, classify each line into a category, apply the approval policy — with each failure attributed to the step where it originated:

The pass rate that matters for each step is its own-fault rate on clean inputs: extraction 90.5%, classification 93.4% of the 3,620 clean extractions it received, policy 97.0% of the 3,380 clean classifications. Those three multiply to 0.820, which is the 82% the pipeline reports, and the fact that they multiply out correctly is itself a finding — it says the pipeline is losing roughly what independent steps at those rates would lose, with no large unexplained gap.

Every number in that sentence comes out of the table by subtraction and division, and doing it once is what makes the rest of the chapter usable. Extraction ran on 4,000 documents and owns 380 failures, so it was right on the rest, which is the 90.5%. Those 380 are also what it did not hand on: 4,000 minus 380 is the 3,620 clean extractions classification received. Classification owns 240 failures out of those 3,620, which is the 93.4% — and note that the rest of its 448 failures are not in that fraction at all, because they were not its doing. Subtracting its 240 own-faults from 3,620 gives the 3,380 clean classifications the policy step received, of which it owns 100, which is the 97.0%. Multiply the three and the answer is 0.820.

The denominators are the whole design, and they are where a team that builds this instrument badly goes wrong. Each step is scored against the population it was actually given rather than against the 4,000 documents that entered the pipeline, because a step cannot be held responsible for inputs it never had a chance with. Score every step against 4,000 and the three rates stop being independent factors, stop multiplying out to anything, and start reporting the first step's failures three times over.

Read down the table and the ranking inverts. The policy step has the most failures by a wide margin and 486 of its 620 are inherited, so its prompt is the last thing anyone should touch; it is at 97% on inputs it deserves. Classification holds the largest single block of own-faults at 240. Extraction owns 380 failures outright, every one of which also costs whatever the downstream steps do with them.

The belief that inverts is worth stating in the form somebody would defend it, because holding it is not laziness. It is that the step with the most failures is the step with the most to gain. It is tempting for three good reasons: failure counts are what an error dashboard naturally surfaces, every one of those failures is real and somebody downstream saw it, and the owner of that step is the person who has been paged about it. What makes it wrong is that a failure count answers "where did the pipeline notice this" while the planning question is "where did it start". The two coincide only in a system with one step. In a chain, the final step accumulates everything every earlier step got wrong, which is exactly why the count is highest at the end and the leverage is highest at the front.

What a fix is worth before anyone attempts it#

Own-fault rates convert directly into an upper bound on what fixing each step is worth, which is the calculation that turns the table above into a planning conversation. Take the three own-fault pass rates and recompute the product with one of them replaced:

Each row is the same product with one factor swapped, which is worth watching once. Replace extraction's 90.5% with a one and multiply what is left — 93.4% times 97.0% — and the answer is the 90.6% in the first row. Replace classification's factor instead and 90.5% times 97.0% gives 87.8%. Replace the policy step's and 90.5% times 93.4% gives 84.5%. The right-hand column halves an own-fault rate rather than removing it: extraction's own-fault rate is the complement of 90.5%, so halving it puts the step a little over ninety-five in a hundred, and running that through the same product gives the 86.3%.

Read as a ranking, those rows say something the failure counts do not. The gain from fixing a step is bounded by how much of the pipeline's loss that step originates, and that is its own-fault rate — not its failure count, which includes everything handed to it. The policy step is last in this ranking because it originates almost nothing, despite topping the failure count by a wide margin.

The three perfect-step gains sum to 16.9 points against 18 points of total loss, and the missing 1.1 is the interaction term: documents that fail at two steps at once are counted twice in the column and once in the pipeline. The right-hand column is the one to plan against, because halving a step's own-fault rate is an ambitious but recognisable outcome for a sprint of prompt work, while perfecting a step is not an outcome anyone has ever delivered.

Both columns are upper bounds rather than forecasts, and they are bounds in a specific direction. They assume that fixing a step's own faults does not change its behaviour on the inputs it was already handling, which a prompt rewrite absolutely can violate — a new category definition that recovers 40 misclassifications may quietly break 12 that were passing. That is what the pre-change and post-change per-step numbers are for.

The sample size that supports this arithmetic is smaller than the annotation cost suggests, and worth checking rather than assuming. At 4,000 documents the standard error on a per-step rate near 90% is under half a percentage point, so a 4.3-point counterfactual gain sits far outside the noise and the ranking between the three steps is not in question. At 300 documents the same standard error is about 1.7 points, which still separates extraction's +4.3 from policy's +1.3 comfortably and would not separate two steps that were close. Size the annotation against the decision you want to make: ranking three steps that differ by points needs a few hundred rows, and tracking a single step's drift week over week needs the larger sample or a production proxy.

Both of those standard errors come out of one expression, and it is the one that decides how much annotation to buy. Multiply the rate by one minus the rate, divide by the number of documents, take the square root. At nine in ten over four thousand documents the multiplication gives nine in a hundred, dividing gives a little over two in a hundred thousand, and the square root of that is the sub-half-point figure. Swap four thousand for three hundred and the division produces a number more than ten times larger, so its square root is a bit over three times larger, which is the 1.7. That relationship is the one to hold onto: cutting the sample tenfold costs only a bit over threefold in precision, which is why a few hundred well-annotated documents go further than the annotation budget makes it feel like they should.

Wiring it in without building an eval framework#

The instrumentation is smaller than it sounds, because it is a wrapper around each step and a comparison function per boundary.

def run_instrumented(doc, gold):
    trace, current, upstream_ok = [], doc, True

    for step in (extract, classify, policy):
        out = step(current)
        gold_out = gold.expected(step.name, given=current)   # gold for THIS input
        output_ok = matches(out, gold_out)
        trace.append({
            "name": step.name,
            "input_ok": upstream_ok,
            "output_ok": output_ok,
            "fault": label(upstream_ok, output_ok, out, gold.ideal(step.name)),
        })
        upstream_ok = upstream_ok and output_ok
        current = out

    return trace, current

gold.expected(step, given=current) is the part that carries the design. It asks what the correct output is for the input the step actually received, which for a bad input is often "the same defect, faithfully carried" rather than the ideal answer. Teams that skip this and compare every step against the ideal output collapse own and inherited into one bucket and end up back where they started, blaming the last step in the chain for everything the first one did.