Pass rate against your own stopping signal rises by construction

The verification pass rate is the easiest number to put on a slide, because it comes free with the loop: the counter increments whenever the verifier returns an empty problem list, which is the same event that ends a run. It is also the number a retry loop is designed to raise, and a number a system is designed to raise is a poor witness on its own behalf. The subject here is how much of a rise is mechanical, and what has to sit beside the rate before anybody can claim the system got better.

The running system is the claims extractor of the previous chapters, which turns scanned insurance claims into structured records and re-prompts until a constraint list accepts the output. Its pass rate went from 71% under a single pass to 96% under the loop, on the same production counter, and neither figure is wrong. What follows is what they mean.

The arithmetic that makes a rising pass rate uninformative#

Start with the simplest possible model of a retry loop and see how far it gets. Suppose a single attempt passes with probability p, and suppose for the moment that attempts are independent, meaning that failing one tells you nothing about the chance of failing the next.

One attempt fails with probability 1 − p. Two independent attempts both fail with probability (1 − p) × (1 − p), because independence is exactly the condition that lets you multiply the two chances. Extending that, k independent attempts all fail with probability (1 − p)^k. The loop passes if at least one attempt passes, which is the complement of all of them failing, so

pass rate with k permitted attempts  =  1 − (1 − p)^k

Every term in that line has a meaning: p is what the generator does on its own, k is a number somebody chose, and the pass rate is what gets reported. At p = 0.71, the formula gives 91.6% at two attempts and 97.6% at three. A measured 96% therefore sits between two and three permitted attempts, and — this is the part that matters — it would have appeared for any generator with a 71% first-attempt rate, regardless of whether the outputs were good, because nothing about output quality appears anywhere in the formula. The reported number is a function of p and k, and it converges to 100% as k rises for the same reason that a coin flipped enough times eventually comes up heads.

Attempts are not independent in practice, and it is worth knowing which way the assumption fails, because a wrong model is only useful once you know its direction of error. Documents that fail the first attempt are systematically the harder ones — that is the selection effect the previous chapter used to explain the latency tail — so the conditional pass rate declines attempt over attempt rather than staying at p. Multiplying equal chances therefore overstates how quickly failures get cleared, which means the independence formula is a ceiling on what retries can achieve rather than a prediction of what they will. The claims extractor's measured curve bears that out: 93.8% at three attempts against the formula's 97.6%, and five attempts needed to clear 96%.

That ceiling is still the useful object, because the argument the slide is making only needs the ceiling to be demolished. Reading a measured 96% against the real curve says the loop behind it was permitting four to six attempts, which by the attempt-bucket defect rates of chapter 4 is precisely the region where roughly half of the passing outputs are wrong. A number that can be produced by raising k alone cannot distinguish a system that improved from a system that was allowed to try more times.

What "an improvement" would have to consist of#

Four measurements together can support the claim, and no one of them can do it alone.

The attempts distribution comes first because it is free and it prices everything else. A histogram of attempts per completed request, segmented by input class, tells you what fraction of the pass rate is first-attempt work and what fraction was bought with retries — which is exactly the k in the formula above, measured rather than assumed. The single summary worth putting on a dashboard is the share of delivered outputs that took three or more attempts, since that is the share that needs reading rather than counting.

Latency is the delta at the percentiles the callers care about, reported as p50, p95 and p99 rather than as a mean, since the whole effect of a retry loop lives above p90 for the reason the previous chapter derived: the loop cannot touch a request that passed first time, and most requests do.

Token cost has to be expressed per delivered output, not per attempt or per request, and the numerator has to include the tokens burned by exhausted runs. The reason is arithmetic rather than fastidiousness. Cost per attempt stays roughly flat as the ceiling rises, because an attempt is an attempt; cost per delivered output rises, because the additional attempts are concentrated on documents that need many of them and the extra deliveries are few. Omitting the exhausted runs from the numerator understates the loop by however large the exhausted path is, and the exhausted path grows as the inputs get harder, which is exactly when somebody will be looking at the number.

The fourth is an independent quality measure on a fixed task set, which means a set of inputs with known-correct outputs, frozen, along with a pinned model version, a pinned prompt and a pinned verifier version. Everything else is a comparison across two things that changed at once. When the only available quality signal is human labelling, it is stratified by attempt count and reweighted, for the reasons the labelling-budget section covered.

def score_run(results: list[Result], labels: dict[str, bool], traffic_share: dict[int, float]) -> dict:
    delivered = [r for r in results if r.status == "verified"]
    correct = [r for r in delivered if labels[r.doc_id]]
    tokens = sum(r.tokens for r in results)          # includes exhausted runs
    return {
        "verification_pass_rate": len(delivered) / len(results),
        "accuracy_of_delivered": len(correct) / len(delivered),
        "correct_per_1000_inputs": 1000 * len(correct) / len(results),
        "review_queue_per_1000": 1000 * (len(results) - len(delivered)) / len(results),
        "tokens_per_delivered": tokens / len(delivered),
        "p99_seconds": percentile([r.seconds for r in results], 99),
    }

The third key in that dictionary is the one that settles arguments, and its relation to the first two is worth writing out because the relation is the whole reason it settles them. Verification pass rate is delivered / results. Accuracy of delivered outputs is correct / delivered. Multiply them and delivered cancels, leaving correct / results — the third key, expressed per 1,000 inputs. So the quantity the business actually receives is the product of the two headline numbers, and the two factors move in opposite directions when the ceiling changes. Loosening the stopping condition raises the first, because more runs terminate in a pass, and lowers the second, because the runs it rescued are the ones chapter 4 showed are mostly defective. The product can therefore go either way, and it is the only one of the three that cannot be improved by loosening alone.

A check the loop never sees can measure what the loop did#

Human labelling is the strongest independent measure and it costs twelve to twenty minutes per item, which is why most teams take it quarterly on a sample and have nothing in between. There is a cheaper measure available whenever a useful constraint exists that you have chosen to keep out of the loop.

The claims team had one already: the requirement from chapter 3 that every extracted string appear verbatim in the document text, which rejected 51 of the 80 adversarial outputs and costs about 380 milliseconds per output because it needs the OCR text — the machine-read transcription of the scanned page — and fuzzy matching over it. Running it inside the loop would have been affordable at that price. The team deliberately did not, and the reason follows from the pass-rate argument rather than from caution. Any constraint the loop can see becomes part of the stopping condition; every delivered output satisfies the stopping condition by definition; so the moment the source check goes into verify, its rate on delivered outputs is 100% and stays there whatever happens to the extractions. A constraint the loop can see is a constraint the loop optimises against, and the team wanted one check whose verdicts remained informative about outputs the loop had already accepted. Keeping it out is what preserves its ability to vary.

HELD_OUT = (strings_appear_in_source, deductible_matches_policy_schedule)

def held_out_report(runs: list[Result], docs: dict[str, Document]) -> dict[int, tuple[int, int]]:
    """{attempts: (delivered, held_out_clean)} over verified runs only."""
    buckets: dict[int, list[int]] = defaultdict(lambda: [0, 0])
    for run in runs:
        if run.status != "verified":
            continue
        bucket = buckets[min(run.attempts, 4)]
        bucket[0] += 1
        if not any(check(run.value, docs[run.doc_id]) for check in HELD_OUT):
            bucket[1] += 1
    return {k: tuple(v) for k, v in sorted(buckets.items())}

Run over the same 1,000 labelled documents at a cap of three, the held-out check tracks the human labels closely enough to be used weekly in their place.

Before reading the columns, check the last row against the ones above it, because a summary row that does not reconcile is a summary row worth distrusting. The delivered counts add up: 780 first-attempt passes, 121 second-attempt and 37 third-attempt come to the 938 in the bottom row, which is the 93.8% cumulative pass rate from the previous chapter's yield curve expressed per thousand documents. The percentage in the bottom row is not the average of the three above it — it is their weighted average, each bucket's rate multiplied by that bucket's delivered count, summed, divided by 938. Do that arithmetic on the held-out column and it returns 91.1%; do it on the human column and it returns 89.0%. The bottom row is dominated by the first row because the first row holds four fifths of the deliveries, which is why an overall figure of 91.1% sits so close to the first bucket's 97.3% and so far from the third bucket's 40.5%.

The middle two columns are the whole argument of this chapter in one table. The in-loop column is 100% in every row because that column is the stopping condition, and a delivered output satisfied it by definition — the same reason the pass-rate column in chapter 4's attempt table was 100% in every row. The held-out column falls from 97.3% to 40.5% across the same rows. Two checks over identical outputs, one constant and one varying by more than half its range, and the difference between them is not that one is more sophisticated. It is that one was inside the loop and the other was not.

The held-out figure sits above the human label in every bucket, by roughly a point where the loop did least work and by eight points in the hardest bucket. The direction is the one to expect: a fabricated field is only one of the ways an extraction goes wrong, and the check has nothing to say about a real string lifted from the wrong box on the page, so it will always be the more generous of the two. Being biased in a known direction and by a stable amount is what makes it usable as a weekly signal, with the quarterly human sample recalibrating the offset — an unbiased measure would be better, and a biased one whose bias you have measured is a great deal better than nothing four times a year.

The discipline this requires is that a held-out check stays held out. The first instinct on seeing that table is to add the source constraint to verify so the loop stops producing fabrications, which is a reasonable product decision and it costs you the measurement, for the reason given above: from that point the column reads 100% like the one beside it. A team that makes that move needs a new held-out check to replace it, and the candidates are the same ones chapter 2 priced: a second independent extraction to compare against, a lookup of the policy number in the policy table, a reconciliation of the deductible against the schedule. Whatever the loop is allowed to optimise cannot also be what tells you how well it did.

The same rollout, measured properly#

Here is the claims extractor on a frozen set of 1,000 labelled documents, at three configurations, with the model and prompt held constant across all three.

Every row in the top block is derived from the two above it, which makes the table checkable rather than something to be believed. Multiply the pass rate by the accuracy and scale to a thousand and you get the bold row. Subtract the bold row from the delivered count — the pass rate per thousand — and you get the wrong outputs shipped. Subtract the delivered count from a thousand and you get the review queue. Four rows, one measurement pair, and no way to move one without moving the others.

Three things in that table are invisible on a two-bar slide. The loop is worth having, since 835 correct outputs beats 750 and the review queue drops from 220 items to 62. Raising the cap from three to eight increases the pass rate by three points and decreases the number of correct outputs, which is the product argument arriving as a measurement: the pass rate rises by a few per cent of itself while the accuracy of delivered outputs falls by rather more, so the product goes down. Put concretely, every document rescued in attempts four through eight arrives with worse than even odds, and 47 wrong outputs get shipped to buy 30 fewer review items.

The third thing is that the wrong-outputs row triples between the first column and the second — the same move that the correct-outputs row calls an improvement. The bold row nets those against each other without pricing them, which is a real limitation of it. Whether 835 correct and 103 wrong beats 750 correct and 30 wrong depends entirely on what a wrong claim extraction costs relative to a manual review, and that is a business question with an actual answer that somebody should be asked for. The bold row is the right thing to optimise once that exchange rate is known and it is not a substitute for knowing it.

Pass rate is not worthless once the ceiling is fixed. It is a good operational health signal: with the cap pinned at three, a drop from 93.8% to 88% means something about the inputs or the model changed, and that is exactly the alert you want. What it cannot do is compare two configurations that permit different numbers of attempts, because then it is measuring the configuration rather than the system. The lower-level version of this — a conformance rate reported as an accuracy rate — is familiar from verifier-first design; the loop-specific twist is that the retry ceiling is a dial that moves the reported number without touching the outputs.

Two numbers computed by different verifiers are not a comparison#

There is a second defect that survives every argument made so far, and it would remain even if the loop's pass rate were a meaningful quantity. A rate is a count divided by a count, and comparing two rates requires that both counts measure the same event over the same population. The pass-rate counter measures "the verifier returned an empty problem list", and the verifier is a thing that gets edited.

The 71% was recorded in February and the 96% in April, from the same counter, which by then was counting something else. Between the two measurements the constraint list had been amended twice: the loss-date window widened to admit latent-damage claims, and the deductible-against-total rule loosened after the credit-note rejections from chapter 3. Both amendments enlarge the accepting set, so both raise the pass rate on any fixed population of outputs, with no change to the outputs at all. Re-running the February single-pass configuration over the same 1,000 documents under the April verifier gives 84.2%, which splits the slide's twenty-five points cleanly: thirteen of them belong to the amendments and twelve to the loop.

This is why the attempt histogram in chapter 4 carries a verifier_version tag, and the tag is only useful if the reporting honours it. A pass-rate series that is not segmented by verifier version is a series whose meaning changes silently at every deployment, and the shape of the failure is a step change in the chart that gets attributed to whatever else shipped that week. Anything presented as a before-and-after needs four things pinned and stated: the input set, the model version, the prompt, and the verifier version. A baseline is then a number you re-run rather than one you quote, at a cost — from the table above — of about $14.60 and a few minutes of batch time per 1,000 documents, which is cheap enough that quoting a stale figure is never the economical choice.

Deciding whether the loop is worth keeping at all#

The code-fix agent of chapter 4 gives the cleanest instance of this question, because someone eventually proposes removing the loop on the grounds that models have improved. The suite takes 40 seconds, the median task takes two attempts and p95 takes six, and the argument is that a single pass would be simpler and much faster.

What settles it is the single-pass rate against the looped rate on the same task set, and nothing else — not intuitions about model quality, and not the current pass rate, which as established reports the ceiling. The reason the single-pass rate is the load-bearing figure is that a loop is worth exactly the tasks that fail on the first attempt and succeed later. Tasks that pass immediately would pass without it, tasks that never pass cost it money, and the difference between the two configurations on a fixed set is the only thing that isolates the remainder. On a frozen set of 300 tasks with the current model:

The loop is worth 27 points of suite-green and 13 points of real fixes. The gap between those two numbers is the special-casing and the assertion edits from chapter 4, arriving at a rate of roughly one wrong fix for every two real ones among the rescued tasks. Against that, p95 goes from 58 seconds to 348 seconds, which is five extra suite executions and about four extra minutes on the tail. Note that the single-pass column has the same figure at the median and at p95, because a design with no retries has almost no spread to speak of — the whole shape of the p95 column belongs to the loop.

Whether four minutes is affordable is a question about where the loop runs, and the two answers point in opposite directions from the same numbers. In a CI job that a developer is watching, four minutes at p95 is unpleasant and 13 extra points of real fixes probably does not buy it, because the developer can rerun the agent themselves and will be reading the diff anyway — the human is already in the loop, so the loop's rescues are duplicating work someone was going to do. In a nightly batch that files pull requests against flaky tests, nobody is waiting, the four minutes are free, and the same 13 points are straightforwardly worth having as long as the review step is honest about the rescued-task defect rate. The measurement is the same in both settings and the decision is not, which is why the decision belongs in a written record rather than in whoever remembers the numbers.

Everything in this chapter reduces to one question asked of any reported number: could this have moved without the outputs changing? A verification pass rate can, by raising the ceiling or by loosening a constraint, and both happened here. A held-out check's rate cannot, as long as it stays held out. Correct outputs per thousand inputs cannot, because its two factors move against each other. That question is cheap to ask and it is the difference between a measurement and a counter.

Every configuration compared here assumed the check exists and means something — a test suite that encodes real behaviour, a constraint list tied to a document. The next chapter takes the assumption away and asks what remains of all this when the only available signal is a model's score out of ten, which is the situation most teams are actually in when they first reach for this pattern.