Measuring a checker with independent labels

A verifier-first design produces rich telemetry about the wrong error class. A service can say precisely how many outputs its validator rejected, how long the checks took, and how many requests exhausted their retries, because every one of those is an event the system generated. It cannot say how many outputs it accepted while they were wrong, and no query will produce that number, then or later.

That is structural rather than an oversight in instrumentation. The checker's accept decision is the only opinion the system holds about an output, so within the system a wrongly accepted output and a rightly accepted one are indistinguishable. Asking a service for its own false accepts is asking it to disagree with itself. This chapter is about the outside opinion that supplies the missing column, what it costs, and what it makes measurable that was not measurable before.

Where the second opinion has to come from#

The missing column comes from labels produced without reference to the checker's verdict. In practice that means drawing a sample from production traffic, stripping the verdict, having someone competent to judge the output decide whether it is correct against the request and the source material, and only then joining their labels back to what the checker said.

Three details in that sentence are load-bearing. The labeller must not see the verdict, because knowing that a validator passed an extraction makes a reviewer read it more forgivingly, and the effect is large enough to swamp the rates you are trying to measure. The sample has to include rejected outputs as well as accepted ones, or the false-reject column stays empty. And "correct" has to be written down before labelling starts, as a short adjudication rule that says what to do with the ambiguous cases — a supplier name abbreviated the way it appears on the letterhead, a date the document itself states twice in two formats — because otherwise two labellers will disagree at a rate comparable to the error you are chasing.

Here is a real measurement of an invoice extractor's schema-plus-invariants checker, taken over 300 accepted and 150 rejected outputs sampled from two weeks of traffic. Rejected outputs are deliberately over-sampled relative to their frequency, because they are rare and the interesting question about them is what fraction were correct all along.

The false accept rate among accepted outputs is 21/300 = 7.0%, with a 95% interval of roughly 4.1% to 9.9%. The false reject rate among rejected outputs is 34/150 = 22.7%, which means nearly a quarter of the regenerations the loop paid for were repairs of outputs that did not need repairing. Because rejections were over-sampled, these two rates have to be reweighted by the true rejection rate — 5.8% of production traffic — before they combine into a system-level accuracy figure: accepted work is 94.2% of traffic at 93.0% correct, so the accuracy of served output is about 93.0%, and the checker is throwing away 1.3% of traffic that was already correct.

The reweighting step is the one to slow down on, because it is where the over-sampling has to be paid back and it is easy to skip. Each rate is a conditional quantity: 7.0% is the error rate among outputs the checker accepted, and 22.7% is the error rate among outputs it rejected. Neither is a share of traffic, because the sample deliberately contains one rejected output for every two accepted ones, while production contains roughly one for every seventeen. To convert a conditional rate into a share of traffic, multiply it by how much traffic that condition covers. Rejections are 5.8% of traffic, so correct-but-rejected outputs are 22.7% of that 5.8%, which is about 1.3 in every hundred requests. Acceptances are the remaining 94.2%, and the correct ones among them are 93.0% of that. Skip the multiplication and the 22.7% reads as though a fifth of all traffic were being thrown away, which is some seventeen times the truth, and that is the single most likely misreading of a table like this one.

The two figures that come out of it answer different questions and should never be swapped. The 93.0% describes what a consumer of an accepted output gets, and the 1.3% describes what the business is paying to have thrown away — one is a quality number, one is a waste number, and neither can be derived from the other without the rejection rate. Notice too that the 1.3% is a floor on the loop's cost rather than the whole of it: those requests were not merely wasted, they were regenerated, so they also consumed a further generation and a further check.

The width of that first interval is the thing to plan around. Sample size sets the smallest error rate you can see, and the relationship is unkind at the low end. With 300 labels you can resolve a 7% false accept rate to within about three points, which is enough to make decisions. If the rate were 1%, 300 labels would expect three hits and the interval would run from 0.2% to 2.9%, which is not enough to tell an improvement from noise. Getting a 1% rate to within half a point takes on the order of 1,500 labels, and that is a real budget line — at four minutes per invoice, 100 hours of somebody's attention. This is the honest constraint on the whole practice: measuring rare failures costs roughly the inverse of how rare they are, so the low-frequency, high-severity failures are exactly the ones you will be least able to quantify.

Every one of those numbers falls out of one relationship, and having it makes the budget arguable rather than mysterious. The uncertainty on a proportion measured over some number of labels shrinks with the square root of that number, and it also shrinks as the proportion itself gets closer to zero — the noise is set by the square root of the rate divided by the count. On 300 labels at a rate near 7%, that works out at a bit under one and a half points, and a 95% interval is about two of those either side, which is the three points quoted and the 4.1%-to-9.9% span in the table. Nothing else is going on in that interval.

Now push the rate down and watch what happens to the relative precision. A 1% rate over 300 labels expects three hits, and the same expression gives a spread that is small in absolute terms and enormous next to the quantity — 0.2% to 2.9%, an interval whose top is more than ten times its bottom. That is why the 1% case is unusable while the 7% case is fine: both are measured to within a few points, and a few points is a detail on 7% and the whole answer on 1%. Demanding half a point instead of three is asking for six times the precision, and precision costs its own square in labels — thirty-six times over. Pulling the other way, a rate near 1% carries much less variance than one near 7%, which hands back most of a factor of seven. What survives is roughly a fivefold increase, and that is how 300 becomes 1,500. Multiply 1,500 by four minutes and the answer is six thousand minutes, which is the 100 hours.

The general form is the sentence to carry out of this chapter. To measure a rate to within a fixed fraction of itself, rather than to within a fixed number of points, the sample size needed goes as the inverse of the rate. Halve the failure rate and you double the labels. A failure that happens once in a thousand needs tens of thousands of labelled items before anybody can say whether a change helped, which for most teams means it is not going to be measured at all — and those are exactly the failures severe enough to have prompted the question.

Label quality sets a second floor. If your labellers agree with each other 96% of the time, you cannot credibly report a 2% false accept rate, because the disagreement between labellers is larger than the quantity being measured. Running a double-labelled overlap of 50 items and reporting the agreement statistic alongside the rates is what keeps that honest; the invoice measurement above had 47 double-labelled items with a Cohen's kappa of 0.81 — kappa being an agreement score discounted for the agreement two labellers would reach by guessing, where 1.0 is perfect and 0 is chance — which supports a 7% estimate and would not support a 1% one.

The reasoning behind that floor is worth separating from the sample-size argument above, because it is a different constraint and adding labels does not relieve it. Sample size describes how precisely you have measured your labellers. Agreement describes whether your labellers are measuring the same thing. If two competent people looking at the same invoice reach different verdicts on four in every hundred, then the labels carry four points of disagreement about which side of the line an item falls on, and any quantity smaller than that is inside the instrument's own noise. Doubling the sample makes the estimate of a two-point rate tighter and no more trustworthy, because it is estimating a quantity the labels cannot resolve. The remedy is not more labels. It is a sharper adjudication rule, which is why the rule gets written before labelling starts rather than negotiated afterwards when the disagreements are already in hand.

Turning the loop into a measured one#

The same labelled set is what lets you say what the loop bought, which is a different claim from what the checker admits. The measurement is two arms — the experimental sense of the word, where an arm is one run of the whole evaluation set under one configuration — over a single set of cases, holding the model, the prompts, the decoding parameters and the set itself constant, and changing only whether the verifier is in the path.

def run_arm(cases, generate, verify, max_attempts):
    served, unserved, attempts = [], 0, 0
    for case in cases:
        feedback, output = None, None
        for _ in range(max_attempts):
            attempts += 1
            candidate = generate(case.request, feedback=feedback)
            verdict = verify(candidate)          # a no-op verifier in the baseline arm
            if verdict.accepted:
                output = candidate
                break
            feedback = verdict.reason
        if output is None:
            unserved += 1
        else:
            served.append((case, output))
    return served, unserved, attempts

# baseline: max_attempts=1, verify=lambda c: Verdict(True, None)
# loop:     max_attempts=3, verify=schema_and_invariants

Both arms produce outputs that then go to the same labellers, blind to which arm they came from, and the accuracy numbers come from the labels. What comes back from the invoice service's own evaluation set of 400 cases is worth reading carefully:

• Baseline arm: 400 served, 331 correct. Accuracy 82.8%.
• Loop arm: 377 served, 351 correct, 23 unserved after three attempts. Accuracy on served output 93.1%; accuracy over all 400 requests, counting unserved as failures, 87.8%.

Both of those arms reconcile against 400, and putting the counts side by side says more than either accuracy figure does. The baseline served all 400 and got 331 of them right, so it served 69 wrong outputs. The loop served 377, of which 351 were right, so it served 26 wrong outputs, and refused 23 requests entirely. Add the loop's three groups: 351 and 26 and 23 make 400. Now compare the wrong columns. Wrong outputs served fell from 69 to 26, a fall of 43 — and 43 is exactly 20 plus 23, where 20 is how much the correct count rose and 23 is how many went unserved. So the counts are consistent with a loop that repaired twenty requests and refused twenty-three, with every refusal being a request the baseline would have got wrong.

That last clause is a reading rather than a measurement, and the distinction matters. The totals alone cannot say which individual cases moved; a loop that repaired twenty-five and refused five previously-correct requests produces the same four numbers. Only the per-case labels settle it, which is the argument for labelling both arms case by case rather than accepting two accuracy percentages and a coverage figure. What the totals can say without any assumption is that the loop is not merely filtering: the correct count rose, and filtering alone can never raise the number of correct outputs.

Two numbers, both true, and they answer different questions. The 93.1% is what a downstream consumer of an accepted output experiences. The 87.8% is what the product experiences, because the 23 unserved requests did not evaporate — they became errors, or fell back to a manual queue, or were retried by a user who then waited twice as long. Reporting only the first is the standard way a generate-then-verify loop is oversold, since any loop can raise accuracy-on-served arbitrarily high by rejecting more. Coverage and accuracy have to be reported as a pair, always, and the honest summary of this deployment is "accuracy on served output up 10.3 points, coverage down to 94.3%".

"Arbitrarily high" is not a figure of speech, and running the limiting case shows that accuracy-on-served is not a bounded claim at all. Tighten the checker until it accepts only the one output it is surest of. That output is very likely correct, so accuracy on served output is essentially perfect, and the service has answered one request out of four hundred. Loosen the checker slightly and the number falls. The entire curve between those two points is available to anybody tuning a threshold, which means a reported accuracy-on-served can be set to whatever the reporter would like by moving a dial the report never mentions. Coverage is that dial. Quoting one without the other is not a question of emphasis; it is quoting an answer without its question. Both halves of the honest summary come from the same two counts — 351 correct out of 377 served, and 377 served out of 400 requested — and neither half means anything alone.

Anything that changes between the arms invalidates the attribution. A team that adds a verifier and simultaneously moves the schema from the prompt into a structured-output constraint has measured the sum of two changes; a team that runs the baseline on last month's evaluation set and the loop on this month's has measured the drift in their traffic. The discipline is boring and it is what makes the delta mean something: same model version, same prompt text, same set, same labellers, one variable.

Neither a conformance rate nor a model judge#

A labelled accuracy figure is not the same as a conformance rate, and the two get reported under the same word. A conformance rate — 99.4% of outputs passed the schema — is a property of the checker's own behaviour and is available for free from the logs. It goes up when the generator improves, and it also goes up when the checker gets weaker, which is what makes it dangerous as a headline metric.

It is also not the same as evaluation by a second model, however well-calibrated that model appears. A model-scored eval is fast enough to run on every commit and is genuinely useful for that, but it does not produce independent labels: it shares training data, tokenisation and failure modes with the generator, so it agrees with the generator most reliably in the cases where the generator is wrong for reasons rooted in its training. The right relationship between the two is that the labelled set calibrates the model judge — you measure how well the judge's verdicts match human labels on the same items, and if the agreement is high enough on the classes you care about, you can use the judge for continuous monitoring while re-checking it against human labels periodically. That relationship, and its limits, is taken up when we compare deterministic checks against model judges directly.

Finally, none of this is a once-off. The rates you measure are properties of a checker against a traffic distribution, and traffic moves — a new supplier onboards with a different invoice layout, an upstream team changes a chunking strategy, a model version rolls forward. A labelled measurement six months old is a historical fact about a distribution that no longer exists.

Labels are the only instrument that can see a false accept, they cost roughly the inverse of how rare the failure is, and they are what converts two plausible-sounding claims — that the checker works and that the loop improved accuracy — into numbers with intervals attached. Everything qualitative in the previous chapter about accept sets becomes quantitative at the price of a week of somebody's attention, and stays true for as long as the traffic does.

With labels in hand you can say what a loop does to accuracy and what it costs in coverage. The remaining question is what it costs in money and time, and the invoice team's answer at this point was that verification was cheap: 11 ms at the median against a 1.4-second generation. That comparison is between the wrong two numbers. The cost of a verifier-first design is set by how often it rejects and by what the slow end of both the generator and the checker do on the requests that get rejected, and the next chapter puts those on the same page.