The checker independence ladder

There is a ladder of checkers, ordered by how much of the generator's machinery the checker still shares: the same model with the answer's reasoning still in context, the same model with a fresh context, a model from a different family, and a check whose verdict is computed rather than generated. Every rung further up removes something the generator and the checker had in common, and every rung costs something different — tokens, a second provider in the request path, or engineering time spent writing a predicate.

The rung a team is standing on is usually asserted rather than measured, and the word that does the asserting is independent. Two components are independent when knowing that one failed tells you nothing about whether the other did, and two calls to the same weights are nowhere near that, so a design doc describing a fresh-context critic as an independent check has licensed everyone reading it to multiply two error rates together.

The quantity that decides what a checker is worth#

Suppose a contract-clause extractor pulling auto_renewal_notice_days out of customer agreements is right 87.6% of the time, and a critic — given the clause and the extracted value with no reasoning in context — is, evaluated on its own as a classifier of "is this value correct", also right about 88% of the time. If those two error sets were disjoint you would catch nearly everything. If they overlap almost completely, the second component is decorative.

Overall accuracy cannot tell those two situations apart, and that is the whole reason it is the wrong number to put in a design doc. A critic whose errors are disjoint from the extractor's and a critic whose errors are identical to the extractor's can both score about 88%, because the set being scored is dominated by cases the extractor already got right. Any figure computed across that whole set is mostly a measurement of the extractor. Learning something about the critic means scoring it on a population where the extractor's behaviour is held fixed, and there is only one population where a checker can change the outcome at all.

The number to compute from a labelled set is the critic's accuracy conditioned on the generator being wrong, next to its unconditional accuracy. On 500 labelled cases the extractor got 62 wrong; the critic flagged 9 of those 62 and raised 14 objections on the 438 the extractor had got right. So those two figures are 15% and 88%. A checker that is 88% accurate overall and 15% accurate on precisely the cases you need it for is telling you that its errors live in the same places as the generator's, which is what you should expect from the same weights.

Both of those percentages are a division, and which count sits underneath is the entire content of each. The 15% is 9 divided by 62 — the flags raised on cases the extractor got wrong, over every case the extractor got wrong. That denominator is the right one because those 62 requests are the only ones on which a checker can improve anything. On the other 438 the extraction was already correct, so the best available behaviour there is silence and the only available contribution is damage. Recall on the generator's errors therefore answers the question the design doc was really asking: of the bad answers that would otherwise ship, what share does this component stop?

The 88% is the same critic scored across all 500, counting a flag on a wrong extraction and a silence on a right one alike as correct. That denominator merges the population where the checker can help with the population where it can only interfere, and the second is seven times the size of the first. A mixture is dominated by its larger part, so most of what the figure reports is how often the extractor was right. That is a fact about the extractor, arriving under a label that says critic.

The cheapest way to see how little the overall figure carries is to ask what the laziest possible critic would score on it. A component that answered "supported" to everything, without reading the clause at all, would be correct on all 438 extractions that were right and wrong on all 62 that were not, which is 87.6%. The measured critic comes in at 87.4%. Its overall accuracy is not merely close to the do-nothing baseline, it is on the wrong side of it, and a number that a constant function beats is not evidence that anything is working.

What such a critic does catch is worth looking at, because the pattern is consistent. The 9 were cases where the extraction was locally implausible on its face: a notice period of 3,650 days, a value pulled from a clause about payment terms that mentioned no days at all. The 53 it missed were cases where the extractor had made a mistake that any careful reader of similar contracts might make, which is to say the mistakes that follow from how contracts usually read.

This has an exact precedent outside machine learning. In the 1980s, safety-critical systems were built on N-version programming: pay three independent teams to implement the same specification, run all three, and vote. Knight and Leveson's 1986 experiment had 27 versions written independently from a common spec, and found that the versions failed together far more often than an independence assumption predicted, because the specification itself had ambiguous corners and hard cases, and every competent programmer stumbled on the same ones. Independence of process did not deliver independence of failure. Two calls to the same model share far more than a common spec — they share the weights — so the correlation is correspondingly worse.

The last step of that comparison is the one that carries the argument, and it is worth walking rather than asserting. What the 27 teams shared was a description: one document, with its ambiguous corners sitting in the same places for everyone who read it. Each team then built its own machinery, so the only available route to a common failure was a common misreading of that document — and that alone was enough to sink the independence assumption. Two calls to the same model share the description too, since they are handed the same prompt and the same clause. On top of that they share the machinery. Every ambiguity that would have made two human implementers diverge gets resolved the same way both times, because it is resolved by the same weights. The mechanism that defeated N-version programming is present here in full, with a second one stacked on top of it.

Four rungs, and what each one actually buys#

Moving from the first rung to the second is a genuine improvement and the numbers show it: on the same 500 cases the second-turn critic flagged 4 of the 62 errors against the fresh call's 9. Removing the reasoning chain from context removes the pressure to stay consistent with it, so the checker at least evaluates the claim on its own terms. It is a de-anchoring move. Calling it independence is the mistake, because the word invites you to multiply the two error rates together and conclude that the combined system fails 1.5% of the time, when the measured combined failure rate was 10.6%.

Both of those figures are worth reconstructing, because the distance between them is what the word independent costs when it is wrong. The 1.5% is a product. The extractor is wrong a little over twelve times in a hundred, the critic is wrong about twelve times in a hundred, and if neither component's failure said anything about the other's, the chance of both failing on the same request would be the one rate multiplied by the other, which lands near one and a half in a hundred. Multiplication is the step that requires independence, and it is permitted only when knowing that the extractor failed leaves the critic's failure probability where it was.

The 10.6% comes from counting rather than multiplying. A request fails end to end when the extractor got it wrong and the critic did not flag it, so the count is the 62 errors minus the 9 that were flagged, which is the 53 above, and 53 out of 500 is 10.6%. Set that beside the extractor's own failure rate of a little over twelve in a hundred, and the size of what the critic bought is legible: it moved the end-to-end failure rate by under two points, where the independence assumption had promised nearly eleven. The component delivered roughly a sixth of the improvement the word had been used to promise, and the design doc contained no number that would have exposed the gap.

The third rung is where the correlation genuinely drops. Routing the same 500 cases through a different provider's model as critic caught 27 of the 62, roughly three times the same-family critic, at the price of 41 objections on the 438 correct extractions. Some of those objections were real disagreements about ambiguous clauses; most were the second model preferring a different reading of "days" versus "business days" and saying so. The state of the art in evaluation harnesses has moved toward panels of judges from different families for exactly this reason — a judge here meaning a model call whose job is to score or compare outputs rather than to produce them — and the honest caveat is that the independence is partial: model families are trained on heavily overlapping web corpora, and a good deal of post-training data is generated by other models, so shared blind spots survive the family boundary more often than the architecture diagram suggests.

Two qualifications belong next to that gain, and both come out of the same four counts. The first is that a different family is not a solved problem: 27 of 62 means 35 of the extractor's errors survived the best model-based rung on this ladder, so more than half of what you are trying to catch is still getting through. The second is that the exchange rate did not move. Nine catches at fourteen objections and twenty-seven catches at forty-one objections work out to nearly the same share of flags being real; the cross-family critic finds about three times as much and complains about three times as often. What improved is coverage of the generator's error set, which is what correlation is about and what moves the end-to-end failure rate. What did not improve is how much correct output gets disturbed per genuine catch. A team that budgets for the first number and not the second is budgeting for the model bill and not for the review queue.

The third rung also carries operational weight that the table understates. A second provider means a second API key, a second rate limit, a second set of availability incidents in the request path, and a data-residency conversation that may take longer than the engineering work. Latency stops being additive in a helpful way too, since the critic call cannot start until the generation finishes, and a p99 — the latency that the slowest one request in a hundred exceeds — on a second provider that you do not control lands squarely in your own p99. Teams that run cross-family critics in production usually run them asynchronously — the answer ships, the critic writes to a review queue — which is a different product than a blocking check and worth deciding deliberately rather than discovering.

The fourth rung is the only one where the correlation is zero by construction, because there are no weights involved. A function that recomputes a sum, validates against a JSON Schema, executes a generated query against a staging database, or checks that every quoted string appears verbatim in the retrieved source, fails for its own reasons entirely.

"Its own reasons" is doing precise work there, and it is worth cashing out against the definition of independence at the top of this chapter. A JSON Schema validator returns the wrong verdict in two situations: the schema is wrong, or the schema is silent about the field that went bad. Both are properties of a document a person wrote before the request arrived, and neither of them shifts according to what the model generated. So knowing that the extractor was wrong on a particular clause tells you nothing about whether the validator will also be wrong on it — which is exactly the definition given above, satisfied here for a structural reason rather than an argued one. No prompt can reach that position, however it is worded, because a prompt's verdict is produced by the same machinery whose output is in question.

Comparing a critic against single-pass output#

Whichever rung you pick, the thing worth building first is the harness that tells you which rung you are on. The critic itself is a short prompt; the comparison against single-pass output is what turns it into evidence.

from dataclasses import dataclass

CRITIC = """You are checking one extracted field against the clause it came from.

CLAUSE:
{clause}

EXTRACTED: auto_renewal_notice_days = {value}

Quote the exact sentence from the clause that states this notice period.
If no sentence states it, write NOT FOUND.
Then output JSON: {{"quote": "...", "verdict": "supported" | "unsupported"}}"""


@dataclass
class Row:
    case_id: str
    gold: int
    single_pass: int
    verdict: str          # "supported" | "unsupported"


def score(rows: list[Row]) -> dict[str, float]:
    wrong = [r for r in rows if r.single_pass != r.gold]
    right = [r for r in rows if r.single_pass == r.gold]
    return {
        "single_pass_acc": len(right) / len(rows),
        "recall_on_errors": sum(r.verdict == "unsupported" for r in wrong) / len(wrong),
        "false_alarm_rate": sum(r.verdict == "unsupported" for r in right) / len(right),
        "critic_acc_overall": (
            sum(r.verdict == "unsupported" for r in wrong)
            + sum(r.verdict == "supported" for r in right)
        ) / len(rows),
    }

# 500 contract clauses, same-model fresh critic:
# {'single_pass_acc': 0.876, 'recall_on_errors': 0.145,
#  'false_alarm_rate': 0.032, 'critic_acc_overall': 0.874}

The two numbers that end up in the design doc are recall_on_errors and false_alarm_rate, and reporting critic_acc_overall on its own is how a useless critic gets shipped: 87.4% looks like a working component, and it is almost entirely the base rate of the generator being right.

Both numbers need their uncertainty attached, and the recall figure is the fragile one because it is estimated from a small denominator. Fifteen percent recall measured on 62 errors has a 95% interval — the range that would contain the true value in nineteen repetitions out of twenty — running from roughly 7% to 25%, so a follow-up run reporting 22% is not an improvement, it is the same critic. Getting that interval down to something you can make decisions with — say plus or minus five points — needs on the order of 150 to 200 labelled generator errors, which at an 87% base accuracy means labelling well over a thousand cases. That cost is the real reason critics ship unmeasured, and the cheaper compromise is to label a set enriched with known-hard cases and report recall on that, clearly marked as a biased sample, rather than to report nothing.

Why a small denominator produces a wide interval is worth making concrete rather than taking on authority. The recall estimate rests on nine countable events. Had four of the extractor's mistakes happened to be locally implausible rather than plausibly wrong — a different draw of contracts, nothing else altered — the count would have been four higher and the reported recall would have jumped by around six points, with nothing about the critic having changed at all. An estimate built from a handful of events is coarse for that reason, and the interval is the honest width of the coarseness. The labelling arithmetic at the end of the previous paragraph is the same fact read backwards: errors arrive at a bit under one case in eight, so accumulating a couple of hundred of them means labelling something over a thousand cases to find them in.

The false-alarm rate deserves as much attention as the recall, because at a low error base rate it dominates what the critic costs you. With 62 errors and 438 correct outputs, a critic flagging 9 real problems and 14 correct ones has a precision of 39%: for every genuine catch, roughly one and a half good answers get sent for revision. If flags trigger an automatic rewrite, that is 23 outputs disturbed to repair 9, and the rewrites of already-correct answers are not free — they are the mechanism from the previous chapter operating in the other direction, where a model asked to fix something will fix something. If flags instead route to a human queue, precision translates directly into reviewer time spent confirming that nothing was wrong.

Precision is a third ratio with a third denominator, and confusing it with the other two is the commonest reading error in a table of this shape. It is 9 over 23: the flags that were right, over every flag raised. That denominator is chosen because it is what the consumer of the flag actually sees — a review queue has no way of knowing which of its items are real, so it pays for all 23. Recall answers what share of the problems get caught. Precision answers what share of the alarms are worth acting on. The two are computed over different populations and they move in opposite directions whenever a threshold is loosened, which is why one of them alone is never enough to decide anything, and why the design doc carries the pair.

Notice what the critic prompt does. It does not ask whether the extraction is good. It asks for a quote, which is a retrieval task with a determinate answer sitting in the context, and derives the verdict from whether the quote exists. That is roughly the ceiling of what a prompt-based critic can do: convert an appraisal into a lookup.

It is a ceiling rather than a foundation because a lookup only works where the answer is in the context. Rewrite the appraisal as a retrieval and the critic stops guessing about anything the clause states, since a quote either exists or it does not. It goes on guessing about everything the clause does not state, because for those questions there is nothing to retrieve and the prior is the only resource left in the room. Every remaining rung on the ladder is an attempt to bring in a resource that is not the prior.

Agreement between two models#

Two models agreeing is not evidence of correctness, and it is treated as such more often than any other error in this area. When a generator and a same-family critic agree, the most likely explanation is that they share a prior, so agreement should not move your confidence much at all. When a generator and a cross-family critic agree, that is worth a little more, and quantifying how much requires the labelled set — there is no way to read it off the agreement rate alone.

The step underneath that is worth holding onto, because it generalises past this chapter. Agreement carries information in proportion to how likely disagreement was. Two components that would have parted company over a wrong answer say something real when they do not part company. Two components that would have produced the same wrong answer say nothing, because agreement was the outcome whichever way the answer went. Recall on the generator's errors is precisely the measurement that tells you which of those two you have: a critic that flags 9 of 62 is a critic that sided with the extractor on 53 of the answers you most needed it to challenge. Its agreement on the next request is therefore weak evidence, and the reason it does not feel weak is that a verdict of "supported" reads the same on a dashboard whether it cost the critic anything or not.

The ladder also flattens something real: rung four is only uncorrelated within its scope. A schema validator has zero correlation with the generator's errors and also zero coverage of anything the schema does not describe, which is why "add an objective check" is not a complete answer to "make this correct". A totals-recompute check on an invoice extractor catches arithmetic and catches nothing about whether the supplier name came from the right block. The value of the fourth rung is that its coverage is knowable, since you wrote the check and can enumerate what it looks at, whereas the coverage of a model critic has to be measured and re-measured whenever the model version changes.

Errors of missing context#

There is a class of error that no rung on this ladder touches, and it is worth naming before it gets lost among the mechanics. If the clause genuinely says thirty days and the extractor writes thirty days, every checker here says supported — including the schema validator, including the cross-family critic — and the answer is still wrong if the governing document is an amendment that superseded that clause. Errors of missing context are invisible to a checker that is handed the same context. The fix is retrieval and provenance rather than critique, and no amount of reviewing the answer against the clause will produce it.

So the ladder answers a narrower question than the one teams bring to it. It does not tell you whether a checker is any good; it tells you how much of the generator's blindness the checker has inherited, which is why a critic can be 88% accurate and still be worth almost nothing on the traffic that matters. Recall on the generator's errors and the false-alarm rate are the pair that make that visible, both with an interval attached, and the rung is the lever that moves them: fresh context buys de-anchoring, a different family buys a genuine if partial drop in correlation at real operational cost, and a computed verdict buys zero correlation over a scope you can enumerate.

Having a ladder does not tell you which rung is worth its cost for a given feature, and the argument that a critic step is a good idea is nearly always made from a headline number: accuracy went from here to there. That number is where most of the damage gets done.