Error independence in repeated runs

A model that cannot say which of its answers are wrong is hard to operate, which is why a particular proposal comes up on every team running document extraction: send every image twice, once at full resolution and once downscaled, accept a field only when the two runs return the same value, and queue the rest for a human. The arithmetic offered in support is straightforward — if each run is wrong 6% of the time and the errors are independent, both runs are wrong together 0.36% of the time, so the accepted fields land north of 99.5% and the human queue absorbs the remainder.

Every step of that arithmetic is correct. The premise it rests on is false, and it is false in a specific way that makes the scheme not merely weaker than advertised but pointed at the wrong errors. This chapter is about that premise — what independence means for two runs of a model, why varying resolution is the one axis guaranteed to violate it, how to measure the violation in an afternoon, and what a check that does carry signal looks like.

Independence is a claim about the two runs, and it is checkable#

Start with the multiplication, because it is the step nobody checks. Take a hundred fields and let the first run be wrong on six of them, which is the 6% error rate written as a count. A wrong value survives into the accepted set only when the first run is wrong and the second run is wrong on that same field, so the quantity the arithmetic needs is not how often the second run is wrong across all hundred fields. It is what fraction of those six particular fields the second run also gets wrong. Suppose it is six per cent, the rate the second run posts on fields in general: then it is wrong on six per cent of six fields, a little over a third of one field out of the hundred, and a third of a field in a hundred is 0.36% of them. Multiplying the two rates is what taking six per cent of six per cent means, which is why multiplication is the operation rather than some other way of combining two error rates.

Every step of that is bookkeeping except the supposition, and the supposition carries the whole result. In the form it is actually used it says that knowing the first run was wrong on a field tells you nothing about whether the second run is wrong on it — a claim about one particular pair of runs, true of some pairs and false of others. When it is false the multiplication is not slightly optimistic. The rate the calculation needed was the second run's error rate restricted to the fields the first run failed, the rate substituted was its error rate everywhere, and the answer is wrong by the gap between them.

The word doing the work is "independent". It would hold if the two runs were two different observers with different vantage points — different models, different training, different views of the page. What the proposal actually creates is one observer and a copy of the same observer looking at a degraded version of the same image. Everything the downscaled run can see, the full-resolution run can also see, plus more. The relationship between them is not two samples from a distribution; it is a subset relation.

That has a direct consequence for the two failure quadrants, and it follows from the nesting rather than from anything measured. A field the downscaled run gets wrong through lack of information is a field where that information was present in the full-resolution image and destroyed by the downscale — and present in the full-resolution image means available to the full-resolution run. So the cheap run's information-loss errors land in the disagreement cell with the expensive run holding the right value, which is why disagreement is overwhelmingly the cheap run being wrong about something the expensive run got right, and flagging it sends a correct answer to a human.

The both-wrong quadrant is that argument run backwards. If a field defeats the full-resolution run, whatever was needed to read it was already missing from the full-resolution image, and everything a downscale can remove was missing from the smaller image too — so the smaller run cannot supply it, and any disagreement it produces here is a coincidence rather than a rescue. Missing from the image rather than from the sampling means the cause lives in the ink: a faint or broken digit, a scanner artefact through a character, a printed 8 whose crossbar did not survive the fax. That digit looks like a 3 at full resolution and it looks like a 3 at half resolution, because halving the pixels does not change what the ink did — a downscale is a function of the pixels, and the pixels faithfully record a crossbar that was never there. The two runs agree, the field is accepted, and it is wrong. The condition to keep attached is that such errors survive these two resolutions: a digit illegible in the image you hold may be legible in a better capture, so a rescan is a different intervention with a different failure profile.

The belief underneath the proposal is that agreement is evidence, and it is tempting for a good reason: two observers whose mistakes have unrelated causes rarely make the same mistake on the same field, so a match demands an explanation and the simplest one is that both were right. Agreement between an observer and a degraded copy of itself explains itself without anybody being right — same ink, same weights, less image.

Run it on 10,000 fields and the shape comes out clearly. A full-resolution run scoring its usual 94% is paired with a downscaled run at 89%, five points lower, which is itself the first sign that these are not peers.

The full-res-wrong row is where the independence claim gets tested, because it holds the second run's behaviour restricted to the fields the first run failed. The full-resolution run was wrong on 600 fields, and on 450 of them the downscaled run was wrong too — three quarters. Its error rate over the whole sample is the 1,100 in the totals row divided by 10,000, or eleven per cent. So the second run is around seven times more likely to be wrong on a field the first run failed than on a field drawn at random, and that ratio is the dependence, since independence is precisely the claim that the two rates are one number.

The same fact stated as rates: under independence the both-wrong rate is one run's error rate times the other's, and 6% times eleven per cent is 0.66%. Observed, it is the bottom-right cell over the sample, 450 out of 10,000, which is 4.5%.

Of the 450 both-wrong fields, 380 carried the same wrong value and 70 differed. So the accepted set — everything the two runs agreed on — is 8,750 + 380 = 9,130 fields, of which 380 are wrong; the two runs agree when both are right, which is the top-left cell, and also when both are wrong in the same way, which is the 380. Dividing the errors that survive by the set they survive in, 380 over 9,130 is a little over four per cent wrong, so accuracy on accepted fields is 95.8%, against the 99.5% the independence argument predicted and the 94% the pipeline started with. Subtracting that starting point from the accepted-set figure gives the whole return. One and eight-tenths of a point, for double the inference cost.

The review queue is worse than unhelpful. It is every field the two runs disagreed on, which is the two off-diagonal cells plus the both-wrong fields whose wrong values differed, since two different wrong answers do not agree either: 650 + 150 + 70 = 870 fields, 8.7% of the total. Its composition carries the rest of the story: 650 of them (75%) are fields the full-resolution run already had right, 150 are fields the downscaled run rescued, and 70 are fields both got wrong in different ways. Three quarters of the human's time goes to confirming answers that were already correct. Meanwhile, of the 600 errors the scheme set out to eliminate, the queue surfaces only the two cells where the runs disagreed — the 150 the downscaled run rescued and the 70 both got wrong differently, which is 220, or 37% — and lets the other 380 through, 63% of the errors, and those are precisely the correlated ones, which is to say precisely the hard ones.

Queue size alone is the most misleading figure in the scheme: 870 flags out of 10,000 reads as a system in control, and the inference invited is that the check finds little because little is left to find. A queue shrinks when the check improves and it also shrinks when the two runs grow more alike, and a queue of confirmations of correct answers is cost with no yield.

Building that table is the thing to insist on before any verification scheme ships, and it needs the same ingredient chapter 3's resolution experiment needed: ground truth on a few hundred fields, written down by a person. With it, the check takes an afternoon. Compare the both-wrong count the independence assumption predicts against the count you observe — here, 0.66% predicted against 4.5% observed, a factor of seven — and the gap is a direct measurement of how correlated the two runs are. That ratio is the number to quote in a design review, because it converts an argument about whether the runs are independent into an observation about how far from independent they are. Any proposal of the form "run it twice and accept agreement" can be tested this way, whatever the two runs differ in, and the test costs less than building the thing. It settles less than that phrasing suggests, though: it measures the two variants you ran on the documents you have, so it reports how far from independent that pair is on that population rather than whether a family of schemes is sound in general.

Tuning the downscale factor#

The instinct after seeing those numbers is to adjust: downscale less, so the second run is closer to the first. That moves both problems in the wrong direction at once, because the two failure modes have different causes and only one of them is attached to the knob. Errors in the disagreement cell are caused by the information gap between the runs — they exist because the downscale destroyed something the full-resolution run had. Errors in the both-wrong cell are caused by the ink, and the downscale factor is not one of their inputs, since the crossbar is missing from the full-resolution image at every setting.

As the second run approaches the first, its independent errors shrink — good, fewer false flags — but its correlated errors grow as a share of what remains, because the two runs are converging on the same view of the same ink. No error got worse and nothing about the model changed: a population the knob does not touch became a larger fraction of a total the knob is shrinking. At the limit where the second run is identical to the first, they agree on everything including every error, the queue is empty, and accepted accuracy is exactly the original 94%. Downscale harder and you get the opposite failure: a huge queue of the cheap run being blind, the disagreement cell swollen with fields the expensive run reads without difficulty, and the both-wrong cell exactly where it was, because the ink is where it was.

There is no setting in between where the scheme works, because the axis being varied is information content, and ordering two runs by information content is what guarantees the correlation. Without the vocabulary: if one run can only ever see less than the other, whatever defeats the better-informed run defeats the worse-informed one too, and no choice of how much less makes that stop being true.

There is a related idea that is not wrong, so it is worth separating. Running the same model twice under genuinely varied conditions — a different prompt, a different field ordering, a different model family — does produce partially independent errors, and agreement between those runs carries real signal. Repetition earning confidence is how redundancy works everywhere else it is used, and the belief that running something twice makes a result more reliable is correct whenever the two runs can fail differently. The flaw here is not repetition. It is that resolution is the one axis along which two runs are guaranteed to be nested.

And if it turns out the downscaled run is nearly as good as the full one, the conclusion is not that verification works; it is that you have been overpaying for resolution and should ship the cheaper run alone. A second run only earns its cost if it is a peer, and the table shows why: the one cell where it pays for itself is the 150, the fields it got right and the first run got wrong. A run whose input is a subset of the first's holds no information the first lacked, so it can only land there by luck, and the more it agrees the smaller that cell gets.

What actually gives you a signal#

The property you want in a checker is that its errors have a different cause from the errors it is checking. Ranked by that property, on the same 10,000-field sample:

That ordering is one criterion applied five times. A second run on a downscaled image fails outright, since its input is a degraded copy of the first run's and its errors therefore have the same causes and more of them. Asking for the same fields in a different order leaves the pixels alone and changes the sequence the model is continuing, so errors that came from how earlier fields conditioned later ones can move while errors that came from the ink cannot — partial independence, and the catch rate reflects that split. A different model family brings different weights and different failure habits and still reads the same image, which is the shared floor holding it at 62% rather than higher. An OCR cross-check reads those same pixels by a different mechanism entirely, matching character shapes with a system that was never asked to understand the document, so the image properties that defeat it are not the ones that defeat a language model. The arithmetic validators answer the independence question by stepping outside it — they read no pixels at all, which is why theirs is the only row with a cost of none and the only one whose catches include errors whose cause is the ink.

The bottom row is the one to internalise. Line items summing to the subtotal, subtotal plus tax equalling the total, a date falling inside the statement period, a quantity times a unit price matching the amount — these are constraints the document already satisfies, and a misread digit breaks them almost always, whether it was misread at one resolution or at both. A single transposed digit in a line-item amount will fail a sum check regardless of why it was transposed, so this test catches exactly the correlated errors the two-resolution scheme is structurally blind to. It costs nothing per field, it runs in microseconds, and its false flags are the small number of documents that genuinely do not add up, which you want to see anyway.

The chain is worth stating in steps, because it is what points that row at a different population rather than merely making it cheap. A document that adds up was produced by something that made it add up, so its internal sums were consistent before any model read it; a misread digit changes a value; a changed value breaks the sums that value took part in. None of those steps mentions how the value was read or how many times, so the check fires whether a digit was misread once, misread twice differently, or misread identically at both sizes — and that last case is the entire 380 the two-resolution scheme accepts.

def flag_fields(page) -> list[str]:
    flagged = []
    if abs(sum(i.amount for i in page.line_items) - page.subtotal) > Decimal("0.01"):
        flagged += [f"line_item[{n}]" for n in range(len(page.line_items))]
        flagged.append("subtotal")
    if abs(page.subtotal + page.tax - page.total) > Decimal("0.01"):
        flagged += ["subtotal", "tax", "total"]
    for n, i in enumerate(page.line_items):
        if i.quantity is not None and abs(i.quantity * i.unit_price - i.amount) > Decimal("0.01"):
            flagged.append(f"line_item[{n}]")
    return sorted(set(flagged))

Where validators run out#

Validators only exist where the data has redundancy. A receipt has arithmetic; a supplier name does not, and neither does a handwritten note, a chart's trend, or a free-text description. For those fields there is no cheap independent check, and the honest options are a genuinely different second reader, a human, or an accepted error rate that you measure and publish rather than paper over.

The general form of the question is worth keeping. Any scheme that converts repetition into confidence is making a claim about the causes of its own errors, and that claim is checkable in an afternoon with ground truth you probably already have. Predicted both-wrong against observed both-wrong is the test, on the pair of variants you actually intend to ship. Manufacturing a confidence signal out of a model that does not produce one remains an unsolved problem, and every scheme that appears to solve it cheaply is worth the ten minutes it takes to ask what would have to be independent for the arithmetic to hold.

That question is the through-line of this whole topic. An image is not a free input: it has a token cost you can compute before you send it, a legibility budget you can measure field by field, a pricing staircase whose steps do not line up with the resolutions your client happens to produce, and a detail setting that changes what the model can see rather than how hard it tries. When you go to build with it, the order that keeps you out of trouble is the order these chapters came in. Label every image so the model knows which one you mean. Compute what it costs before the invoice tells you. Measure which of your fields survive a resize, on your own documents, with ground truth you wrote down. Find the boundaries of your provider's step function so a resize is a decision rather than a hope. Choose the detail level per task, at the site where the prompt is written. And gate the output on something that does not depend on the model having read correctly — because that gate, not the model's own agreement with itself, is what makes the rest of the system safe to trust.