18 out of 20 is consistent with anything from 70% to 99%

Two teams presented PoC results in the same portfolio review. The fraud-signals team reported 90% accuracy and the document-classification team reported 90% accuracy, and the room treated the two numbers as equivalent because they were the same number. The fraud team had run 20 cases. The classification team had run 400. Only one of those results could distinguish a system that clears an 85% bar from one that sits at 75%, and the review had no mechanism for noticing which.

The scores were 18 of 20 and 360 of 400. Both round to 90%, and their 95% confidence intervals — the range of true underlying success rates that would produce a result like the observed one reasonably often, rather than only by a fluke — do not overlap in any useful way: the small sample is consistent with a true rate anywhere from about 70% to 99%, while the large one narrows to roughly 86.8% to 92.5%. The first result is compatible with a system that fails the bar badly and a system that is nearly perfect. The second is compatible only with passing.

What 18 of 20 licenses you to say#

Two intervals are worth having on hand, and they disagree slightly in a way that does not matter for decisions. Both are standard recipes for turning a count of successes out of a number of trials into that range of plausible true rates, and they differ only in how cautiously they handle small samples and results near 0% or 100%. The Wilson score interval for 18/20 at 95% confidence runs from 69.9% to 97.2%. The Clopper–Pearson exact interval, which is more conservative, runs from 68.3% to 98.8%. Either way the range covers a system that would be rejected outright and a system that would be approved without discussion.

The frequency framing is more persuasive in a meeting than the interval is. A system whose true rate is 75% — clearly a fail against an 85% bar — produces 18 or more successes out of 20 about 9% of the time, which is roughly one PoC in eleven. A system whose true rate is exactly 85% produces 18 or better about 40% of the time. So "18 out of 20" is an entirely ordinary result for a system sitting at the bar, and a not-especially-rare result for a system well below it. Nothing about the sample was done wrong; twenty draws simply do not contain enough information to separate those hypotheses.

This cuts in the optimistic direction too, which is worth saying because otherwise the argument sounds like pessimism dressed as statistics. A genuinely good system, at a true 94%, still returns 17 of 20 about 24% of the time, and a team that kills a project on 17/20 has done the same thing in reverse. Small samples are not biased downward or upward; they are uninformative in both directions, and the failure mode is treating them as informative in whichever direction suits the room.

How many cases it takes to resolve a bar#

The number of cases you need is not a general property of evaluation; it depends on where the bar sits and how close the observed rate lands to it. The question a frozen set has to answer is whether the lower end of the interval clears the threshold, so the relevant calculation is how many cases push the lower bound above the bar at the rate you expect to observe. Widths in the table are in percentage points, written pp, which is the plain arithmetic gap between two percentages rather than a relative change: 90% minus 85% is 5 pp.

The frozen 200 from the criteria file in chapter 1 was not a round number chosen for tidiness. To hold the 95% half-width at 5 percentage points near a true rate of 85%, the normal approximation gives n ≈ z² · p(1−p) / w² = 3.84 × 0.1275 / 0.0025 ≈ 196, and 200 is that rounded up. Tightening to ±3 points requires about 545 cases, which for invoices means labelling 545 documents by hand at maybe four minutes each — 36 hours of somebody's week — and that trade is the actual reason PoC sets are the size they are. Sample size is a labelling budget question with a statistical constraint attached, not the other way round.

The consequence for a kill criterion is that the sample size and the bar have to be designed together. An 85% bar checked on 200 cases can be resolved if the system lands near 90% or near 78%; it cannot be resolved if the system lands at 86%, because the interval then runs from about 80.6% to 90.0% and contains the bar. Deciding in advance what happens in that band is part of writing the criterion, and the honest options are few: default to fail, or precommit a single bounded extension — 200 more labelled cases, five extra days, same bar — with the extension authorised in the original document rather than negotiated after the result.

from scipy.stats import beta, binomtest

def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
    p = k / n
    d = 1 + z**2 / n
    centre = (p + z**2 / (2 * n)) / d
    half = z * ((p * (1 - p) / n + z**2 / (4 * n**2)) ** 0.5) / d
    return centre - half, centre + half

def clopper_pearson(k: int, n: int, alpha: float = 0.05) -> tuple[float, float]:
    lo = beta.ppf(alpha / 2, k, n - k + 1) if k else 0.0
    hi = beta.isf(alpha / 2, k + 1, n - k) if k < n else 1.0
    return lo, hi

def verdict(k: int, n: int, bar: float) -> str:
    lo, hi = wilson(k, n)
    if lo >= bar:
        return f"PASS  {k}/{n} = {k/n:.1%}  CI [{lo:.1%}, {hi:.1%}]"
    if hi < bar:
        return f"FAIL  {k}/{n} = {k/n:.1%}  CI [{lo:.1%}, {hi:.1%}]"
    return f"UNRESOLVED  {k}/{n} = {k/n:.1%}  CI [{lo:.1%}, {hi:.1%}] spans {bar:.0%}"

print(verdict(18, 20, 0.85))    # UNRESOLVED  90.0%  CI [69.9%, 97.2%]
print(verdict(180, 200, 0.85))  # PASS        90.0%  CI [85.1%, 93.4%]
print(verdict(172, 200, 0.85))  # UNRESOLVED  86.0%  CI [80.6%, 90.0%]
print(verdict(156, 200, 0.85))  # FAIL        78.0%  CI [71.8%, 83.2%]

The third line is the one worth staring at. An 86% result against an 85% bar reads as a pass in every slide deck ever made and is not one; it is an experiment that did not resolve its own question. Printing the word UNRESOLVED rather than a percentage is a small piece of engineering that changes what a review meeting can conclude.

A second source of noise sits underneath the first#

Sampling error is not the only reason the number moves. Running the same 200 invoices through the same prompt and the same model on three consecutive evenings produced 169, 172 and 171 correct — 84.5%, 86.0% and 85.5%. Temperature was zero, which is the sampling setting that tells the model to always take its highest-scoring next token instead of drawing at random among the plausible ones, and which is therefore supposed to make a run repeatable. Inference at scale is not deterministic in practice, because request batching, expert routing in mixture-of-experts models — architectures that send each token through a small subset of specialised sub-networks rather than all of them — and floating-point non-associativity across different hardware all shift the logits slightly, meaning the raw scores the model assigns each candidate token before they are turned into probabilities, and a token near a decision boundary flips. On extraction tasks the effect is small, and it is comparable in size to the margin teams routinely treat as a result.

Prompt sensitivity is larger and lands in the same place. Reordering two instructions in the system prompt, or adding a sentence about date formats, moved the same evaluation by 1.5 to 3 percentage points in either direction with no principled reason to prefer either version. That variance is real and is part of what you are measuring, so the defensible practice is to run the frozen set three times on the configuration you intend to report, and to report mean and range: 85.3% ± 0.8 rather than 86.0%.

Composing the two noise sources gives the practical rule. With 200 cases, sampling uncertainty is around ±5 points and run-to-run variance around ±1, so any result within roughly 5 points of the bar has not been resolved by the experiment, whatever the point estimate says. Teams that internalise this stop arguing about 84% versus 86% and start arguing about whether to buy more cases, which is a question with a price attached and therefore a question that can be settled.

Comparing two variants needs the pairs, not the intervals#

A related mistake shows up as soon as a PoC has two candidates. Model A scores 170/200 and model B scores 175/200, the intervals overlap almost completely, and the team concludes there is no evidence of a difference. Both facts are true and the conclusion does not follow, because the two systems were run on the same cases and the aggregate throws away that pairing.

What matters is the discordant cases: the invoices A got right and B got wrong, and the reverse. On that run there were 11 cases B fixed and 6 cases B broke, seventeen discordant pairs in total. Under the hypothesis that the two systems are equivalent, the split of those seventeen should look like a fair coin, and an exact binomial test on 11 of 17 gives a two-sided p of about 0.33, meaning a split at least this lopsided in either direction turns up about a third of the time when the two systems really are equivalent. That is weak evidence, which is a genuinely different statement from no evidence, and it comes with something more useful than a p-value: a list of six specific invoices that got worse, which can be read.

# a_ok, b_ok: aligned booleans over the same frozen 200, in the same order
b_fixed = sum(1 for a, b in zip(a_ok, b_ok) if not a and b)   # 11
b_broke = sum(1 for a, b in zip(a_ok, b_ok) if a and not b)   # 6
p = binomtest(b_fixed, b_fixed + b_broke, 0.5).pvalue         # 0.332

Reading the six regressions is often where the decision actually gets made. If B fixed eleven scanned invoices and broke six PDF-native ones, that is not a wash; it is two systems with different failure surfaces, and the right move may be routing by document type rather than picking a winner.

What an interval does not tell you#

None of this is statistical rigour in the sense a paper would need. There is no correction for the fact that you tried four prompts and are reporting the best one, which inflates the winner by a point or two, and no attempt to model the dependence between fields within an invoice. That is an acceptable level of care for a spend decision, provided nobody claims more precision than the method supports.

Two limitations matter more than the ones just waived. The first is clustering. The interval formulas assume 200 independent draws, and a frozen set of 200 invoices drawn from 25 supplier layouts contains far less than 200 independent pieces of information about layout-driven failures — if a whole layout parses wrong, eight invoices fail together. For failure modes that track layout, the effective sample size is closer to the number of distinct layouts than to the number of documents, which makes 200 invoices from 25 layouts weaker evidence than it appears. Stratifying the set and reporting per-stratum rates exposes that, and a stratum with four cases in it should never be quoted as a percentage.

The second is that an interval quantifies uncertainty about the rate on the population you measured, and says nothing whatsoever about whether that population resembles the one production will see. A 95% interval of 88.0% to 91.7% on a thousand curated invoices is a precise measurement of performance on curated invoices. December's mailbox will contain seasonal suppliers who were not in the sample, a depot that switched to phone photographs, and a batch of credit notes nobody thought to include. Tight intervals on the wrong population are precisely measured irrelevance, and the conditions under which the number was produced travel forward with it in a way that is easy to miss.