The accuracy curve flattens sooner than the sample count suggests

The pull request was one line, SAMPLES = 7, and its description was a table: 71% at N=1, 78% at N=3, 79% at N=5, 79% at N=7, measured on the 200-case held-out set. The argument attached to it was that the curve is monotone, the budget covers seven calls, and more samples cannot hurt. Two of those three statements are true. The first one is a claim about four numbers, and the four numbers do not support it.

Converted back to case counts, the results are 142, 156, 158 and 158 correct out of 200. The whole disagreement between N=3 and N=7 is two cases. Two people on the team could label those two cases differently and the entire argument for the change would evaporate, which is a sign that the measurement is being asked to resolve something finer than it can.

Two intervals, and only one of them is the right instrument#

The interval most people reach for is the marginal one on a single proportion. At 158 of 200, the standard error — the typical distance between a measured proportion and the true one — is sqrt(0.79 × 0.21 / 200) = 0.0288, and a 95% interval is roughly two standard errors either side, so about 73% to 85%. That range is wide enough that repeating the measurement on fresh cases would land inside it 95 times in 100, and it contains every point on the curve except N=1.

It is also the wrong tool for this decision, and the reason is worth stating rather than asserting. A marginal interval answers "what is this system's accuracy", so it accounts for all the ways a different sample of 200 cases could have come out — including cases that are hard for every configuration and cases that are easy for every configuration. Those cases move both measurements together, so they contribute nothing to the difference between the two, and an interval that includes their variance is charging the comparison for noise that cancels.

Both systems were evaluated on the same 200 cases, so the comparison is paired and almost all of that noise does cancel. What matters is only the cases where the two disagree.

Between N=3 and N=5, twelve cases differ: seven that N=5 gets right and N=3 gets wrong, five the other way, for the net of two. Under the null hypothesis that the two configurations are equally good, each discordant case is equally likely to fall either way, so the number of wins among twelve discordant cases is a fair coin flipped twelve times. That is McNemar's test for paired binary outcomes, and seven heads out of twelve is an utterly ordinary result for a fair coin — its two-sided probability, the quantity written as p in the tables below, is 0.77. Between N=5 and N=7, six cases differ, three each way, and the probability is 1.0. Between N=1 and N=3, thirty-eight cases differ, twenty-six of them wins for N=3, and the two-sided probability is 0.035.

def paired_bootstrap(a: list[bool], b: list[bool], reps: int = 20_000) -> tuple[float, float]:
    """95% interval on (accuracy of b) - (accuracy of a), resampling cases, not calls."""
    n = len(a)
    diffs = []
    for _ in range(reps):
        idx = [random.randrange(n) for _ in range(n)]
        diffs.append(mean(b[i] for i in idx) - mean(a[i] for i in idx))
    diffs.sort()
    return diffs[int(0.025 * reps)], diffs[int(0.975 * reps)]

paired_bootstrap(correct_at_n3, correct_at_n5)   # (-0.020, +0.040)
paired_bootstrap(correct_at_n1, correct_at_n3)   # (+0.010, +0.130)

The docstring's last clause is the part to read twice. The bootstrap re-draws the 200 cases with replacement, not the model calls, because the question is how much the answer would move on a different sample of cases from the same population — and the same resampled index list is used for both configurations, which is what keeps the comparison paired. Sorting the twenty thousand differences and reading off the values at the two-and-a-half and ninety-seven-and-a-half percentiles gives an interval directly, with no distributional assumption at all.

It agrees with the sign test and gives an interval rather than a verdict, which is more useful for a decision: the N=3 to N=5 difference is somewhere between minus two and plus four percentage points, and that range includes zero comfortably. The N=1 to N=3 difference does not include zero. Everything defensible about this curve is contained in those two statements.

Putting the other costs on the same table#

Accuracy is not the only column that moves with N, and once the other columns are present the decision stops being close.

The latency column is not a measurement error and it is not the mean branch getting slower. Its two figures are percentiles of the request population, so p50 is the latency half of requests come in under and p95 the latency 95% of them come in under.

When you wait for all N samples, your latency is the maximum of N draws from the single-call latency distribution, and the quantiles of a maximum can be written down exactly. The maximum of N draws is at most x precisely when every one of the N draws is at most x. If a single call comes in under x a fraction F of the time, and the draws are independent, all N do so a fraction F^N of the time. So to find the value the maximum beats 95% of the time, set F^N equal to 0.95 and solve for F, which gives F = 0.95^(1/N): the p95 of the maximum of N is the 0.95^(1/N) quantile of a single call.

The same derivation runs at any percentile, which is what the p50 column is. Half of the fan-outs come in under the value that a single call beats the Nth root of a half of the time, and that fraction rises towards one as N grows — so the median of a fan-out drifts up the single-call distribution too, just far more slowly than the tail does. Both columns are the same identity read at two points.

That identity is worth keeping, because it converts a distribution you already have into one you were about to measure. For N=3 the exponent works out at the 98.3rd percentile, and for N=7 the 99.3rd. This service's single-call latency is 1.9 s at p50, 4.6 s at p95, 6.1 s at p98.3 and 8.2 s at p99.3, which reproduces the column exactly. Fanning out therefore drags your tail latency out along the single-call tail, and a provider with an occasional ten-second stall that you saw once an hour at N=1 you will see roughly seven times as often at N=7. What the identity assumes is independence between branches and a common distribution, and the previous chapter on queueing is a catalogue of ways to lose the first of those — when branches queue behind one another, the maximum is a sum rather than a maximum and the prediction understates the tail badly.

The money column is arithmetic on $0.0062 per call. Framed against the accuracy column, N=1 to N=3 buys 14 cases for 400 extra calls across the 200-case set, or one additional correct answer per 29 calls. N=3 to N=5 buys 2 cases for the same 400 calls, one per 200. N=5 to N=7 buys nothing for 400 calls. At the feature's 400,000 requests per month, the gap between the N=3 and N=7 rows is the difference between their per-thousand prices, multiplied by four hundred thousands of requests, which comes to $9,920 a month — spent to move an accuracy figure that the evaluation says is the same number.

The adaptive rule from the aggregation chapter belongs on this table too, because it occupies a row between N=3 and N=5 rather than replacing either. Running three samples and drawing two more only when the three fail to produce a majority spends three calls on 81% of requests and five on the other 19%, averaging 3.38 calls, or $20.96 per thousand against $18.60 for flat N=3.

It scored 158 of 200, matching flat N=5, and that is the expected result rather than a lucky one. A request where three samples already agree is a request where the two extra samples would have had to overturn an existing majority to change the winner, and two samples cannot outvote three. So the top-up can only change the outcome on requests that had no majority — which is exactly the set it is spent on.

The column that gets worse is the tail, at 9.4 seconds p95, because the 19% that top up pay two fan-outs in sequence and the second one is drawn from the same stretched-out maximum-of-N distribution as the first. On this feature the latency budget was the binding constraint and the shipped configuration is flat N=3; on a batch pipeline with no user waiting, the adaptive version is the better buy.

What the shape of the curve was always going to be#

The flattening is not a surprise if you carry the correlation arithmetic from the previous chapter forward. With independent samples at p=0.71, the majority-vote accuracy would be 79.6% at N=3, 85.0% at N=5 and 88.6% at N=7 — a curve that keeps climbing, slowly, for a long way.

Correlated samples do not follow that curve, because the effective sample size grows much more slowly than N does. At ρ=0.17, going from N=5 to N=7 moves the effective sample size from 3.0 to 3.5, and going from N=7 to N=15 moves it to 4.3. The curve you get is the independent curve read at N_eff rather than at N, which is why it flattens at a value determined by the correlation rather than by your budget. Doubling the budget from seven samples to fifteen buys less than one further effective sample, and the independent curve at that point is nearly flat anyway.

That gives you a prediction to check rather than a shrug. Estimate ρ from the observed lift at one value of N, use it to predict the lift at the next, and see whether the prediction holds. If the measured curve keeps climbing faster than the correlated model predicts, your samples are more independent than you assumed and a larger N may be worth measuring. If it flattens earlier, the correlation is worse than estimated and there is probably a systematic misreading in the prompt.

Why the monotonicity was never evidence#

Three of the four points on this curve are statistically indistinguishable from one another, and a sequence of indistinguishable numbers still comes out in some order. Three exchangeable measurements — three numbers with no real difference between the things they measure, so that any ordering of them is as likely as any other — come out in non-decreasing order by chance one time in six, since there are six orderings of three items and one of them ascends. The last three points here are ordered 78, 79, 79, which is one of the orderings that reads as a rising curve. Reporting the four numbers as a trend gives a one-in-six coincidence the appearance of a mechanism.

The related trap is to keep going. If N=7 had scored 80%, the next pull request raises N to 9; when it comes back at 79%, the reading becomes "noise" rather than "the curve was flat all along". A rule that treats increases as signal and decreases as noise will climb forever on a flat curve, because a flat curve produces both in equal measure.

The way out is to fix what the evaluation can resolve before looking at any of the numbers, and that is a calculation you can do in advance. With about 6% of cases discordant between adjacent configurations, 200 cases produce roughly twelve discordant pairs. Reaching p < 0.05 on twelve coin flips requires about ten of them to fall the same way, which is a net of eight cases, or four percentage points on 200. So this evaluation cannot resolve a difference smaller than roughly 4pp, full stop, and every comparison below that threshold will return "cannot tell" no matter how many times it is run.

Widening the set helps at a punishing rate. The minimum detectable effect improves as the square root of the number of cases, so cutting it by a factor of four needs sixteen times as many cases: resolving 1pp on this feature would take roughly 3,200. That is the honest price of settling the N=5 versus N=7 question, and it is almost certainly not worth paying. If the difference genuinely matters — if a single percentage point is worth $9,920 a month plus the latency — then the correct next step is a bigger evaluation set, not a bigger N.

There is a second ceiling underneath the statistical one, and it does not move however many cases you add. A 200-case set annotated once, without adjudication, typically carries a few percent of label error, and label error caps measurable accuracy: if 3% of the labels are wrong, a perfect system scores 97% and two systems differing by 1pp are competing inside the noise floor of the ground truth. Before spending money on more cases, it is worth double-labelling a sample of the existing ones to find out where that floor sits, since a wider set with the same error rate buys precision around a number that is already displaced.

That leaves the 44 cases that N=3 gets wrong, and the more interesting fact about them: 38 of them are still wrong at N=7, and in 31 of those all seven samples returned the identical wrong answer.

Unanimous errors are the signature of a shared cause, and the argument in the previous chapter says what follows. Drawing more samples from one prompt on one model estimates that configuration's mode more precisely, and these cases are ones where the mode is wrong — so the one lever this chapter has been measuring cannot reach them at any budget. The next chapter is about what does move them, and about the moves that look like they should but do not.