The accuracy-versus-N curve

An accuracy-versus-N chart rising left to right is the most persuasive artefact in this whole area and the easiest one to draw badly. Run your sampling loop at N=1, then 3, then 5, 9 and 15, plot five points, and the line goes up. It goes up because the technique works, and it also goes up when the differences between the last three points are smaller than the noise in how they were measured, because a chart with no error bars cannot show the difference between those two situations.

Two things fix it, and both are cheap. Measure every setting from one collection of draws so that the points share their data and their provider conditions. And put the cost and latency of each setting in the same table as the accuracy, because the question a curve is drawn to answer is never "does accuracy rise with N" — it always has — but "is the next increment worth buying".

Measure the curve once, not five times#

The obvious way to produce the chart costs 33 calls per case, at 1 + 3 + 5 + 9 + 15, and gives you one estimate at each setting. The better way costs 15 calls per case and gives you a distribution at every setting. Draw fifteen samples per case once, store them all, and evaluate any N by subsampling from the fifteen you already have.

Those two figures come from adding up what each plan buys. Running the settings separately means a separate fan-out for each one — one draw for N=1, three for N=3, five for N=5, nine for N=9, fifteen for N=15 — and adding those five call counts gives 33 calls for every case in the eval. The second plan draws fifteen and stops, because fifteen is the largest setting on the list and every smaller setting is a subset of fifteen. A vote at N=5 needs five draws of the same case taken under the same conditions, and nothing about those five requires that no sixth draw was ever taken. Picking five of the stored fifteen therefore produces a legitimate N=5 vote.

Doing that once would give one estimate per setting, which is all the expensive plan gives either. The gain is in doing it repeatedly. There are many different ways to pick five draws out of fifteen, each one an equally valid N=5 experiment, so drawing a fresh five a few hundred times produces a few hundred N=5 accuracy figures instead of one. The repeats argument in the function below is what sets how many, rng.sample(draws, n) is the line that does the picking, and the spread of the resulting scores is what the error bars are made of.

import random
from collections import Counter

def vote_accuracy(samples: dict[str, list[str]], gold: dict[str, str],
                  n: int, repeats: int = 200, rng=random.Random(7)) -> dict[str, float]:
    """Estimate vote accuracy at a given N by resampling stored draws."""
    scores = []
    for _ in range(repeats):
        correct = 0
        for case_id, draws in samples.items():
            picked = rng.sample(draws, n)           # n of the 15 stored draws
            winner = Counter(picked).most_common(1)[0][0]
            correct += (winner == gold[case_id])
        scores.append(correct / len(samples))
    scores.sort()
    return {
        "mean": sum(scores) / len(scores),
        "p05": scores[int(0.05 * len(scores))],
        "p95": scores[int(0.95 * len(scores))],
    }

The subsamples are not fully independent, since a case's fifteen draws are reused across repeats, so this understates uncertainty slightly compared with fifteen genuinely fresh runs. Against that it costs a fifth as much, gives you every N on identical data, and removes the possibility that the N=9 run happened to hit a better hour on the provider's fleet than the N=3 run. When the decision is where a curve flattens rather than whether a two-point difference is real, that trade is the right one — and it can be validated by re-collecting the whole set a few times, weeks apart, and checking that the spread across genuinely independent repeats matches what the resampling predicted.

The tempting objection is that five separate runs are the more honest experiment, on the grounds that they share nothing and independence is usually what makes a measurement trustworthy. That objection is backwards here, and the reason is what each design holds fixed. Separate runs let two things vary at once: which draws you happened to get, and what the provider's fleet was doing that hour. A difference between the N=3 run and the N=9 run then carries both, and no analysis afterwards can say how much of it was the setting. Resampling one stored collection holds the cases, the draws and the hour fixed at every point on the curve, so the only thing differing between two points is the one thing being varied. The price is that the uncertainty you measure describes subsampling rather than a fresh collection, which is precisely why the re-collection weeks apart is worth the calls.

Reading the curve with its spread#

Every row is the same 1,200 cases and the same stored draws, read at a different setting. The spread column is the one doing the work, and it is worth being exact about what it measures: it is how far the accuracy figure in its own row moves when nothing changes except which draws the resampling happened to pick. That makes it the yardstick for every comparison across rows. A gap between two settings is a finding only if it is bigger than the wobble inside one setting, because if it is not, the two rows could swap places on a re-run with nothing about the system having changed.

Read the last three rows together. Between N=5 and N=15 the measured accuracy rises by 0.4 points while the spread across repeats at either setting is 0.4 to 0.5 points, so the difference is inside the noise of the measurement, and the honest statement is that this experiment cannot distinguish 5 from 15. Between N=1 and N=3 the gain is 4.1 points against the same spread, which is a difference the experiment resolves comfortably.

Both differences are one subtraction on the accuracy column: 89.4 − 89.0 = 0.4 for the first and 88.1 − 84.0 = 4.1 for the second. What separates them is the comparison that follows the subtraction. The first is the same size as the measurement's own wobble; the second is about ten times it. A chart without error bars publishes the subtraction and withholds the comparison.

The cost column makes the comparison decidable rather than aesthetic. Going from 5 to 15 triples generation spend to buy four tenths of a point the measurement cannot confirm exists. Going from 1 to 3 triples spend from a much smaller base to buy 4.1 points that it can. The accuracy number by itself supports neither conclusion, which is why it should not travel without the two columns beside it.

The multiplication behind that column is worth seeing once, because it fixes the column's shape. Generation is billed per call, so the money for a setting is the number of calls per request, times the price of a call, times the number of requests. The last two are held fixed across the table, which leaves the call count as the only thing moving: $2.40 buys a thousand single-call requests, and three, five, nine and fifteen calls per request cost three, five, nine and fifteen times that, giving $7.20, $12.00, $21.60 and $36.00. Tripling the calls from 5 to 15 triples the money from $12.00 to $36.00 for exactly that reason, and no property of the task is involved.

Treating the two columns as independent facts to be weighed by taste is the stumble here, and it is tempting because both are just numbers going up. They have different shapes, and the shapes settle the argument on their own. Every call added to a request costs what the first call cost, so the cost column climbs by the same amount each time. Every call added to a vote arrives into a tally that already holds several draws, so it can only change the outcome on the shrinking set of cases where the existing draws are near a tie, and the accuracy column climbs by less each time. A constant increment on one side and a shrinking increment on the other cross somewhere, and the table is how the crossing point is found.

The latency column deserves a note, because at first glance it is too flat. Fifteen calls take 2.8 seconds at p95 — the p95 being the time that 95% of requests come in under, so that the figure describes a bad request rather than an average one — when one takes 1.4, which is a factor of two rather than fifteen, and the reason is that the calls run concurrently. What you pay in latency is not the sum of N calls but the maximum of N draws from the per-call latency distribution, and the maximum of a sample grows slowly — logarithmically for well-behaved distributions — as the sample gets bigger. That is the good news about fan-out. The bad news is in the same sentence: your p95 is now governed by the tail of the per-call distribution rather than its centre, so a provider hiccup that makes one call in fifty slow shows up in a sizeable fraction of requests once N is 15, instead of in one request in fifty. The 2.8-second figure also assumes the concurrency is actually available, which stops being true the moment fifteen simultaneous calls per request meet a rate limit and start queueing.

Both halves of that follow from one property of fan-out: a request is not finished until its slowest call is finished, so its latency is the maximum of N draws from the per-call latency distribution rather than their sum. Adding calls pushes that maximum further into the upper tail, and the upper tail thins out — there are progressively fewer slower outcomes left to find — so each extra call raises the expected worst draw by less than the call before it did. That is why fifteen calls come in at about twice the time of one rather than fifteen times it.

The same property is what makes the tail dangerous, and the arithmetic is the reverse of the arithmetic above. For a request to escape a slow call, every one of its calls has to escape, so the chance of a clean request is the single-call chance of being fast multiplied by itself once per call. A number just under one shrinks steadily under that repeated multiplication, so at a per-call slow rate of one in fifty, roughly three quarters of fifteen-call requests come back clean and the remaining quarter carry a slow call — against one request in fifty when N is 1. The per-call rate has not moved at all. Only the number of chances to hit it has.

Sample size deserves a note too. With 1,200 held-out cases, the binomial standard error on a single accuracy estimate near 89% — how far that estimate would typically move if you had drawn a different 1,200 cases from the same population — is about 0.9 points, which is more than the difference such a proposal usually turns on. Paired resampling over shared draws is far tighter than that, since the same cases and the same generations underlie every setting, which is why the observed spread is 0.4 rather than 0.9. Even so, resolving a genuine 0.4-point difference with confidence would need something closer to 10,000 labelled cases. Recognising that the question is unanswerable at the sample size you have is a legitimate outcome of an experiment, and it argues for the cheaper setting.

That 0.9 is worth being able to reproduce, since it is the number deciding whether a proposal is answerable at all. Scoring an eval is a run of yes/no outcomes, one per case. The variance of a single yes/no outcome is the probability of yes times the probability of no, which is largest near a half and small at either end. Averaging over 1,200 cases divides that variance by 1,200, and a standard error is the square root of a variance. So the recipe is: multiply the accuracy by one minus the accuracy, divide by the number of cases, take the square root. Near 89% that lands at about 0.9 points.

Why the paired spread comes in at less than half of that is the more useful half of the story. The 0.9 answers a question about a different sample of contracts — swap in another 1,200 agreements and the absolute accuracy moves by around that much. The spread column answers a question about the same contracts: hold the cases fixed, change only which draws were picked, and see how far the number travels. A contract that is hard is hard at every setting, so its difficulty appears on both sides of a comparison between two rows and mostly cancels out of the difference. That cancellation is the entire reason a paired design can resolve gaps the absolute error could not.

Shrinking the absolute error enough to resolve four tenths of a point is then arithmetic with no experiment in it. Standard error falls with the square root of the number of cases, so cutting it to a third takes about nine times as many cases — and nine times 1,200 is where a figure near 10,000 comes from.

The plateau tells you something the peak does not#

The shape of the curve is not arbitrary, and reading it as "diminishing returns, as usual" leaves information on the table. Voting converges on the mode of the model's answer distribution as N grows, so the height of the plateau is an estimate of how often that mode is correct, and the gap between the plateau and the N=1 accuracy is an estimate of how much of the single-sample error was dispersion rather than a displaced mode.

Each claim in that sentence has a step underneath it. Every draw is a sample from the same distribution over answers, so the answer carrying the highest probability is the one expected to turn up most often in a batch of draws. As the batch grows, the observed counts settle towards the underlying probabilities, so the answer with the most votes settles onto the answer with the highest probability, which is the mode. Voting therefore does not compute the correct answer. It computes the model's favourite answer, more and more reliably. Where the favourite is correct, more draws help; where the favourite is wrong, more draws make the wrong answer win by a wider margin.

With a plateau at about 89.4%, the model's most probable answer is wrong on roughly 10.6% of cases and no value of N will fix any of it. Single-sample accuracy at 84.0% means 5.4 points of the original 16 points of error came from draws landing off the peak. That is the entire budget the technique has on this task, and N=5 has already collected 5.0 of the 5.4. Anyone proposing more samples is arguing over the last four tenths of a point of a quantity that was capped before the experiment started.

Those figures reconcile, and checking that they do is what separates a reading of the curve from a story told over it. Total error at a single sample is a hundred per cent minus 84.0%, which is the 16 points. The part sampling can reach is the climb from single sample to plateau, and 89.4 − 84.0 = 5.4. The part it cannot reach is a hundred per cent minus the plateau, which is 10.6. Those two pieces add back to 16, so every point of error at N=1 is accounted for as either dispersion or a displaced mode, with nothing left over. And 89.0 − 84.0 = 5.0 says N=5 has already banked well over nine tenths of the reachable part.

The belief the plateau exists to dislodge is that a curve still rising must have room left in it. On a task where the model's favourite answer is right more often than a single draw is — which is exactly what the two ends of this curve report — the curve always rises, because every draw added pulls the tally further towards that favourite. Rising is therefore evidence of nothing. The plateau height says the ceiling was fixed by the model's answer distribution before the first sample was drawn, and the things that move a ceiling are the things that change that distribution: a different prompt, different context, a different model. Sampling climbs to the ceiling and stops.

There is a second reading available if you compare the observed curve against what independent errors would predict.

from math import comb

def independent_prediction(p: float, n: int, classes: int = 9) -> float:
    """Plurality accuracy if wrong answers scattered uniformly over the wrong classes."""
    return sum(comb(n, k) * p**k * (1 - p)**(n - k)
               for k in range(n // 2 + 1, n + 1))

The sum in that function is the probability that a majority of n draws is correct, assembled one case at a time. comb(n, k) counts the ways of choosing which k of the n draws are the correct ones. p**k is the chance those particular draws all come out correct, and (1 - p)**(n - k) is the chance the remaining ones all come out wrong. Multiplying the three gives the chance of exactly k correct draws. The range starts at n // 2 + 1 because that is the smallest k that is more than half of n, so summing upwards from there adds the chance of a bare majority, then one better than that, and so on up to every draw correct.

Nothing in that expression uses the classes argument, and that is the point rather than an oversight. Counting majorities needs to know only how many draws came out correct, never how the incorrect ones were distributed. The number of wrong classes matters only for the softer question of whether the correct answer took a plurality without taking a majority, which the function does not attempt.

If each sample were correct with probability 0.84 and the wrong answers scattered across the eight incorrect classes, the plurality of three would be correct about 93% of the time and the plurality of five about 96%. Observed values of 88.1% and 89.0% sit well below both, and the size of the shortfall is a direct measure of how correlated the errors are. A curve that tracks the independent-error prediction closely is telling you the samples are genuinely decorrelated and more of them will keep helping for a while; a curve that flattens far below it is telling you that the shared-error component dominates and the plateau is close. That function also ignores the cases where the correct answer wins a plurality without a majority, so it slightly understates the ideal, which only sharpens the point: the real curve is below a conservative bound on the ideal one.

Reading that shortfall as a statement about the model's accuracy is the mistake to avoid, and it is tempting because a prediction of 93% against an observation of 88.1% looks like a model underperforming. Per-sample accuracy is not what the two sides disagree about: the prediction was built by feeding the observed single-sample rate straight into the formula, so both sides agree exactly about how often one draw is right. What they disagree about is where the wrong draws land. The formula assumes a wrong draw goes somewhere arbitrary among the wrong classes, so wrong draws rarely pile onto the same class and rarely outvote the correct one. The observed curve says they do pile up. The gap measures that piling up and nothing else.

Details that move the number by more than the last two points of N#

Whether N is odd matters more than it looks. At N=4 a two-two split has no plurality, and whatever your code does in that situation — take the first, take the highest-probability sample, fall back to a default — becomes a policy that runs on a real fraction of traffic. On this task, even-N settings produced ties on 6% to 9% of cases, and the tie-break rule was worth about 0.6 points depending on which one was chosen. Odd N avoids the two-way tie and does not avoid three-way splits with two answers at two votes each, which occurred on 1.4% of cases at N=5. A deliberate rule beats most_common(1)'s insertion-order behaviour, and the simplest defensible one is to fall back to the sample the model itself scored highest, if you have logprobs, or to a designated safe class if the task has one.

The parity argument is short enough to check rather than take on trust. With N even, the votes can divide into two equal piles — two and two at N=4 — and there is no plurality to read off. With N odd, two counts cannot be equal and also add to N, because two equal whole numbers always sum to an even number, so a straight two-way tie is impossible. That is the whole of what odd N buys, and it is less than it sounds: three answers at two, two and one votes add to five perfectly well, which is the 1.4% of cases reported above at N=5. Odd N removes the tie between two answers and leaves the tie at the top of three untouched.

An accuracy-versus-N curve is also measured on a fixed held-out set with a fixed prompt and a fixed model version, and every one of those is a moving part in production. A model upgrade changes the plateau height and can change the shape entirely, which means the curve is a measurement with a shelf life rather than a constant of the system. The clause-review team rebuilt it after each model change and found the plateau moved from 89.4% to 92.1% on one upgrade, with the optimal N dropping from 5 to 3 because the newer model's single-sample distribution was already sharper.

So the curve is three artefacts rather than one. A table with spread, cost and latency against N, which says where the buying stops. A plateau height, which says how much of the remaining error is displacement and therefore how much of the work belongs in the prompt rather than in the fan-out. And a set of per-slice curves, which is usually where the actual improvement is, because a single global N is simultaneously too high for the slices that plateaued at N=3 and too low for the ones still gaining at 9.

What the curve does not settle is whether the points are worth having. It reports accuracy against calls, and calls are only a proxy for cost until somebody multiplies by a rate card and a request volume, while accuracy is only a proxy for value until somebody prices what an error does when it escapes. Both conversions are missing from every table here, and without them "89.0% at N=5" is a fact about an eval rather than a basis for a decision.