Paired comparison and the winner's curse

Two prompts, one in production and one proposed, are compared by running both over the same labelled cases and reading the two numbers. That comparison is where most prompt decisions are actually made, and it is unreliable in two directions at once. The first is ordinary measurement noise: a score is computed from a sample of cases, on an endpoint that does not repeat itself exactly, with a judge that is itself a model. The second is the winner's curse — the proposed prompt is not a prompt somebody wrote and tested once, it is the survivor of a search, and the maximum of many noisy measurements sits above the truth by an amount that grows with how many were taken.

This chapter works both corrections on one decision, and ends on a third reading that the arithmetic will lead you to dismiss too quickly: a flat number sometimes means the eval is not looking at the thing that changed.

Run-to-run spread with nothing changed#

The first thing to measure is not the new prompt. It is the same prompt file, the same cases, the same model endpoint, five times in a row, at temperature 0 throughout — the sampling setting that makes the model always take its highest-probability next token — with the LLM judge, the second model call that grades each extraction against its label, also at temperature 0.

$ for i in $(seq 5); do python -m evals.dev_run --prompt prompts/extract@v30.md; done
run 1  0.840   (168/200)
run 2  0.835   (167/200)
run 3  0.845   (169/200)
run 4  0.830   (166/200)
run 5  0.850   (170/200)

Four cases of spread from a prompt that did not change. Temperature 0 is not a determinism guarantee on a hosted endpoint: batched inference schedules requests together and floating-point reduction order varies with batch composition, so logits shift in the last few decimal places and ties at the top of the distribution resolve differently from one call to the next. Retrieval adds its own tie-breaking when two chunks score identically.

What makes a shift in the last decimal places matter at all is that only the ordering of the scores is observable. A change of a millionth is invisible everywhere except where it takes one score past another, and at that point the fragment that gets picked changes, and from there the two runs are writing different text. Most requests never come near such a crossing. The ones that do are the ones that were close — and a case that was close for the model is usually a case that was close for the judge as well, which is why the run-to-run spread concentrates in exactly the cases a comparison depends on. The judge, even at temperature 0, is subject to the same effects, and it is the judge that turns a borderline extraction into a 1 or a 0. Providers also update served models without a version bump on the endpoint you are calling. A two-point difference is inside that spread before any statistics get involved.

Comparing the prompts on the cases#

Comparing 0.84 against 0.86 throws away the fact that both numbers came from the same 200 documents. Most of those documents are answered identically by both prompts, and those cases carry no information about which is better; they only inflate both totals. The comparison worth making is per case.

import numpy as np

def paired_verdict(a: np.ndarray, b: np.ndarray, n_boot: int = 20_000, seed: int = 0):
    """a, b: 0/1 arrays of per-case outcomes for the two prompts, same cases."""
    assert a.shape == b.shape
    n = a.size
    b_wins = int(((b == 1) & (a == 0)).sum())
    a_wins = int(((a == 1) & (b == 0)).sum())
    delta = (b_wins - a_wins) / n

    # bootstrap over cases, resampling the pair together
    rng = np.random.default_rng(seed)
    idx = rng.integers(0, n, size=(n_boot, n))
    boot = (b[idx] - a[idx]).mean(axis=1)
    lo, hi = np.percentile(boot, [2.5, 97.5])

    # McNemar: under the null the discordant pairs split 50/50
    disc = b_wins + a_wins
    se = 2 * np.sqrt(disc * 0.25) / n
    return {
        "agree": n - disc, "b_wins": b_wins, "a_wins": a_wins,
        "delta_pts": 100 * delta, "se_pts": 100 * se,
        "ci95_pts": (100 * lo, 100 * hi), "z": delta / se,
    }

print(paired_verdict(a_outcomes, b_outcomes))
# {'agree': 186, 'b_wins': 9, 'a_wins': 5,
#  'delta_pts': 2.0, 'se_pts': 1.87, 'ci95_pts': (-1.5, 5.5), 'z': 1.07}

Of 200 cases, 186 come out the same way under both prompts. Fourteen disagree, the proposed prompt is right on nine of them and the incumbent on five, and that net of four cases is the entire two-point gap. Under the hypothesis that the two prompts are equally good — the null hypothesis the McNemar test in that function assumes — those fourteen discordant cases are fourteen coin flips, and the chance of a split at least as lopsided as 9-5 is about 28%. Counting that is a matter of listing outcomes: fourteen flips can land nine-five, ten-four, eleven-three and so on up to fourteen-nil, in either direction, and adding up how likely all of those are together comes to about 28 in a hundred. Both directions are included because the question asked before the run was whether the two prompts differ, not whether the proposed one is better — counting one direction only would be answering a question chosen after seeing which way the split fell. The bootstrap in the same function re-draws the 200 cases with replacement twenty thousand times and recomputes the difference on each draw, which traces out how much the answer depends on which cases you happened to label. The 95% confidence interval it produces, meaning a range constructed wide enough to cover the true difference in 95 out of every 100 repetitions of this exercise, runs from minus 1.5 points to plus 5.5, which includes zero comfortably and also includes the proposed prompt being meaningfully worse.

The se line in that function is a few operations and each of them is worth having, because the same calculation reappears in three later chapters. Under the hypothesis that the two prompts are equally good, every discordant case is a coin flip about which of them gets it right. The count of heads in a run of coin flips has a standard deviation equal to the square root of the number of flips times a quarter, so fourteen discordant cases give the square root of fourteen quarters, a little under two cases. But the quantity under test is not that count. It is the difference between the two directions, and moving one case from one side to the other changes that difference by two rather than by one — hence the factor of two in the code, and a spread of about three and three quarter cases. Dividing by the 200 cases turns cases into a rate, and reading the rate as points gives 1.87.

The observed difference gets exactly the same treatment. Nine cases one way and five the other is a net of four cases out of 200, which is the two points. Four cases of signal against three and three quarter cases of noise is the z of about 1.07 in the output, and a z near one is the arithmetic saying the gap is about the size of the wobble.

Pairing is worth doing even when it does not rescue the result. The standard error on the paired difference is 1.87 points against 2.5 points for either score taken on its own, because the 186 agreeing cases stop contributing noise once you stop counting them on both sides. On a set where two prompts disagree on only 3% of cases rather than 7%, the reduction is much larger, and pairing is often the difference between a resolvable comparison and an unresolvable one.

The reason sits in how the two calculations are assembled. An unpaired comparison forms two scores, each carrying noise from all 200 cases, and then subtracts them — and the noise in a difference of two independent estimates is larger than the noise in either one alone. A paired comparison never forms the two scores at all. It looks at the 14 cases that came out differently and ignores the 186 that did not, so the only cases contributing uncertainty are the ones contributing signal. Using fewer cases would normally mean more noise, and here it means less, because the cases dropped were adding noise to both sides at once and cancelling on neither.

The maximum of fourteen measurements#

The second correction has nothing to do with either prompt individually and everything to do with how the proposed one was chosen. It was the survivor of fourteen, selected because its number came out highest, and selection alone lifts a winner above the truth by an amount that can be read off a table.

Fourteen equally-good variants produce a winner about 3.2 points above the truth, on average, purely by selection. An observed gap of 2.0 points is smaller than what the selection process would be expected to manufacture from nothing. That does not prove the new prompt is worse. It does mean the observed number is fully accounted for by the way it was picked, and there is no residue left over that needs a real improvement to explain it.

Two cautions come with that table, because it invites one wrong move. It is not a correction to subtract from an observed gap. Each row is an expectation taken over hypothetical runs in which every variant was equally good, so it says what a search of that breadth manufactures on average out of nothing — a benchmark for the observed gap to be measured against, not a quantity to deduct from it. Subtracting 3.2 from 2.0 and reporting a negative improvement would be inventing a measurement that nobody took.

The way to use it is as a threshold fixed before the search starts. Compute your own standard error from your own discordant rate, read the multiplier for the number of variants you intend to try, multiply the two, and treat the product as the gap a result has to clear before it is worth a second look. Setting it beforehand rather than afterwards is what stops the number of variants being chosen to suit the answer, which is the same discipline as pre-registration and is available for free.

This is where the intuition most people carry is wrong, and the wrong version is tempting because it is how measurement usually works. If you measure a table five times with a tape and take the average, more measurements give a better estimate. If you measure fourteen tables once each and report the longest, more tables give a longer answer, and the excess grows with the count. Prompt search is the second situation, and the selection step converts search effort into apparent score.

The comparison has a limit, and naming it heads off the obvious objection. Fourteen tables really do have different lengths, so reporting the longest is a true statement about a real maximum. Fourteen prompt variants may genuinely differ too, and the claim is not that they are identical. It is that the ordering among them is settled by measurement noise whenever their true differences are smaller than that noise, so the one on top is chosen mostly by which cases happened to fall its way. What the comparison is carrying is the direction of the bias: more measurements of one thing sharpen an estimate, and more things measured once each inflate a maximum.

How many cases a two-point difference needs#

The obvious response is to demand a bigger set, so it is worth knowing the price of resolution. For a paired comparison where the two prompts disagree on about 7% of cases, the number of cases needed to detect a given true difference with 80% power — an 80% chance that the test comes back significant when the difference really is there — at the 5% significance level comes out as follows.

Those four rows are one relationship, and seeing the relationship is worth more than the numbers. The cases required scale with the inverse square of the difference you want to detect, because that difference has to clear a noise floor which itself only shrinks with the square root of the cases. Halving the difference you are chasing therefore quadruples the bill. Check it against the table: 5 points to 2 points is a factor of two and a half in the difference, and 220 cases to about 1,370 is a factor of a little over six, which is two and a half squared. From 2 points to 1 point is a doubling, and 1,370 to 5,500 is a quadrupling. Nothing anywhere in that arithmetic depends on the quality of your labels or the cleverness of your judge, which is why the conclusion it supports is about which questions to bring rather than about how to measure better.

Two hundred cases can resolve a five-point change and cannot resolve a two-point one, and that is a statement about arithmetic rather than about your eval set's quality. Getting to 1,370 labelled cases at the four-minutes-per-case rate from topic 3.4, on building a golden dataset, is about 91 hours of labelling for one axis of one feature, which is a real project rather than an afternoon. Most teams should conclude that differences under roughly five points are outside what their eval can adjudicate, and plan their work around landing changes big enough to see.

Refusing to ship inside the noise floor is a different discipline from running a significance test on every change and reporting the p-value, and the difference matters because the second becomes ritual very quickly. A p-value near 0.28, which is the 28% computed above and means the probability of a gap this lopsided arising from two prompts of equal quality, is not a verdict that the new prompt is useless; it is a statement that this measurement cannot tell. What the noise floor gives you is a rule about which questions to bring to the eval at all, and the practical effect is on planning: work aimed at half-point improvements is work whose outcome you will never observe.

Spending a held-out read on a question it cannot answer#

Suppose you push on and spend one read of an independent set on the new prompt, of the kind the previous chapter budgeted. It comes back at plus 0.4 points. That looks like confirmation — the sign is right, the improvement transferred, the independent set agrees.

It is 0.4 points against a standard error of roughly 1.87 points on a 200-case set, which is 0.21 standard errors. The two-sided probability of seeing a gap that large or larger from two identical prompts is about 83%. Both steps are the ones already done above on larger numbers: four tenths of a point divided by a standard error of nearly two points is about a fifth of a standard error, and a gap of a fifth of a standard error or more, counted in either direction, turns up more than four times in five when nothing whatsoever is going on. A single run has no error bar attached to it at all, so the number arrives looking definite and carries no information about its own precision. Reading any positive value as confirmation makes an independent set a rubber stamp: under that rule it approves roughly half of all genuinely worthless changes, and each approval costs one of the reads that cannot be got back.

That half is not a pessimistic estimate; it is what symmetry forces. If a change is genuinely worth nothing, the difference you measure is noise centred on zero, and noise centred on zero comes out positive half the time. A rule that accepts any positive number therefore accepts half of everything worthless that reaches it, and it does so while producing a figure and a sign that both look like evidence. The rule is not weak because its threshold is too low. It is weak because it has no threshold in it at all.

The defensible version of the same spend is to decide in advance what result would change your mind, then run enough to get it. Three runs of each prompt, paired, gives a difference with a spread you can quote, and a pre-registered rule — ship only if the paired difference clears plus 3 points with a lower bound above zero — makes the read decisive in both directions. That is four to six runs of budget instead of one, which is why the question of whether a change is worth testing independently at all should be settled first on the set you are allowed to iterate against.

Small real gains, and how to bank them#

Small real gains exist and they compound, which is the honest objection to a hard noise floor. A prompt change worth a genuine 1.5 points is worth having, and four of them across a quarter is a six-point move an eval can see even though none of the individual steps was measurable. The way to bank those is to batch them: land several small independent changes together, measure the bundle against the baseline, and accept or reject the bundle. That gives up the ability to attribute the gain to any one edit, and it converts four unmeasurable decisions into one measurable one.

The instrument argument has a legitimate form and a fraudulent one, and they are separated by a single test. "The improvement is obvious and the metric is missing it" is legitimate exactly when you can name the property that no case in the set scores, and it then becomes a work item: name the property, add the cases, measure again. If nobody can name it after trying, what remains is the impression a change makes on the person who wrote it, which is the least reliable evaluator available and the one every prompt author has continuous access to.

The batching answer also has a limit worth stating. A bundle that comes back flat leaves you with no idea which of its four changes was worthless and which two cancelled each other out, so bundles should be built from changes that are independent in what they touch. Four edits to the same instruction are not a bundle, they are one edit with four drafts, and measuring them together answers nothing.

What all of this amounts to is that a score difference is a claim requiring two deductions before it means anything: subtract the run-to-run spread, then subtract what a search of that breadth manufactures on its own. What survives both is a candidate for a real effect, and on most weeks nothing survives, which is a correct result rather than a failure of the method. The cases where the number is flat and the change is nonetheless real are identified by naming the property the eval never scored, not by asserting that the eval is somehow behind the times.

Individual decisions are now on firmer ground, and a longer-running failure remains that no single decision reveals. A team that makes thirty defensible-looking calls over six weeks, each plausible on the day, can still arrive somewhere the numbers do not support, and the evidence for that lives in the shape of two curves over time rather than in any one comparison.