Paired comparison and confidence intervals

A release gate is a decision procedure, not a number. Something has to either block or allow a release on a Thursday afternoon, from a job that finishes before anyone goes home, and knowing that a three-point move sits near the edge of the noise on 200 cases does not by itself produce that decision. Building one means choosing four things: a comparison, a confidence level, a budget in model calls, and a list of inputs the job refuses to let move between the two arms.

All four are decided against a pinned case set: a fixed collection of scored inputs, identified by content hash, that both arms of a comparison run against unchanged. Of the four, the comparison is the choice that matters most, and it is the one usually made without noticing a choice was available. The obvious move — score arm A, score arm B, subtract the rates — throws away most of the evidence the run already collected, because it discards which case produced which result.

Only the cases that disagree carry information#

The usual gate reads one number from this release and one from the last, and treats them as two independent measurements. That is what makes the interval so wide, and the two arms are not independent: every case in the pinned set was scored under both prompts, and most of them behaved identically under each.

Cross-tabulate rather than subtract:

The margins reproduce the familiar figures: v6 at 166/200 = 0.830, v7 at 160/200 = 0.800, a drop of three points. The interior says something the margins cannot. A total of 190 cases returned the same verdict under both prompts — 158 that pass either way, 32 that fail either way — and contribute nothing to the question of whether v7 is worse. All the evidence lives in the ten discordant cases: eight that v6 passes and v7 fails, two that go the other way.

"Contribute nothing" is a strong claim and it is arithmetically exact rather than a manner of speaking. The difference between the two pass rates is v6's passes minus v7's passes, divided by the number of cases. Both counts include the 158 cases that pass under either prompt, so those 158 cancel in the subtraction. Neither count includes the 32 that fail under both, so those never entered. What is left is the eight that only v6 passes minus the two that only v7 passes, which is six, and six out of 200 is the three-point drop the margins reported. The headline figure was never a summary of 200 cases. It was a summary of ten, dressed as one.

That is why the paired standard error is so much smaller. For a paired difference of proportions it is √(b + c − (b−c)²/n) / n, where b is the count of cases the old prompt passes and the new one fails, c is the count going the other way, and n is the number of cases in the pinned set. With b = 8, c = 2 and n = 200 that gives about 1.6 points, against 3.9 points for the unpaired comparison of the same two numbers. Same 400 model calls, same data, an interval two and a half times tighter, purely from not throwing away the pairing.

The shape of that expression follows from the paragraph above rather than from a textbook. Since the difference is (b − c)/n, the only quantities that can wobble between imaginary repetitions of the experiment are b and c. So the uncertainty is built from the discordant counts and not from n: b + c is the ten cases that could have landed either way, the term subtracted from it is a small correction — thirty-six divided by 200, well under one — and the square root of what remains is a little over three. Dividing by the 200 cases converts that back into a rate, which lands at 1.6 points.

Read against the unpaired figure, the source of the improvement is visible. The unpaired standard error is the one from the previous chapter, built from p(1−p)/n on each arm and combined by adding squares: it treats all 200 cases in each run as free to move, because two independent samples of 200 genuinely are. The paired version knows that 190 of those cases were the same cases and behaved identically, so they were never free to move at all. The gain does not come from a cleverer formula. It comes from an experiment that scored both prompts on the same inputs, which is a choice made when the job was written and cannot be recovered afterwards from two stored numbers.

Ten discordant cases out of 200 is also worth reading directly, before any formula. Under the null hypothesis — the boring explanation being ruled out, here that the prompt change made no difference at all — each discordant case is a coin flip, and seeing 8–2 or more extreme has probability 2 × P(X ≤ 2 | n = 10, p = 0.5) = 112/1024 ≈ 0.11. About one time in nine a change that did nothing at all would produce a split this lopsided. Suggestive, short of conclusive, and considerably more informative than "the number went from 0.82 to 0.79".

That probability can be counted out by hand, which is the best reason to do it once. Ten discordant cases, each landing one way or the other, give two possibilities per case and therefore two multiplied by itself ten times — 1024 equally likely patterns, equally likely precisely because the null hypothesis says each case is a fair coin. Now count the patterns at least as lopsided as the one observed. Exactly zero of the ten going the new prompt's way can happen in one way. Exactly one can happen in ten ways, since any one of the ten cases could be the odd one out. Exactly two can happen in forty-five ways, which is the number of pairs you can choose from ten. One plus ten plus forty-five is fifty-six patterns as extreme as or more extreme than 8–2 in that direction.

Doubling gives the 112, and the doubling is a decision rather than an inevitability. The question being asked is whether the prompt change made any difference, not whether it made things worse, so a split of 2–8 in the opposite direction would have been equally surprising and equally worth counting. With fifty-six patterns on each side, that is 112 out of 1024, a little over one in nine. Had the question been the one-sided "did it get worse", the answer would have been half that — which is why a p-value should never be quoted without saying which question it answered.

Turning the spread into a confidence interval#

The interval follows from the paired standard error. A confidence interval is a range of values a measurement is consistent with, produced by a recipe that brackets the true value a stated fraction of the time — a 95% interval is built so that, over many repetitions of the whole experiment, nineteen intervals in twenty contain the truth. It replaces a single number with the honest width around it. Here the 95% interval on the paired difference is −3.0 ± 1.96 × 1.6, or roughly [−6.1, +0.1] points. The unpaired version of the same comparison gives [−10.6, +4.6]. Both intervals contain zero; the paired one only just, and it excludes any regression worse than about six points, which is already a useful statement.

Closing the remaining gap uses the cheap lever: each case can be scored more than once. Instead of a boolean per case, run every case k times under each prompt and record a per-case pass rate — 5/5, 3/5, 0/5. The per-case difference becomes a continuous quantity, boundary cases stop being coin flips that land in one bucket, and the paired comparison gets a much finer signal from the same 200 cases.

import random
import statistics

def paired_bootstrap(diffs: list[float], iters: int = 10_000) -> tuple[float, float]:
    """95% CI on the mean per-case difference, resampling cases with replacement."""
    n = len(diffs)
    means = sorted(
        statistics.mean(random.choices(diffs, k=n)) for _ in range(iters)
    )
    return means[int(0.025 * iters)], means[int(0.975 * iters)]

# rate_v7[i] and rate_v6[i] are each case's pass rate over k=5 repeats
diffs = [rate_v7[i] - rate_v6[i] for i in range(len(cases))]
print(f"mean diff {statistics.mean(diffs):+.3f}  95% CI {paired_bootstrap(diffs)}")

mean diff -0.028  95% CI (-0.041, -0.015)

The interval now excludes zero. v7 is worse by somewhere between 1.5 and 4.1 points on this case set, and the release conversation has an answer rather than an argument. The bootstrap — draw 200 cases at random from your 200, with replacement, recompute the mean difference, do that ten thousand times, and read the spread of the results — resamples cases rather than runs, which is the right choice: it treats the 200 cases as the sample they are, so the interval it reports is about the population of inputs rather than merely about these 200.

Why that procedure produces an honest interval is worth one paragraph, because on first reading it looks like getting something for nothing. The quantity you want is how much the mean difference would move if you had sampled a different 200 cases from traffic. You cannot go and get those other samples. What you have is one sample, which is your best available picture of the population it came from — so drawing from it, with replacement so that any case can appear more than once and the draw stays the same size, imitates the act of sampling from the population itself. Each of the ten thousand resamples is one imaginary repetition of the experiment. The spread across them estimates the spread you would have seen across real repetitions, and cutting off the lowest and highest fortieth of the resampled means leaves the middle 95%.

The limit of the trick is the same as its premise, and it is the one to remember. The bootstrap can only resample what is in the sample. If a category of traffic is absent from the 200 cases, no amount of resampling invents it, and the interval will be confidently narrow about a population that does not include it. It measures sampling noise honestly and says nothing at all about coverage — a distinction the end of this chapter comes back to.

What each gate design costs per release#

With Claude Opus 5 at $5 per million input tokens and $25 per million output, a case in this suite runs about 1,400 input and 220 output tokens, or roughly $0.0125 per call. That makes the budget question concrete:

Twenty-five dollars and seven minutes for a release gate is not a difficult sell, and the middle row is the one to notice: re-running the old prompt alongside the new one, rather than comparing against a stored number from three weeks ago, is the single cheapest improvement available. It doubles the call count and removes every source of drift between the two measurements at a stroke.

What the gate does with the interval is a separate choice, and both options are defensible. Blocking on the lower bound — refuse the release if the lower end of the 95% interval on the paired difference is below −2 points — is conservative, and it will occasionally hold a release for a change that turns out to be neutral. Reporting only, with the interval printed in the release channel and a human deciding, is faster and works while a team is still calibrating what a two-point move means for their users. What does not work is blocking on a point estimate, because a point estimate has no honest reading and the gate ends up encoding whichever noise the job happened to draw.

Holding everything else still#

Every arrow in that diagram assumes one thing: that the only difference between the two arms is the prompt. When a gate reports a drop and nobody changed the prompt in a way that could explain it, the cause is almost always something on this list.

The first and the fourth cause most real incidents. A case set anyone can edit stops being a measuring instrument the moment somebody adds fifteen hard cases from last week's escalations, and the resulting drop looks precisely like a regression. A floating alias is worse because it leaves no trace: the report says 0.83 last month and 0.79 today, git says nothing changed, and the honest answer is that the thing being measured moved underneath the measurement.

Where the gate stays blind#

Pinning is the control for drift, and it creates its own problem over time. A case set that never changes is a case set the prompt gets tuned against — every iteration that fixes a failing case makes the set slightly less representative of traffic, until the reported rate measures fit to 200 remembered inputs. The standard mitigation is a rotating holdout, a second set of cases kept out of the tuning loop entirely so nobody has ever iterated a prompt against it: keep the pinned set for release comparisons, refresh a second sample from recent traffic every month or two, and treat a widening gap between the two as the signal that the pinned set has aged out. That gap is the number that says when to re-sample, and a team which never looks at it will eventually be gating on a set from launch day.

The gate can also miss a real regression entirely, in two ways no amount of statistical care fixes. If the failure lands in a category the pinned set under-represents — the long, false-premise conversations that a hand-built sample never contains, three of which are in a 200-case set drawn from traffic — then a regression doubling the failure rate in that category moves the headline number by less than a point and hides comfortably inside the interval. And if the failure mode is one the checks do not look for, the rate does not move at all: a referential check confirms that every cited id was among the ids retrieved for that query, so an answer citing a real document to support a claim that document contradicts passes the gate at any sample size and any confidence level. The interval is honest about sampling noise and silent about coverage.

A gate built this way is a small amount of arithmetic wrapped around one structural choice. The choice is to run both arms in the same job over the same pinned inputs, which is what makes a paired difference available and what makes every drift in the table above impossible rather than merely unlikely. The arithmetic is then a bootstrap over per-case differences, an interval, and a rule about the lower bound. What comes out is a sentence a release manager can act on — worse by 1.5 to 4.1 points on these cases — instead of two rates and an argument about whether three points is a lot.

There is one assumption underneath all of it, sitting there since the cross-tabulation. Both arms were scored by the same yardstick: the same requirement checks, applied identically to v6 and v7, authored before either existed. That symmetry is what makes a paired difference mean anything. The next chapter takes a comparison where it does not hold — forty stored replies captured from the model currently in production, used to grade a candidate replacement.