Sizing an eval set

Most eval sets are not sized, they accumulate. A hand-written set from the first month, a batch lifted from a week of production transcripts, a handful added after individual customer complaints, a few that arrived with a refactor and were never labelled properly — and the total is whatever the additions happened to sum to. That number then gets treated as fixed until a budget conversation proposes cutting it, at which point nobody in the room can say what the cut costs.

This chapter derives the number instead. The input is the smallest regression a team would refuse to ship, and the output is a case count, by way of three quantities: the standard error of a single run, the larger uncertainty on a difference between two runs, and the discordance rate — the fraction of cases whose pass or fail verdict flips between two runs of the same unchanged commit, which is the empirical noise floor of the instrument and the thing pairing exploits.

What a score of 84.2 is telling you#

Start with the smallest change a suite can register and count what it does to the number. A suite of 40 cases reports its pass rate in fortieths, so one case changing its verdict — a single answer that passed on Tuesday and failed on Wednesday — moves the reported score by 2.5 points. The same single flip in a suite of 400 moves it by a quarter of a point. Nothing about the assistant differs between those two situations. The leverage one case holds over the total is ten times larger in the smaller set, because each case is a fortieth of it rather than a four-hundredth. Run-to-run noise is a pile of such flips arriving at once, and the standard error is what puts a size on the pile.

An eval reports a pass rate, and a pass rate over a sample is an estimate of a pass rate over a population. The uncertainty on that estimate is the standard error, which for a proportion is sqrt(p(1-p)/n). At a pass rate around 0.85:

from math import sqrt

def standard_error(p: float, n: int) -> float:
    return sqrt(p * (1 - p) / n)

standard_error(0.85, 400)   # 0.0179  -> ±1.8 points
standard_error(0.85, 100)   # 0.0357  -> ±3.6 points
standard_error(0.85, 40)    # 0.0565  -> ±5.6 points

Three pieces of that expression are worth earning rather than accepting, because each one is a place the intuition goes wrong later.

The p(1-p) is the variance of a single case, and it falls out of the fact that a case has only two outcomes. Score a pass as one and a fail as zero. The average of those scores across the set is the pass rate, p, by definition of a pass rate. Variance is the average squared distance from that average, which is the same as the average of the squares minus the square of the average — and squaring a value that is only ever zero or one leaves it unchanged, so the average of the squares is also p. Variance is therefore p minus p squared, which factors into p times one minus p. At a pass rate of 0.85 that is 0.85 multiplied by the fraction that fail, and the two multiply out to a little over an eighth.

Dividing by n is where the size of the set enters, and it rests on variances of independent quantities adding. The sum of n case scores carries n times the variance of one case. A pass rate is that sum divided by n, and dividing a quantity by n divides its variance by n squared, because variance is built out of squared distances and shrinking every distance by a factor shrinks every square by that factor squared. So n lots of p(1-p), divided by n squared, leaves p(1-p) over n.

The square root is a change of units rather than a piece of statistics. Variance is measured in squared points, because it averages squared distances, and a squared point cannot be compared against a 5-point regression. Taking the root brings the quantity back onto the scale the score is quoted on. Putting the three together at a pass rate of 0.85 and 400 cases: a little over an eighth, divided by 400, square-rooted, gives the 0.0179 the code prints, which is the ±1.8 points beside it.

A single run of a 400-case suite reporting 84.2 says the true rate is somewhere around 84.2 ± 1.8 points at one standard error, so roughly 80.6 to 87.8 at two. Cutting to 100 cases doubles that width, because the standard error scales with 1/sqrt(n) and a 4× reduction in cases is a 2× increase in error. That relationship is why sample-size cuts feel cheaper than they are: halving the resolution costs a quarter of the money, which sounds like a bargain right up until the resolution matters.

Where p sits changes the answer too, and in a direction most people guess backwards. p(1-p) is a product of two numbers that always add to one, and a product of two numbers with a fixed sum is largest when the two are equal. So a single case is at its noisiest when a pass and a fail are equally likely, and gets quieter as the pass rate moves towards either extreme. A suite sitting at 0.85 is therefore cheaper to measure precisely than one sitting in the middle, and a suite where almost everything passes is cheaper still — which is not the good news it sounds like. A suite that everything passes has almost no variance because it has almost no information in it. Its cases have stopped distinguishing a working assistant from a broken one, and the narrow interval around its score is reporting how easy the set is rather than how well the system does.

The number you want is a detectable difference#

A gate is not asking what the pass rate is. It is asking whether this run is worse than the previous one by enough to act on, which is a comparison between two estimates, and the uncertainty on a difference is larger than the uncertainty on either side of it. If two runs were independent samples, the standard error of the difference would be sqrt(2) times the single-run error — 2.5 points at n=400 — and detecting a difference reliably takes roughly 2.8 of those. Reliably is a quantity rather than a mood: the 2.8 is the sum of the two multipliers that fix a 5% false-positive rate and 80% power, meaning a real drop of that size shows up in four runs out of five and a run with no regression behind it goes red one time in twenty. That puts the smallest reliably detectable drop at about 7 points.

The sqrt(2) is the rule that variances of independent quantities add, used a second time. A difference is one quantity minus another, and for independent quantities the variances add whether they are being added or subtracted, because variance is built from squared distances and squaring destroys the sign. Two runs of the same size at about the same pass rate carry about the same variance, so the difference between them carries twice one run's worth. The square root of twice a variance is the square root of two times the standard error, and at 400 cases that turns ±1.8 points into a little over 2.5.

The 2.8 is two separate decisions added together, and teams usually make only the first of them. The 1.96 is a threshold measured in standard errors: a difference has to sit at least that far from zero before anyone is willing to call it real, and that particular distance is chosen because a comparison with no regression behind it lands further out one time in twenty, counting both directions. That is what fixes the false-positive rate at 5%. On its own it says nothing about catching regressions, and the reason is worth spelling out. A true drop sitting exactly on the threshold produces an observed difference above it half the time and below it half the time, so a gate tuned only on 1.96 misses half of the real regressions that are exactly the size it was tuned for. The 0.84 is the extra distance the true effect has to sit beyond the threshold for the observed difference to clear it four times in five, and four in five is the 80% power. A regression must therefore measure 1.96 plus 0.84 standard errors to be caught reliably, and 2.8 lots of the 2.5-point difference error is where about 7 points comes from.

That is a bad number, and it is the number most teams are unknowingly operating with. An alert wired to a 4-point drop from a trailing mean, against a 7-point resolution, fires on noise regularly while a genuine 5-point regression slips through often enough to be useless.

The first fix is not more cases. It is that the two runs are not independent samples, because the suite is frozen and both runs score the same cases. What matters in a paired comparison is only the cases whose verdict changed between the runs:

from math import sqrt

Z = 1.96 + 0.84          # 5% two-sided significance, 80% power

def cases_needed_unpaired(p: float, delta: float) -> int:
    return int(Z**2 * 2 * p * (1 - p) / delta**2) + 1

def cases_needed_paired(discordance: float, delta: float) -> int:
    # discordance = fraction of cases whose pass/fail flips between the two runs
    return int(Z**2 * discordance / delta**2) + 1

cases_needed_unpaired(0.85, 0.05)     # 800
cases_needed_paired(0.10, 0.05)       # 314
cases_needed_paired(0.10, 0.03)       # 872
cases_needed_paired(0.20, 0.05)       # 628

Both functions are the same rearrangement done twice, and the rearrangement is short enough to follow line by line. Start from what the gate is being asked for: the drop it must catch, delta, has to be at least Z standard errors of the difference. A standard error of the difference is the square root of some variance divided by n, so the requirement reads delta is at least Z times that square root. Squaring both sides removes the root, and squaring is allowed here because both sides are positive quantities and squaring preserves the ordering of positive numbers. That gives delta squared is at least Z squared times the variance over n. Multiplying by n and dividing by delta squared isolates n on one side, and delta ends up squared in the denominator.

Delta being squared down there is the fact worth carrying away, because it settles a question budgets keep asking. Halving the drop you want to see divides the denominator by four and so multiplies the case count by four. The table below shows it running in the other direction: 100 cases resolve 8.9 points, and 400 cases — four times as many — resolve 4.4, which is half.

What differs between the two functions is only what goes in for the variance. Unpaired, the two runs are separate samples, the difference carries two runs' worth of case variance, and that is the 2 * p * (1 - p). Paired, the two runs score the same frozen cases, so a case that returns the same verdict on both sides contributes exactly nothing to the difference — the two identical values cancel, and they cancel exactly rather than on average, because it is the same case with the same outcome twice. Only the cases that flipped are left, so the variance of the difference is governed by the flip rate alone, and that flip rate is the discordance argument.

Pairing on a frozen set buys the resolution of an 800-case unpaired comparison for 314 cases, which is the strongest practical argument for freezing beyond comparability. It also makes discordance a number worth measuring rather than guessing: run the suite twice on the same commit, count how many cases flipped, and the result is your own instrument's noise floor rather than a textbook assumption.

Putting the whole thing in one place, at p ≈ 0.85 and a measured discordance of about 10%:

Read that table from the third column rather than the first. The question to start from is what size of regression the team would be unwilling to ship, and the case count falls out of it. Below the threshold that customers notice within a release cycle, a weekly measurement is enough; above it, the regression shows up in support escalations before any eval reports.

A 100-case suite with a 4-point alert#

The damaging combination is a smaller set with the old alerting condition left in place. A 100-case suite alerting on a 4-point drop fires on roughly a third of runs for no reason at all, since 4 points sits well inside its noise band, and each false alert costs somebody twenty minutes of investigation ending in "ran it again, it's fine". Three weeks of that and the team stops investigating, which arrives at a job that runs, costs money and has no consequence — the same end state as a bypassed gate, reached without anybody adding a flag and with nothing to point at.

The table puts that suite's smallest reliably detectable drop at 8.9 points, so a threshold of 4 points is set at under half the distance the instrument can resolve. An alert wired there is not detecting 4-point regressions; it is reporting which way the noise went on the day, and the investigation it triggers has nothing at the end of it because there was nothing there to find.

The alternative cuts are worth having ready before a budget meeting rather than after, because "how much can we save" has a much worse answer than "what can we give up". Reducing cadence keeps the resolution intact and lengthens time to detection. Splitting the suite — running a fixed 150-case subset on every merge and the full 400 nightly — keeps a coarse detector on the fast path and a fine one on the slow path, which is a real design rather than a degradation, as long as the subset is itself frozen and the two series are never compared to each other. Dropping the judge to a cheaper model saves roughly 40% and raises the discordance rate, which shows up directly as a worse detectable difference, so it has to be measured rather than assumed.

A merge gate is not a hypothesis test#

The arithmetic above is for choosing a case count and setting a threshold that can be defended once. It is not an argument for running a significance test on every merge and blocking when p < 0.05, which sounds rigorous and behaves badly: thirty merges a week is thirty tests a week, so at a 5% false-positive rate the team is blocked by chance roughly 1.5 times a week even when the code is perfect. Thirty independent opportunities to be unlucky, each with a chance of 0.05 of taking it, gives thirty multiplied by 0.05 on average — and the multiplication is legitimate here for the same reason it is legitimate for a biased coin flipped thirty times, since the merges are separate events and one run going red does not change what the next one does. Once the case count and threshold are chosen, the gate itself should stay a simple comparison against a fixed number, with the statistical work done once rather than continuously.

Where the distinction matters most is in what to do with a red run. A gate firing is a claim that this build should not be promoted, and it is worth acting on even at a 20% chance of being noise, because investigating costs twenty minutes and shipping a real 5-point regression costs a week of support load. A gate firing is not a claim that the regression is real, which is what a second run is for: re-running the same suite on the same commit either reproduces the drop or does not, and that single confirmation step removes most false alarms at a cost the team barely notices. Building the re-run into the alert flow — the alert says "score 74.1, re-running to confirm" and updates itself — is cheaper than building a more sensitive gate.

One more lever hides behind the case-count discussion, which is the number of draws per case. A suite taking one generation per case leaves the generator contributing directly to the flip rate. Taking three draws per case and scoring the majority cuts the generator's share of the noise floor by roughly a factor of sqrt(3), which in the suite above would take discordance from 9.8% to around 7% and improve the detectable difference from 4.4 points to about 3.7, for triple the generation cost. Whether that beats 800 cases at one draw depends on where the noise actually lives, which is why splitting it is worth the two runs it takes: score the same set twice with generation cached and only the judge re-run, and the discordance observed is the judge's alone, with the remainder belonging to the generator.

Two conditions sit on that arithmetic and both belong beside it. The sqrt(3) applies to the generator's contribution to the flip rate and not to the judge's, so how much of the 9.8% it removes depends entirely on how that figure divides between the two. The two shares also do not simply add, because a case can flip for both reasons on the same pair of runs. Where the judge holds most of the flips, which is what the measurement in the scenario above reports, majority voting over draws buys less than the improvement quoted here — the ceiling on what it can remove is the generator's share, whatever that share turns out to be. That is exactly why the split is a measurement rather than an assumption, and why a team about to spend triple the generation budget should run the cached-generation comparison before committing to it.

Where the arithmetic stops being trustworthy#

The formulas assume independent cases drawn from the population you care about, and eval sets violate that in ways worth knowing. Cases lifted from the same week of production traffic cluster — twelve variants of "where is my parcel" behave as fewer than twelve independent observations, which makes the real resolution worse than the calculation says.

The step that breaks is the one where variances were allowed to add. Adding them is valid only when the cases are independent, which here means that knowing how one case scored tells you nothing about how another scored. Twelve near-duplicates of the same question violate that directly. If the assistant's handling of parcel-tracking questions regresses, all twelve flip together; if it does not, none of them do. Their verdicts move as one, so the twelve carry roughly one case's worth of independent information into the calculation while contributing twelve to the count that goes under the square root. The denominator is too large, the standard error that comes out is too small, and every resolution figure in the table is better than the resolution the suite actually has. No formula in this chapter repairs that. What repairs it is a habit: when asking whether a set is large enough, count distinct behaviours rather than rows.

Cases added after individual complaints break the assumption from the other side, and break the "drawn from the population you care about" half of it rather than the independence half. They are not a sample of anything; they are a curated hard set, and mixing them into an average dilutes them until a regression that destroys performance on exactly that subgroup moves the aggregate by half a point. The defence is to keep subgroups as separate reported numbers rather than dissolving them into one score, and to accept that a 400-case suite is really several smaller suites whose individual resolutions are worse than the table suggests.

The judge contributes its own instability, which sits inside the discordance figure and moves independently of your code. A provider updating the judge model shifts both the score level and the flip rate, which is why the judge model is pinned alongside the case file and why a judge upgrade is treated as a new baseline rather than as a continuation of the series.

There is also a live research direction here, out of scope to implement and worth knowing exists. Variance reduction techniques from online experimentation — using each case's historical score as a covariate, in the manner of CUPED, which stands for controlled experiments using pre-experiment data — apply directly to repeated eval runs on a frozen set and can cut the required sample by a third or more. Sequential testing methods let a run stop early once the evidence is conclusive, turning a fixed 25-minute run into one that finishes in eight minutes when the regression is large. Neither is standard in eval tooling yet, and both are why the numbers in the table should be read as today's arithmetic rather than a floor.

A case count, then, is a claim about resolution, and it is the only claim the number makes. Four hundred cases at a 10% flip rate say the gate can see a 4-point drop and cannot see a 2-point one; a hundred say it can see nine points and nothing smaller. Neither is right or wrong until somebody names the smallest regression worth catching, and that naming is a product decision rather than an engineering one — which is why it is worth extracting explicitly before the budget conversation, since afterwards the number gets set by what was affordable and the resolution becomes whatever it becomes.

None of this has touched the bill. A 400-case suite at $6 is affordable or unaffordable depending entirely on how often it fires, and the multiplier outside that expression turns out to dominate everything inside it — which is the next chapter's subject.