Three points on thirty cases is not a difference

A colleague sends a pull request. It edits one sentence of a skill description, and the body of the request says: activation 96% → 99% on the 30-case suite. The gate function from the previous chapter passes it — activation went up, and nothing else in the baseline moved by more than a point.

Start with the arithmetic. Thirty cases run once each produce thirty-one possible values for a rate, spaced 3.33 points apart: 0%, 3.3%, 6.7%, and so on up through 93.3%, 96.7%, 100%. Neither 96% nor 99% is on that list. Asked about it, the colleague explains that the harness runs each case five times, so the denominator is 150 and the two numbers are 144/150 and 148/150. Four observations out of a hundred and fifty, on a system that produces a different trajectory every time you run it.

What one run in twelve does to a rate#

The suite's own flakiness had been measured, roughly, at about one run in twelve failing for reasons nobody could reproduce — a tool timing out, a container that came up slowly, a model that took a different branch on identical input. Put that number next to the four observations and the comparison collapses.

The cleanest way to see it is to run the baseline against itself. Take the unmodified skill, the unmodified corpus, and run the whole suite twice, changing nothing. That arrangement is an A/A test, the name borrowed from A/B testing with the same variant on both sides. Five such pairs on the deploy-release suite produced these deltas on activation: +1.3, −2.0, +0.7, −0.7, +2.7 points. Nothing changed between any of those pairs. The instrument moves by two or three points on its own, and a proposal claiming a three-point improvement is inside a range the harness generates from nothing.

That table is how the gate numbers in the previous chapter should have been derived, rather than chosen because they were round. Each gate sits above the measured noise floor for its own metric, which is why they differ: deterministic assertions on files and tool calls are the most stable thing in the suite, and a rubric-graded quality score carries both the agent's variance and the grader's on top of it. A single gate applied to all four would either wave through real regressions in compliance or fire every night on quality.

Reuse the cases so the comparison is paired#

Two runs over different random samples of requests differ for two reasons — the change, and the sample. Running the same cases before and after removes the second, and the improvement in precision is large enough to change what a suite of a given size can do.

The comparison to make is per case rather than per rate. For each case, record whether it fired before and whether it fired after, and count the two kinds of disagreement.

from math import comb

def mcnemar_exact(before: dict[str, bool], after: dict[str, bool]) -> tuple[int, int, float]:
    gained = sum(1 for k in before if not before[k] and after[k])
    lost   = sum(1 for k in before if before[k] and not after[k])
    n, k = gained + lost, min(gained, lost)
    if n == 0:
        return gained, lost, 1.0
    p = 2 * sum(comb(n, i) for i in range(k + 1)) / 2 ** n
    return gained, lost, min(p, 1.0)

gained, lost, p = mcnemar_exact(baseline_fired, candidate_fired)
print(f"gained {gained}, lost {lost}, p = {p:.3f}")

Run on the colleague's change, that printed gained 2, lost 0, p = 0.500. Two cases went from not firing to firing, none went the other way, and a coin flipped twice comes up heads twice about a quarter of the time in each direction. Whatever else is true about the edit, the suite has not distinguished it from doing nothing.

The cases that make the difference are the ones that do not behave consistently. Across five repeats of each of the thirty cases, twenty-two fired 5/5, three fired 0/5, and five were mixed — 4/5, 4/5, 3/5, 2/5, 1/5. All of the movement between any two runs comes from those five, and a suite that runs each case once assigns each of them a coin flip and reports the total as a rate. The per-case proportions are also more useful than the aggregate for diagnosis, because a case sitting at 2/5 is a description boundary problem in a way that a case at 0/5 and a case at 5/5 are not.

How many cases buy how much resolution#

The last column is quoted at 80% power, which is the probability that a suite of that size notices a difference of that size when one is really there — four times in five, with the fifth comparison coming back looking like nothing. It also assumes a discordance rate around 10%, meaning about one case in ten changes its behaviour between the two arms, the arms being the before run and the after run of the same cases; a change that affects almost nothing needs more cases than that, and a change that reroutes half the suite needs far fewer. The direction of the table is what matters. A thirty-case suite is an instrument for detecting differences of fifteen points and up, which makes it fine for the original 17% → 79% rewrite from chapter two and useless for the incremental tuning that follows it. Nobody sizes a suite for the first edit; they size it for the fiftieth.

Given a fixed budget of agent invocations, the split between cases and repeats has a defensible answer. Repeats reduce within-case variance and cases reduce between-case variance, and in every skill suite measured here the between-case component was the larger of the two, so invocations spent on new cases buy more precision than invocations spent on re-running old ones. The floor on repeats comes from a different requirement: with fewer than three runs per case you cannot see which cases are unstable, which means you cannot measure the noise floor and therefore cannot set a gate. Three repeats and as many cases as the budget allows is a reasonable default, moving to five for a suite whose gates are close to its noise floor.

Noise plus a one-sided suite is a ratchet#

Small differences on a positives-only suite compound into something worse than a wasted afternoon, and the mechanism is worth following through. Every proposed edit is judged on a movement of two or three points; roughly half of those movements are noise in the favourable direction, and the ones that get written up as pull requests are, by selection, the favourable ones. An edit that changed nothing at all has about a one-in-three chance of showing a gain large enough to look like progress on thirty cases, and the edit that showed a gain gets merged.

What survives that filter is not a random walk. Broadening a description raises the positives rate reliably, so broadening edits show real gains and narrowing edits show real losses, which means the process selects for permissiveness at every step regardless of whether the individual step cleared any threshold. Ten such edits over a quarter, each merged on a two-point positives gain and each costing one to two points of false activation, arrives at a description that fires on a fifth of the repository's traffic — and no single commit in that history looks wrong on review.

The two defences are the ones this topic has already built, and they work together rather than separately. The negative half from chapter two makes the cost of broadening visible as a number that moves in the opposite direction, so an edit has to clear two gates that pull against each other and cannot be satisfied by loosening alone. Sizing from this chapter makes those two gates readable, since a false activation rate measured on forty negatives has a confidence interval wide enough to hide the entire cost being incurred. A suite with negatives and thirty cases per side is a suite that can see the shape of the problem and not the size of it.

The failures this arithmetic does not prevent#

None of this makes the colleague's change wrong. It makes the evidence for it absent, which is a different claim, and the distinction matters because "not significant" is routinely read as "shown to have no effect". A two-case gain on thirty cases is entirely consistent with a real six-point improvement, and it is also consistent with nothing. The correct response is to run the change against a larger set — which for a change this small means several hundred cases — or to accept it on the strength of the reasoning about the description rather than on the strength of a number, and to say so in the pull request rather than dressing it in a percentage.

Statistical significance is also not importance, and here the direction runs the other way. A one-point rise in false activation that is dismissed as noise costs, at the traffic volumes from chapter two, about eight sessions a week paying a four-minute verify step for nothing. That is a real cost sitting under the detection threshold of the instrument, and the answer is to price it rather than to lower the gate until it fires.

The last trap is arithmetic of a different kind. A dashboard showing four metrics across twelve skills is forty-eight comparisons, and testing all of them nightly at a 5% threshold produces something like two false alarms a day, every day, without anything being wrong. A team that then investigates each one learns within a fortnight to ignore the dashboard. Either compare only the skill the change touched, or hold the alerting threshold to a rate that survives the number of comparisons being made.

Sizing, pairing and repeats all point the same way: a suite that can resolve small differences is a suite with several hundred cases and multiple runs each, and that is a substantial amount of compute to spend. Where that spending happens turns out to matter as much as how much of it there is, because a suite of that size attached to the wrong point in the development process does not survive contact with a team trying to merge pull requests.