Two tasks flipping is not a five-point improvement

The change was to the context-selection policy: instead of ranking candidate files by a blend of path similarity and recency, the new version ran a symbol-level index and included the definition sites of every identifier named in the task statement. It took two weeks. The evaluation was a run of the 40-task development split before and after, one attempt per task, and the results page showed 61% and 66%. The pull request description said "improves suite resolution by 5 points", three people approved it, and it shipped.

Underneath the percentages are counts. Two tasks errored out in one arm — one side of the comparison, the old policy or the new one — for infrastructure reasons — a runner ran out of disk, a package index timed out — so both figures were computed over the 38 tasks that produced valid runs in both arms. The before arm resolved 23 of 38 and the after arm resolved 25 of 38. The five-point improvement is LEDG-1877__c4a0e2, an idempotency fix, and SVC-994__71bb03, a serialiser change, going from red to green. Everything else in the suite landed the same way twice.

How many tasks flip when nothing changes#

The suite had already been run ten times per task during the pass@k work, so the per-task resolution rates are on disk. Most tasks are not near 0 or 1. Fourteen of the 38 resolved between three and seven times out of ten, and those are the tasks that decide a single-run comparison.

If a task resolves with probability p, then two independent single runs disagree with probability 2p(1-p), which peaks at 0.5 for p = 0.5. Summing that across the suite gives the number of flips to expect between two identical configurations:

rates = load_per_task_rates("suite_10x.json")   # {task_id: c/10}

expected_flips = sum(2 * p * (1 - p) for p in rates.values())
# 8.9

# Distribution of the *net* change in resolved count, nothing changed
import random
def null_net_change(rates, trials=100_000):
    out = []
    for _ in range(trials):
        before = sum(random.random() < p for p in rates.values())
        after = sum(random.random() < p for p in rates.values())
        out.append(after - before)
    return out

net = null_net_change(rates)
sum(abs(x) >= 2 for x in net) / len(net)   # 0.62
sum(abs(x) >= 5 for x in net) / len(net)   # 0.19

Between two identical runs of an unchanged agent on this suite, about 8.9 tasks flip in one direction or the other, and the net count moves by two or more tasks 62% of the time. Getting +2 from a change that does nothing is the ordinary case rather than an unlucky one. Even a +5 net change — a 13-point swing on a 38-task suite — happens by chance about one run in five. The comparison that shipped this change could not have distinguished a real improvement from no change at all, and it would have looked much the same if the new context policy had been slightly worse.

The mistake is tempting because the aggregate looks like the thing you care about. Suite resolution is the metric on the dashboard, so comparing it before and after feels like the natural experiment, and the arithmetic of percentages hides the counts. Writing "23 → 25 tasks" in the pull request description instead of "61% → 66%" would have provoked the right question from at least one of the three approvers.

Asking about tasks instead of about the aggregate#

The fix is to run each task several times in each arm and analyse the arms as paired by task, because the tasks are the units that vary and they are the same units on both sides. Pairing removes the largest source of variance immediately: LEDG-2041 is hard under both policies and SVC-310 is easy under both, and an unpaired comparison of two aggregates makes you re-estimate that difficulty from scratch on each side.

With five runs per task per arm, each task contributes a pair of resolution rates:

The two tasks that "flipped" in the original comparison did move, and so did one in the other direction that the single-run comparison happened not to catch. What matters is whether the mean difference is distinguishable from zero given how much individual tasks bounce, and a paired bootstrap over tasks answers that without assuming anything about the shape of the distribution. The bootstrap works by resampling the per-task differences with replacement a few thousand times and looking at how much the resampled mean wanders, which is a way of asking how differently the experiment could plausibly have come out with a different draw of tasks:

import random, statistics

def paired_bootstrap(before: dict, after: dict, iters=20_000) -> tuple[float, float]:
    ids = list(before)
    deltas = [after[t] - before[t] for t in ids]
    means = []
    for _ in range(iters):
        sample = [random.choice(deltas) for _ in ids]
        means.append(statistics.fmean(sample))
    means.sort()
    return means[int(0.025 * iters)], means[int(0.975 * iters)]

paired_bootstrap(before_5x, after_5x)   # (-0.021, +0.118)
paired_bootstrap(before_10x, after_10x) # (+0.008, +0.094)

At five runs per task the 95% interval spans zero, so the honest report is that the change may have helped by up to twelve points and may have hurt slightly, and the experiment cannot say. At ten runs the interval clears zero with a point estimate of about five points, which is the same headline number the original comparison produced and is now supported by evidence. The cost of getting there was 38 tasks × 10 runs × 2 arms = 760 agent runs, about $125 in tokens and eleven hours on eight parallel workers. That is a real expense and it is roughly the price of one afternoon of the two weeks that went into building the change.

Spending the run budget in the right place

Given a fixed number of agent runs, there is a choice between more tasks and more runs per task, and the two buy different things. Adding tasks reduces the uncertainty that comes from having sampled a small slice of the repository's work; adding runs per task reduces the uncertainty that comes from the agent being stochastic, meaning that the same task under the same configuration does not always come out the same way. Which one dominates depends on where the variance is, and that is measurable from the ten-run data already collected. On this suite, the between-task standard deviation of resolution rate was 0.34 and the average within-task standard deviation was 0.21, so tasks differ from each other more than a task differs from itself — but the within-task component is what a paired analysis leaves behind, and at one run per task it is the whole error term.

The practical shape of the trade-off, for detecting a 5-point mean difference at this suite's variance:

Ten runs on 38 tasks and five runs on 76 tasks cost the same and the narrower interval belongs to the deeper sampling, which is the opposite of most people's instinct. The reason is pairing: a wider set of tasks helps generalisation but each new task arrives with its own difficulty, while extra runs on a task already in both arms attack the noise term directly. Generalisation still matters, so the full suite gets used for the periodic reporting run; the development split gets used deep for change comparisons.

For a purely binary framing — did the task's outcome change, ignoring by how much — McNemar's test on the discordant pairs is the classical instrument, and it is worth knowing because it makes the logic explicit. It looks only at tasks that resolved in one arm and not the other, counts them in each direction, and asks whether the split between those two counts is lopsided enough to be surprising under a fair coin. With 2 flips one way and 0 the other, the exact test gives p = 0.5, which is the arithmetic statement of "you observed two coin flips landing heads".

Both arms have to meet the same world

Pairing by task removes the variance that comes from tasks differing in difficulty, and it leaves untouched everything that changed between the moment the first arm ran and the moment the second one did. In the original comparison the before arm was collected on a Monday afternoon and the after arm on the Thursday, because building the symbol index took the intervening days. Re-running the before arm on the Thursday, with the old policy and not a line of code altered, resolved 24 of 38 rather than 23. One task had gone green on its own, which is 0.4 of the two-task improvement that the change was credited with.

Three days is enough for several things to move underneath a suite, and none of them announce themselves in the results file:

Interleaving means each worker takes a task, runs it under both arms back to back in randomised order, and records the pair, so an afternoon when the provider is slow or the registry is broken hits both arms roughly equally and the difference survives it. It costs nothing beyond scheduling, since the total run count is unchanged, and it turns environment drift from a bias into noise the bootstrap already accounts for. The last two rows are the ones interleaving cannot reach: a suite whose task definitions or base commits move during an experiment is a suite that has to be pinned by revision, with the pin recorded next to the result, or the numbers are describing two different suites.

Where this discipline stops helping#

None of this makes a five-point improvement on the suite worth having. A confidence interval clearing zero establishes that the change did something to this suite; it says nothing about whether the effect transfers to the packages the suite cannot sample, and nothing about whether five points of resolution is worth two weeks of engineering time and the extra 40 seconds per task that the symbol index adds. The statistics answer the question "is this real", and the questions "is this useful" and "is this worth it" are separate and need different evidence.

There is also a failure mode that more runs make easier rather than harder. Once the harness can produce a paired analysis in eleven hours, it becomes cheap to run the comparison, look at the interval, adjust a threshold, and run it again — and repeating that loop until an interval clears zero produces intervals that clear zero. The protection is procedural: decide the number of runs and the decision rule before the first comparison, count how many comparisons a change has been through, and treat a change that needed four attempts to show an effect as unproven rather than as proven on the fourth. A related version is re-running only the tasks that regressed, on the theory that those were flaky, which guarantees that regressions get resampled toward the mean while improvements keep their lucky draws.

Two smaller cautions belong with this. Dropping tasks that errored out, as the original comparison did with its two infrastructure failures, is only safe when the errors are independent of the arm; if the new context policy makes the runner run out of disk, dropping those tasks removes exactly the evidence of harm. Reporting the denominator alongside the rate is what makes that visible. And the development split is 38 tasks precisely because it is the part of the suite reserved for iteration, so the interval above describes a suite the change was tuned against, which is a distinction the last chapter of this topic returns to.

Suppose the change survives all of it and the agent now resolves five points more of the suite, reliably, at a cost the team is happy with. There is one more question that the resolution rate cannot reach, which is whether the patches it produces cost less to deal with than they used to, and that question has its own measurement problems.