Cross the runs to separate a data change from an evaluator change

Answer quality on support-golden read 0.810 in March and 0.740 in June. The run records make the diff mechanical: the judge prompt hash changed in April, the judge model was upgraded in May, and an annotator relabelled 203 of the 2,000 examples across three sessions in May and June. Three fields differ, everything else is byte-identical, and the paired bootstrap says the seven-point drop is real rather than a wobble.

The obvious next move is to re-run the June pipeline against the March dataset. If it comes back at 0.810 the data caused the drop; if it stays at 0.740 the code caused it. That experiment was run, and it returned 0.771 — a number that is neither, sits closer to June than to March, and got written up as "mostly the evaluator." The write-up was wrong in a way the experiment could not have detected, and reconstructing why is the whole subject of this chapter.

One crossed cell decides one factor#

When exactly one thing moved, the cross is clean. Two runs — the old evaluator against the new dataset, and the new evaluator against the old dataset — bracket the change, and whichever cell reproduces the old number identifies the factor that did not move the score.

Read the top-right cell against the baseline and you have the effect of the data change with the evaluator held constant; read the bottom-left against the baseline and you have the effect of the evaluator change with the data held constant. When one factor moved, the two off-diagonal deltas sum to the total, and the attribution is arithmetic.

With three factors that reasoning collapses, because "the June pipeline" is not one change. Bundling the judge prompt and the judge model into a single label called "the evaluator" assumes the two move the score in the same direction, and there is no reason they should. Running one bundled cell buys one number and asks it to answer three questions.

Vary one factor at a time, then check that they add up#

The design that works starts from the March configuration and changes exactly one field per run. Four runs establish the main effects — the movement each factor produces on its own, with the other two held at their March values — and the June configuration is already recorded, so it costs nothing to include:

The main effects already overturn the bundled reading. The judge prompt rewrite costs 4.8 points on its own, the relabelling costs 2.2, and the judge model upgrade raises the score by 2.1 — so a summary that says "the evaluator did it" is averaging a large negative and a moderate positive into one meaningless statement about a bundle. It also explains the 0.771 that started this chapter: that cell is A and B together on March data, and reading it against the two endpoints gave a plausible-sounding answer while silently attributing the relabelling's contribution to code and hiding the judge model's improvement entirely.

The four runs also let you test whether the factors are independent, which is the check most attributions skip. Adding the main effects predicts a June score of 0.810 − 0.048 + 0.021 − 0.022 = 0.761. The actual June score is 0.740, so 2.1 points are unaccounted for. That residual is larger than the ±1.8-point band chapter 3 measured for this pipeline, which means it is a real interaction rather than run-to-run wobble, and it justifies three more cells:

The new judge prompt and the new judge model interact: each is fine against the configuration it was designed for, and together they lose an extra 1.2 points, which is the signature of a rubric written against one model's behaviour being applied to another's. The remaining three-way residual is 0.7 points, comfortably inside the band, so the account is complete: −4.8 from the prompt rewrite, −2.2 from relabelling, +2.1 from the model upgrade, −1.2 from the prompt-model interaction, and the rest is noise.

Seven cells at roughly $30 each is about $210, and cells whose effect lands near the band should be run three times and averaged, since the band on a mean of three shrinks by a factor of the square root of three to about ±1.0 points. Budget $400 and an afternoon. Set against a two-day archaeology exercise that produces a paragraph containing the word "probably," the trade is not close.

Enumerating the cells is a matter of overriding fields on a recorded run rather than assembling configurations by hand, which is what makes this cheap enough to be routine:

from itertools import combinations

FACTORS = {
    "A": {"evaluator.config.judge_prompt_hash": "sha256:5f10dd82…"},
    "B": {"evaluator.config.judge_model":       "claude-sonnet-4-5-20250929"},
    "C": {"dataset_hash":                       "sha256:e0417c6b…"},
}

def cells(base_run: dict, factors: dict, max_order: int = 2) -> list[dict]:
    """Baseline, every single-factor change, and every pair."""
    out = [{"label": "baseline", "config": base_run}]
    keys = list(factors)
    for order in range(1, max_order + 1):
        for combo in combinations(keys, order):
            cfg = deepcopy(base_run)
            for k in combo:
                for path, value in factors[k].items():
                    set_path(cfg, path, value)
            out.append({"label": "+".join(combo), "config": cfg})
    return out

Every produced configuration is a complete run record of the kind chapter 2 specified, which means each cell is itself reproducible and each lands in the run store with its own run id. An attribution that cannot be re-derived six months later has the same defect as the score it was investigating.

When a factor cannot be put back#

The design above assumes every factor can be set to its old value. Judge models are the factor where that assumption fails first: providers retire snapshots, and a rubric pinned to a model that returns 404 makes cells A, C, AC and the March baseline unrunnable. Chapter 4's restore test surfaces this as dependency_unavailable, and no amount of hashing recovers it.

There are four responses, and only one of them is available after the fact.

The first two are the ones worth building into the process, because they cost almost nothing at the moment they are possible and are impossible afterwards. When an evaluator configuration changes, run the new evaluator against the frozen previous dataset once and store the result; the crossed cell exists before anybody needs it. When the judge model itself changes, score a frozen 500-example subset with both judges and record the paired difference. On this pipeline that produced an old-judge mean of 0.812 and a new-judge mean of 0.836, with a mean paired difference of +0.024 and a bootstrap interval of +0.019 to +0.029. With that recorded, a June-versus-March comparison across the judge switch can subtract 2.4 points and carry an explicit ±0.5-point uncertainty, instead of pretending the two numbers were ever on the same scale.

Self-hosting sidesteps the deprecation entirely and is available only for a judge you are allowed to run yourself, which in practice means an open-weights model — one whose trained parameters are published for download — or a commercial model licensed for on-premise serving. The judge then stays byte-identical for as long as you keep serving it, at the price of running inference infrastructure and watching the judge fall behind the frontier while the system it grades moves on.

Where nothing was banked, re-baselining is the honest option and it is unpopular for the right reason: it means telling people that the numbers before the switch cannot be compared to the numbers after it. That is a smaller loss than a chart whose apparent trend is an artefact of a judge upgrade, and it is recoverable — a re-baseline that starts a new series with a recorded calibration run against the current judge means the next switch will not have this problem.

Making this ordinary rather than heroic#

The investigation above was expensive mostly because three eval-affecting changes were allowed to land inside one measurement window with no crossing run between them. Four process changes remove that, and none of them require new infrastructure beyond what chapters 1 to 4 already built.

Serialise eval-affecting changes. A judge prompt rewrite, a judge model upgrade, and a relabelling batch are three releases, not one, and each gets its own run against the unchanged rest of the configuration. This is the single highest-value change and it is organisational rather than technical.

Gate the change on the crossed run. A pull request that alters the evaluator configuration hash does not merge until CI has run the new configuration against the previous frozen dataset and attached the delta. The cell costs $30 and it is the cell that becomes unobtainable later.

Bank the calibration automatically. When the evaluator configuration hash changes, the same job scores a fixed 500-example calibration subset under both the old and new configuration and writes the paired difference into the run store, keyed by the pair of hashes. It is then a lookup rather than an archaeology project.

Store per-example scores, not aggregates. Every technique in this chapter and the previous two — the paired bootstrap, the agreement rate, the interaction check — needs the individual verdicts. Aggregates cost nothing to store and destroy every analysis that matters.

What the cross does not settle#

It attributes the movement to a factor and stops there. Knowing that the judge prompt rewrite cost 4.8 points does not say which change to the rubric did it, and finding that out means diffing the prompt and running the intermediate versions, which is the same design applied one level down.

More importantly, it does not say which number is correct. The relabelling that cost 2.2 points may have been an annotator fixing 203 genuinely wrong labels, in which case the March score was inflated and the June score is the more honest one; attribution and evaluation are different questions, and a crossed design answers only the first. The same applies to the judge model upgrade that gained 2.1 points: a better judge that is stricter in the right places can move the number either way, and the direction of the move is not evidence about the direction of the quality change.

The design also assumes the enumerated factors are the only things that moved. Anything unpinned — a retrieval corpus rebuilt on its own schedule, a library upgrade that changed normalisation, a provider silently updating behaviour behind a stable name — shows up as an unexplained residual and gets misattributed to interaction. When a residual is large and no pair of factors plausibly interacts, the correct conclusion is that the run record is incomplete rather than that the factors are entangled.

There is one shape of change this design cannot handle at all. Every cell above scores the same 2,000 examples, which is what makes the deltas comparable; the relabelling changed the labels on some of them, not which examples exist. The moment a dataset version adds or removes examples, "the old evaluator against the new data" is a comparison across two different populations, and the arithmetic that made this chapter work stops applying. That case — the set that grew, and the score that moved with it — is the next chapter.