Reference bias in a model comparison

A model migration normally arrives as a two-line diff: a model id in a config file, and whatever the suite says about it. When the suite is built from recorded outputs — replies captured from the model currently in production, read once, judged acceptable, and committed — the number it reports about a candidate is not a quality measurement, and the number it reports about the incumbent is not one either.

Reference bias is the name for the general defect: the yardstick was authored by one of the contestants. An interval on the resulting difference turns out to matter more than the point estimate does.

The incumbent was never scored#

Take a suite of 40 stored support replies compared with exact string equality, captured a year ago by running the then-current model over 40 representative tickets, reading each reply, and committing the ones that looked good. On the incumbent it reports 40/40. On a candidate it reports 11/40, which reads as 29 regressions and a 72% failure rate.

Start by asking what produced the 40/40, because that number carries the whole comparison and nobody has looked at it. Nothing produced it. The 40 files were written by taking the incumbent's output and saving it. Comparing the incumbent to its own saved output is a tautology, and 40/40 was true the moment the files were committed, before any test ran. It is a property of how the fixtures were made rather than a measurement of the model.

Watch what the two procedures actually are, side by side, because the tautology is visible in one line each. Recording did this: for every ticket, call the model, keep what comes back. Testing does this: for every ticket, call the model, compare what comes back against what was kept. If the model were a function of its input, the second is the first with an equality sign attached and could not possibly report anything but a perfect score. That is why the 40/40 travelled from release to release as the incumbent's score without anybody re-running it: the answer was known in advance, and a number known in advance stops being read as a measurement and starts being read as a baseline.

The model is not a function of its input, which is what makes the check below interesting rather than merely pedantic. Each call is a draw, so re-running the recording procedure a year later does not reproduce the recording. The 40/40 was therefore never measured and never true of the live system: it was the score the suite would have reported in a world where the model was deterministic, quietly inherited by a world where it is not.

That claim is checkable in about thirty seconds:

matches = 0
for path in sorted(Path("golden").glob("*.txt")):
    ticket = load_fixture(f"tickets/{path.stem}.json")
    reply = generate_reply(ticket, model="claude-opus-4-8")   # the incumbent
    matches += (reply.strip() == path.read_text().strip())

print(f"incumbent vs its own goldens: {matches}/40")

incumbent vs its own goldens: 14/40

Fourteen. The incumbent, live, against the replies it wrote itself a year ago, reproduces 14 of 40 exactly — each call is a draw, and a stored reply is one member of a set. The suite's 40/40 was never the incumbent's score; it was the score of a cache. The real comparison on the existing yardstick is 14 against 11, and both of those numbers mostly measure how much each model's phrasing happens to resemble a year-old recording.

The command costs 40 calls, about fifty cents, and it converts "the new model regressed 72%" into "our yardstick gives the incumbent 14/40 and the candidate 11/40, and neither number is about quality". It is worth running before writing a single new check.

What 11/40 measures#

Look at what the differences actually are. Here is ticket 2214, a delivery complaint, under both models:

-Thanks for getting in touch. I'm sorry your order #2214 hasn't arrived yet.
-Our courier shows it as delayed at the Manchester depot. I've raised this with
-them and you should have an update within 2 working days. If it hasn't moved by
-then we'll send a replacement at no cost.
+I'm sorry to hear order #2214 still hasn't reached you. The tracking shows it
+held at our Manchester depot, and I've escalated it with the courier — you
+should hear back within 2 working days. If there's no movement by then, we'll
+ship a replacement free of charge.

Every requirement holds in both. The order number is present, the depot is named, the two-working-day commitment matches policy, the replacement offer matches policy, and the tone is right. An exact-match test scores the second one zero. Twenty-two of the 29 "regressions" were like this on inspection; five differed in the closing sign-off; two were genuinely different in substance.

The two substantive differences are the only part of the red build worth an engineer's afternoon. On ticket 2288 the candidate offered a goodwill credit where the recording had offered a replacement; the policy permits either, so that is a product decision rather than a defect, and somebody should decide which behaviour they want. On ticket 2301 the candidate committed to "delivery by Friday" where the recording had said "within 5 working days" — a hard date the company does not control, which the reply spec explicitly forbids. That one is a genuine finding, and the candidate is worse on it.

So an exact-match suite produced 29 red cases containing two things worth reading and 27 that were noise: a precision of about 7%. That is a useful way to read any failing suite. The question is not whether it found anything, because a suite that fires on everything always finds something, but what fraction of the failures it hands you are worth the time to triage. At 7%, an engineer reads 29 diffs to reach two findings, and after a few rounds of that they stop reading carefully — which is how the Friday-delivery bug ships inside a red build everyone agreed was mostly cosmetic.

Regrading both models on requirements neither wrote#

The repair is a yardstick with no allegiance. The requirements come from the reply spec and the customer-facing policy pages, and specifically not from reading either model's output:

• The reply names the order number from the ticket.
• Any timeframe stated appears in the policy document retrieved for that ticket.
• Any remedy offered — replacement, refund, credit — is one the policy permits for that ticket type.
• The reply does not commit to a delivery date.
• The reply is between 40 and 180 words.
• The reply ends with a next step the customer or the company will take.

Six checks, all cheap, all mechanical except the last two, which a judge model — a second model call whose only job is to score an answer against a short written rubric — handles against a one-paragraph description. Run both models over the same 40 tickets, three repeats each, and score every reply. The whole thing is 240 calls, around three dollars, and finishes inside four minutes.

incumbent   34/40   0.850
candidate   36/40   0.900

Cross-tabulated over the same 40 pinned tickets: 33 pass under both, 3 fail under both, 1 passes only under the incumbent, 3 pass only under the candidate. The paired difference is +2/40, five points in the candidate's favour, and the paired standard error √(b + c − (b−c)²/n) / n is √3.9 / 40 ≈ 4.9 points. The 95% confidence interval runs from about −5 to +15 points.

Every number in that sentence comes from the four discordant tickets, and it is worth watching them do it. Thirty-six of the 40 agreed under both models and cancel out of the difference exactly as they did in the previous chapter, so the comparison rests on one ticket the incumbent alone passes and three the candidate alone passes. Their imbalance is two tickets, which over 40 is the five points. The standard error takes the four discordant tickets, subtracts the small correction the formula calls for — the squared imbalance over 40, a tenth — to leave 3.9, takes the square root, which is a shade under two, and divides by the 40 cases, giving 4.9 points. The interval is then that standard error multiplied by the factor that covers 95% of a normal distribution — the one derived in the previous chapter, a shade under two standard errors — which comes to about nine and a half points, laid either side of the five: from roughly −5 to +15.

Notice what dominated. The evidence is built from four tickets, not 40, because four is all that moved. Adding more tickets that both models pass shrinks the measured difference and its standard error in the same proportion and leaves the strength of the evidence exactly where it was — which is the counter-intuitive reason a 40-case suite is so weak here. It is a four-case experiment wearing a 40-case denominator, and what would strengthen it is not more cases but more cases on which the two models disagree.

The honest finding is that 40 cases cannot tell these two models apart. The point estimate favours the candidate; the interval comfortably contains zero, contains a five-point regression, and contains a fifteen-point improvement. Exact McNemar — the standard test for this shape of data, which ignores the cases that agree and asks only whether the discordant split could have come from a fair coin — on a 3–1 split gives a p-value of 0.63, meaning a change that did nothing at all would produce a split this lopsided or worse about 63 times in 100. That is close to the strongest available statement of "no evidence either way".

Four discordant tickets is few enough to count the whole thing out, and doing so once makes every later p-value less mysterious. Each of the four could have landed either way, so there are two possibilities per ticket and sixteen equally likely patterns in total, equally likely because the null hypothesis says each is a fair coin. Patterns at least as lopsided as 3–1 in one direction are the four ways exactly one ticket goes the other way, plus the one way none does: five of sixteen. Doubling for the opposite direction, because the question is whether the models differ at all rather than which is better, gives ten of sixteen — the 0.63. Ten times in sixteen, two identical systems would produce a split this uneven or worse. Nothing about the candidate has been established, and nothing about the incumbent has been either.

An interval that wide is also a specification for the experiment that would settle it, and the scaling is the one from two chapters ago. A standard error shrinks with the square root of the number of cases, so halving the width of this interval means quadrupling the tickets, and lifting a five-point effect clear of an interval currently about ten points wide on each side takes several quadruplings rather than a few more cases. That is the arithmetic behind the figure the report below quotes: somewhere around 300 paired cases, with the exact number coming from a power calculation that also budgets for the risk of missing a real difference rather than only for the risk of imagining one. Dropping the pairing roughly doubles that, because an unpaired comparison carries a larger standard error and the number of cases scales with the square of the standard error you are aiming at — so a modest widening of the interval is paid for with a great many tickets.

Where else the yardstick has an author#

Recordings taken from the incumbent are easy to spot once named. The general shape is harder, and it appears wherever one of the things being compared also produced the reference the comparison is scored against. Whoever authored the yardstick wins on it by construction, and the margin has nothing to do with quality.

The retriever row catches careful teams, because the labelling step feels like it launders the bias out. Somebody takes the current retriever's top five documents for each query, has a human mark which are genuinely relevant, and calls the result a labelled evaluation set. It is a labelled set — of documents the current retriever already returns. A candidate that surfaces an excellent document the incumbent never retrieved gets no credit for it, because that document is not in the labels, and the more the candidate differs the worse it scores. The fix is the same one applied above: derive the reference from something neither system produced, which for retrieval means labelling a pool assembled from both systems' results, or from the corpus directly.

The symmetry test is quick and catches all four rows. Ask whether swapping the two arms would swap the result. If A beats B on this yardstick, would B beat A on a yardstick built the same way from B's output? When the answer is yes, the comparison is measuring authorship.

Recorded outputs as raw material#

Recordings remain useful, and reading this chapter as an argument against keeping them would cost two good things.

They are the best raw material for writing requirements. The six checks above were written faster because somebody could read 40 real replies and notice which properties every good one shared and which failure modes recurred. Reading outputs to derive criteria is sound; scoring outputs by equality to them is what fails.

They are also legitimate change detectors for anything upstream of the model call — the rendered prompt, the retrieved document ids, the request payload. A byte-level diff on those is a signal that code changed, because code is what produces them.

An independently authored rubric is not free of the bias either, and that is worth naming honestly. Whoever writes the requirement list has usually spent a year reading the incumbent's output, and the incumbent's house style leaks into what feels like a requirement: a preferred sign-off becomes "ends with a next step", a habitual opener becomes "acknowledges the complaint". The candidate then loses points for not sounding like its predecessor, and the loss is invisible because it is dressed as a criterion. Two mitigations help and neither is free. Write the requirements from the policy documents and the spec rather than from memory, and have somebody who has read neither model's output review them for anything that is really a style preference. Then, where the checks involve human or model judgement, strip provenance and shuffle the order before grading, so nobody scoring a reply knows which system produced it.

A comparison is only as trustworthy as the independence of its yardstick, and the cheapest way to test that independence is to point the yardstick at the system that authored it. An incumbent scoring 14/40 on its own recordings tells you the suite was never measuring the incumbent; a retriever scoring perfectly on labels drawn from its own results tells you the same thing in a different costume. Once the yardstick comes from a spec instead, both arms are measured on the same terms, and the number that comes out deserves an interval printed beside it — which on forty cases will usually say the experiment cannot decide, and say what size of experiment could.

That still leaves the deepest limitation of everything in this topic. Every measurement here is a measurement against requirements somebody wrote down. The failures those requirements do not describe stay invisible at any sample size, with any interval, on any yardstick — a suite reports on the properties it was told to check and is silent about the rest. Growing that list is the ongoing work, and it grows from production: every escalated ticket and every quality review that finds something the checks let through is a requirement that was missing.

Which is where this stops being reading and becomes a task. Take the feature you are responsible for and write the list — the things every acceptable output must have, in concrete terms a check can evaluate. Sample a few hundred real inputs rather than inventing them. Score them, report the rate with its interval, and pin what has to stay still between runs. The first version will be short and will miss things; it will still say more about the system than any suite of stored blobs ever did.