Auditing your own eval

A comparison is fair when each side had a comparable opportunity to do well. That is a different property from running both sides through the same pipeline, and the difference is where most of the error in a home-built eval lives. Prompts, rubrics and sample sizes are all things somebody on your team chose, and each choice was made while looking at one model's behaviour, usually the incumbent's.

The sentence a team writes at this point is that both models were run on the same prompt, against the same rubric, over the same tickets, so whatever difference came out is a difference between the models. Every clause of it is true, which is what makes it hard to argue with, and the conclusion still does not follow. Holding the instrument fixed removes the differences that would have come from running the two models differently — a stale model string, a cached response, a rubric applied to one side and not the other — and it leaves untouched every difference that was built into the instrument before either model was run. Identical treatment controls the second half of the experiment and says nothing about the first.

Each threat is worth stating as a gap adjustment — what the measured difference between two models was before the check, and what it was after — because reporting the adjustments rather than only the final number is what makes the final number credible to somebody who did not run it. An audit that only ever confirms its own result is not an audit, and the adjustments below all run in the same direction, which is downward.

The prompt was written against one model#

A prompt reads as a neutral container for a task, and it is nothing of the kind. It is a record, in text, of everything one model got wrong while somebody was trying to get the job done. Each line in it was added after an output was read: a sentence because the model escalated a ticket it could have answered, a formatting instruction because it drifted out of the schema, an example because it misread a category of question. None of those additions describes the support task, which has not changed at any point in the process. Each of them describes a failure mode of one particular model, patched in place.

That is the property that pulls sameness and fairness apart. A revision that removes a failure mode belongs to whichever model had the failure, so a prompt finished against one model encodes that model's weaknesses as if they were requirements of the work. Hand it to a second model and every one of those lines is still there, spending instructions and attention on problems that model never had, and buying it nothing, because the mistakes they prevent are not its mistakes. The same artefact is also missing every fix the second model needs, since nobody read one of its outputs while writing the text. A prompt with six weeks of adaptation behind it therefore penalises B twice over: it handicaps B on instructions that were only ever fixes for somebody else, and it withholds the fixes B would have wanted.

Sameness is a property of the procedure and fairness is a property of the opportunity, and a clean run only ever demonstrates the first. Both models received the same instructions, from the same file, over the same tickets, and one of them was competing on a course laid out around the other one's mistakes. Nothing in that requires anybody to have been careless, and no amount of care during the run recovers it, because everything the difference consists of was fixed before the run began.

The correction is not to write a neutral prompt, because there is no such thing — every prompt is adapted to something. It is to give each model a comparable adaptation budget. Take the same starting prompt, spend a fixed effort per model (say four hours, or twenty revisions, capped identically) improving it against a development split of cases that is disjoint from the eval split, and then run each model on the eval split with its own adapted prompt. What you are comparing is then "best achievable with this model at four hours of effort", which is much closer to the decision you are actually making, because in production each model would get its own prompt anyway.

Running that on a support eval where model B led by 9.2 points, model A gained 3.1 points from its four hours and model B gained 1.4. What moves the gap is the difference between the two gains rather than either gain on its own, since both totals rose and only the distance between them is being reported. A gained a little over three points, B gained a little under one and a half, and the amount by which A's gain outran B's is a little under two points. Taking that off the 9.2 the eval had originally produced leaves 7.5.

A gained more, and the reason is not that A had more room to gain. It had less: a prompt that has already absorbed forty revisions against a model has had that model's easy wins taken out of it, so the improvement still theoretically available to A from another four hours was the smaller of the two. What A had instead was a team that knew where to look. Six weeks of iteration had taught them which kinds of instruction move A and which do nothing, so four hours spent on A was four hours of applying a method that already worked. B's slack was larger and far less obvious, because it sat in failure modes nobody had gone looking for, and finding those means reading enough of B's wrong answers to know what they are — which is most of what four hours buys.

That cuts both ways, and which way it cuts depends on a number somebody chose. Under a budget short enough that finding a problem costs more than fixing it, the model the team already understands gains more, and the correction runs in the incumbent's favour. Under a long enough budget the two gains can swap order, because the larger pool of unexploited slack was B's the whole time. An equal-hours budget equalises the input and not the opportunity, which is what makes it a defensible protocol rather than a neutral one, and it means the four hours is a parameter of the result rather than a detail of how it was produced.

The failure mode to watch here is tuning against the eval split itself, which turns the audit into the benchmark-overfitting problem chapter 4 described, happening faster and on your own data. Keeping a development split that is never scored is the whole defence, and it costs you items from an already small set.

The rubric encodes one model's conventions#

The schema-validity component on that eval was implemented as: parse the model's output with a JSON parser, and if it parses and matches the schema, score one. Model A's default behaviour is to wrap JSON in a fenced code block. Model B's is to emit the object bare. The parser saw a leading backtick and failed.

That is a formatting convention, not a capability, and in production it would be handled by three lines of stripping code that every team writes anyway. Making the rubric tolerant of both conventions — strip fences, strip a leading "Here is the JSON:", then parse — moved A's schema validity from 91.3% to 98.8% and took another 1.5 points out of the gap, leaving 6.0.

Where that 1.5 comes from is the contribution arithmetic of the previous chapter (chapter 5, where a rubric component's effect on the weighted total is its weight multiplied by the difference between the two models on that component) run in reverse. Schema validity carries a fifth of the rubric's weight. The fix moved A on that component from just over ninety-one percent to just under ninety-nine, a rise of about seven and a half points, and it moved B not at all, because B's bare objects had always parsed. A fifth of seven and a half points is about one and a half points, and that is the entire adjustment: 7.5 before, 6.0 after.

Two consequences are worth reading off that. The size of the correction is bounded by the weight, so the same seven-and-a-half-point fix on a component carrying twice the weight would have taken twice as much out of the gap, and on a component carrying none it would have taken nothing — how far a formatting bias can distort a result is a fact about the rubric's weights before it is a fact about the parser. And nothing in the arithmetic depended on the tolerant parser being the better parser. It depended on the component being scored the same way for both models, which is what it had not been.

The general shape of this threat is that a rubric written by reading one model's outputs will encode that model's conventions as correctness. It shows up in output-format checks most visibly, but it is also present in reference answers written in a particular register, in judge prompts that describe a good answer using phrasing one model happens to favour, and in string matching against terminology one model uses and the other paraphrases. The check is to have someone read a sample of the losing model's scored-wrong outputs and ask, for each one, whether a human would have accepted it.

A point estimate without a spread#

Six points on 300 items is a point estimate, and a point estimate without a spread is not yet evidence. The number summarises one run over one set of items, and both of those were drawn from something larger: the run is one of the runs this setup could have produced, and the set is one of the sets the ticket queue could have supplied. Treating six points as evidence means saying how far it would have moved had either draw come out differently. There are two independent sources of that movement and they need separate treatment.

Run-to-run variance is what you get by running the same model on the same items again. Sampling makes generations differ; where the rubric uses a model as judge, the judge's own sampling adds more. On several current models the sampling parameters that used to be pinned for determinism are not accepted at all, so this variance is a property of the setup rather than something you can switch off. Measuring it is a matter of running the whole eval three to five times and taking the standard deviation of the totals. On this eval, five runs of model B gave totals between 70.1 and 72.8, a standard deviation of 1.1 points. A standard deviation computed from five numbers is itself a noisy quantity, so the honest reading of 1.1 is an order of magnitude and not a figure to trust to its decimal: enough to know that run-to-run movement here is around a point rather than around ten, and not enough to subtract from anything.

Item-sampling variance is the harder one: 300 tickets are a sample from the population of tickets, and a different 300 would have given a different answer. The question worth answering is how different — whether a re-drawn quarter would still have put B ahead by something near six points, or could have put the two level. Answering it directly needs many independent sets of 300 tickets, each with human-written reference answers and each scored twice, and the reason the set is 300 items in the first place is that nobody could afford one such set, let alone a thousand of them.

What makes an answer available anyway is a substitution, and the substitution is the whole idea. The population cannot be re-drawn, but the 300 tickets in hand came out of it, and a sample is the best available description of the thing it was drawn from: the mix of easy and hard, answerable and unanswerable, short and long in those 300 is, up to the very noise being measured, the mix out there in the queue. So the sample is made to stand in for the population. Drawing a fresh set of 300 from those 300 then stands in for drawing a fresh 300 from the queue, and the set it produces differs from the original in roughly the way a genuinely new sample would have differed.

The tool for estimating that without collecting more data is the bootstrap, which comes from mid-century statistics and is unreasonably useful here: resample the items with replacement many times, which means drawing 300 items from your 300 so that some appear twice and others not at all, recompute the statistic on each resample, and read the spread of the resulting distribution. With replacement is the load-bearing phrase, and it is worth being concrete about: every one of the 300 draws is made from the full set of 300, so an item already picked stays eligible and can be picked again. Without it, drawing 300 items from a set of 300 hands back the original set every time, reordered and otherwise identical, and every recomputed statistic equals the observed one. The spread would come out as zero, which reports certainty about a number when what has actually happened is that nothing was allowed to vary.

The rest is repetition. Recompute the statistic on the resample — here the weighted total for A minus the weighted total for B — record it, and do the whole thing again a few thousand times. What accumulates is a list of values the gap takes across sets that differ from yours in the way a re-draw would differ, and the width of that list is the answer the single run could not give. Sort the list, discard the lowest two and a half percent of it and the highest two and a half percent, and what remains between the cuts is the middle ninety-five percent of the recomputed gaps. That range is the 95% confidence interval, and the two lines in the code below that index a sorted list at those two fractions are that step in full.

One assumption makes the substitution legitimate, and it is where the method's authority stops. Standing the sample in for the population is sound only if the sample was drawn by a mechanism that represents the population — one that gave the tickets you have no systematic advantage over the tickets you did not get. Granted that, the bootstrap measures how much the answer moves as the draw changes, which is sampling noise, and sampling noise is the only thing it measures. If the 300 were skewed — a quarter distorted by one incident, or a queue that under-represents the cases the assistant will actually meet — then every resample is built out of those same skewed items and inherits the skew whole. The interval comes back exactly as narrow as before and sits around the wrong centre, so a bootstrap can say how uncertain a sample leaves you and can never say the sample was the wrong sample.

The important refinement is to bootstrap the gap rather than the two scores separately, resampling item indices once per iteration and scoring both models on the same resampled items. This is a paired comparison, and it matters because the dominant source of item-sampling variance is item difficulty, which affects both models the same way.

The cancellation is worth following step by step, because a paired interval coming out narrower than either of the intervals around the two scores reads at first like an arithmetic mistake. A resample that happens to draw the hardest tickets in the set twice each is a harder eval than the original, and both models score lower on it. Because the indices were resampled once and both models were then scored on that same list, a hard ticket enters A's total and B's total alike, pushing each of them down by amounts that are close to equal — difficulty is a property of the ticket rather than of the model reading it. The gap is one total minus the other, so a quantity that entered both with the same sign and roughly the same size largely cancels in the subtraction. Pairing cancels most of it.

Resampling the two models separately throws that cancellation away, since each score is then computed over its own draw of items, and a draw where A met the hard tickets and B did not produces a difference that is an artefact of two independent draws rather than a fact about the models. On this data the unpaired intervals on the two scores were about ±5.6 points each, while the paired interval on the difference was ±4.3 — narrower than either individual interval, which is the point of pairing. The general form is that pairing removes whatever the two sides share, and on an eval like this one, difficulty is nearly all of what they share.

import random
from statistics import mean

def gap(records_a: list[dict], records_b: list[dict], idx: list[int]) -> float:
    """Weighted total for A minus weighted total for B, over the same items."""
    return aggregate([records_a[i] for i in idx]) - aggregate([records_b[i] for i in idx])

def paired_bootstrap(records_a, records_b, iterations=5000, seed=0):
    rng = random.Random(seed)
    n = len(records_a)
    observed = gap(records_a, records_b, list(range(n)))
    samples = []
    for _ in range(iterations):
        idx = [rng.randrange(n) for _ in range(n)]   # resample once, score both
        samples.append(gap(records_a, records_b, idx))
    samples.sort()
    lo = samples[int(0.025 * iterations)]
    hi = samples[int(0.975 * iterations)]
    crosses_zero = lo <= 0 <= hi
    return observed, lo, hi, crosses_zero

observed, lo, hi, crosses = paired_bootstrap(run_a, run_b)
print(f"gap {observed:+.1f}  95% CI [{lo:+.1f}, {hi:+.1f}]  inconclusive={crosses}")
# gap -6.0  95% CI [-10.3, -1.7]  inconclusive=False

Reading an interval is harder than producing one, and the tempting reading is the wrong one. A gap of six with a range running from a bit under two to a bit over ten does not say the gap is probably six and might be slightly more or less. It says the data are consistent with any gap inside that range, one of under two points included, and that six is the value this particular draw of tickets happened to produce.

Six points with an interval of [1.7, 10.3] clears zero, so the reversal survives — but note how close it came, and note what clearing zero actually buys. It buys one negative statement: on this sample, the position that the two models are equal on this rubric is not tenable. It does not buy the six, and a decision that would go one way on a two-point gap and the other way on a ten-point gap has not been settled by this eval at all.

Had the audited gap been three points instead of six, the interval would have straddled zero and the correct conclusion would have been that 300 tickets cannot distinguish these two models, which is a real and actionable finding rather than a failure. It tells you to either enlarge the set or stop treating the choice as quality-driven and decide on cost, latency or vendor terms instead. The general rule that counterfactual carries is that the width of the interval, rather than the size of the gap, is what says whether a comparison has finished. A gap that is not comfortably larger than the spread around it cannot be told apart from no gap at all, however many decimal places it gets reported to.

Two things widen or narrow that interval, and they are the levers when a result comes back inconclusive. More items narrow it roughly as the square root of the count, so going from 300 to 1,200 halves it — expensive, because each item needs a reference answer. The square root is worth unpacking, because it sets the price of a decisive answer. The spread of an average falls as the square root of the number of things averaged, so shrinking an interval by some factor means growing the item count by that factor squared: four times the items for half the width, sixteen times for a quarter of it. Three hundred tickets to 1,200 is four times the items, the square root of four is two, and the interval comes back half as wide; halving it once more means four times 1,200 items, each of which still needs a human-written reference answer, so cost climbs in step with the count while precision climbs only with its square root.

Replacing judge-model scoring with deterministic checks narrows it too, at no ongoing cost: schema validity, presence of a required field, exact match against a known account ID and the abstention flag can all be computed by code rather than judged, and every component you move from judgement to code removes a variance source entirely. The word to notice there is removes. A deterministic check is a function of the output text, so running it twice on the same output returns the same verdict, and across repeat runs it contributes nothing whatever to the spread. A judge model is not a function of the output text; it is itself a model being sampled, so it can score one answer one way today and another way tomorrow, and that movement is added on top of the movement in the answers being judged.

The stakeholder's objection, answered#

The objection was that the public benchmark rests on 14,042 items and yours on 300. The answer has three parts and none of them is dismissive.

The first is that you have audited your instrument and can describe the audit: the prompt was equalised, the rubric was corrected for a formatting bias that was inflating the result by 1.5 points, and the remaining gap is 6.0 with a 95% interval of [1.7, 10.3] on your distribution.

The second is that their four points have no published interval at all. Most leaderboards report a point estimate from a single run with no repeats, no resampling, and no statement of variance, which means the adjacent entries on the board are frequently separated by less than their own noise. Their 14,042 items make the estimate precise as an estimate of accuracy on their items; it says nothing about the reliability of the ranking, which is what was being cited. Nothing about the paired bootstrap is private-eval machinery either: it needs the per-item scores for two models and nothing else, which the harness behind a leaderboard necessarily held and the published table necessarily throws away.

The third is the distribution argument from the previous chapter, which now lands harder because it is attached to numbers rather than to a principle. Their measurement is precise about short multiple-choice items with a guaranteed answer. Yours is less precise about 8,000-token support tickets where a fifth of the cases have no answer at all. Precision about a different quantity is not a substitute for the right quantity, and the interval is what lets you say that without hand-waving.

An audited eval is not an unbiased one#

The audit does not remove the choices; it makes them visible. A prompt equalisation with a four-hour budget is a defensible protocol, not a neutral one — a team with eighty hours per model would get different numbers, and a team with none would get different numbers again. Every choice in the protocol is still a choice, and the honest output is the protocol plus the number, never the number alone.

It also is not the case that reversals usually survive. A fair share of them do not: the gap shrinks under prompt equalisation, shrinks again under rubric correction, and then disappears inside the confidence interval. That is the audit doing its job, and it is a good outcome — an afternoon of statistics has prevented a model migration justified by noise.

And a clean audit does not make the set itself right. Everything here has treated the 300 tickets as given, checking how they were prompted, scored and read. The tickets were sampled uniformly at random from last quarter's traffic, which felt like the neutral choice, and that sampling decision is a larger problem than any of the three threats this chapter fixed.

That sampling decision is also the assumption the third check ran on and could not test. The bootstrap holds the 300 tickets up as a stand-in for the population, so it reports how much the gap moves when the draw changes and says nothing about whether the draw was aimed at the right population. Sampling uniformly at random weights the eval by whatever last quarter happened to contain, and under-weights the rare, expensive cases that are often the reason for buying an assistant at all. Every resample is built from those same items, so the interval around 6.0 comes back just as narrow either way. Chapter 7, which builds the eval set itself, is where that aim gets chosen.

What the audit buys, then, is not confidence in a number but the right to state one. Three checks that each cost an afternoon took a headline reversal from 9.2 points to 6.0 with an interval around it, and every step of that reduction is something a sceptical reader can repeat. A private eval reported as a bare score is the same kind of artefact as a leaderboard entry with no provenance block; a private eval reported as a protocol, a correction and an interval is a different kind of thing entirely, and it is the only kind worth putting in front of the person who has to sign off the migration.