An eval score is a draw from a distribution, not a reading

The first thing worth doing with a complete run record is to run the same thing twice. Take a run whose dataset hash, evaluator version, evaluator configuration, model snapshot, prompt hash, and harness commit are all identical, and execute it ten times in a row on a Tuesday afternoon while nothing is being deployed. On a 2,000-example support set scored by an LLM judge for faithfulness, ten such runs produced:

0.8120  0.8070  0.8190  0.8090  0.8150
0.8040  0.8110  0.8180  0.8060  0.8130

The mean is 0.8114 and the standard deviation is 0.0051, so the widest gap between any two of these runs is 1.5 points. Every one of those runs is defensible, reproducible, and pinned to the last byte, and they still do not agree. A team that reads 0.804 on Monday and 0.819 on Friday and goes looking for what changed will find something, because something always changed, and the finding will be wrong.

Pinned inputs do not buy a pinned output#

The nondeterminism has several sources and none of them are a bug in the harness. Serving a large model involves batched matrix operations on GPUs, and floating-point addition is not associative, so the sum of a batch depends on how work was partitioned across threads and in what order the partial results were reduced. Batch composition depends on what other traffic hit the same replica at the same moment, which means an identical request can produce logits — the raw per-token scores the model converts into a probability distribution over its vocabulary — that differ in the last few decimal places. Where two tokens are close in probability — which is exactly the situation at the boundary of a judgment like "is this answer faithful to the retrieved context" — that difference flips the sampled token, and one flipped token flips the verdict for that example.

Requesting the least random decoding available does not remove this. It removes the sampling noise you control and leaves the numerical noise you do not. Mixture-of-experts routing adds another layer: in that architecture each token is sent to a small subset of the network's specialised sub-networks rather than through all of them, and which experts a token is routed to can depend on the composition of the batch. Beyond the model there are harness-level sources: a request that times out and is retried lands in a different batch; a judge response that fails to parse falls through to a default verdict; a rate-limit backoff changes the concurrency profile and therefore the batching. None of these are visible in the aggregate score, which is why the aggregate score looks like a measurement rather than a sample.

The scale is worth stating concretely. At 0.8114 mean and a standard deviation of 0.0051 over 2,000 examples, the judge is changing its mind about roughly 30 to 40 individual examples between runs — a per-example flip rate of around 1.5 to 2 percent. Aggregation hides most of it because flips go in both directions and partially cancel, and that cancellation is exactly why the aggregate wobbles by tenths of a point rather than by whole points.

Not all of that is irreducible, and it is worth separating the part you own from the part the provider owns before accepting a wide band as inevitable. Harness-level variance is under your control and usually cheap to remove: fix the concurrency rather than letting it scale with the machine, count parse failures as errors instead of silently defaulting them to a pass, cap and record retries rather than letting a flaky minute of API weather change how many examples got a second attempt, and score in a deterministic order so that a partial run truncates predictably. A team that does none of this often measures a band twice as wide as the model's own contribution and then reports that evaluation is too noisy to act on. Measuring the band first and reducing the controllable half second is the right order, because the measurement tells you how much room there is to gain.

Measure the band rather than asserting it#

The procedure is short. Pick a run configuration, freeze every field the previous chapter listed, execute it k times, and characterise the spread of the resulting scores. Two variants answer different questions, and running both costs less than it looks:

import statistics

def variance_band(scores: list[float], z: float = 2.0) -> dict:
    mean = statistics.fmean(scores)
    sd = statistics.stdev(scores)          # sample sd, k-1 denominator
    return {
        "k": len(scores),
        "mean": round(mean, 4),
        "sd": round(sd, 4),
        "band": round(z * sd, 4),          # ± this much is a wobble
        "observed_range": round(max(scores) - min(scores), 4),
    }

judge_only = [0.8120, 0.8070, 0.8190, 0.8090, 0.8150,
              0.8040, 0.8110, 0.8180, 0.8060, 0.8130]
print(variance_band(judge_only))
# {'k': 10, 'mean': 0.8114, 'sd': 0.0051, 'band': 0.0102, 'observed_range': 0.015}

The choice of k is a trade between confidence in the band and the cost of measuring it. A standard deviation estimated from k samples is itself uncertain, with a relative uncertainty of roughly one over the square root of twice k minus two: about 35 percent at k=5, 24 percent at k=10, and 16 percent at k=20. Below about eight repeats the band is too vague to defend in an argument, which defeats the purpose; above about twelve the cost grows faster than the precision improves. Ten is a reasonable default, and rounding the resulting band upward to a round number — 0.0102 quoted as one point — is honest rather than sloppy, since the estimate does not support a third decimal place.

The cheap variant regenerates nothing: the system under test's outputs are produced once, cached by example hash, and only the judge is re-run. That isolates evaluator nondeterminism and costs one judged pass per repeat. On a 2,000-example set where each judge call sends about 1,400 input tokens and returns about 180, a pass is 2.8 million input tokens and 0.36 million output tokens; at Sonnet-class list pricing of $3 per million input and $15 per million output, that is roughly $14 and about eight minutes at a concurrency of 16.

The expensive variant regenerates the system's answers too, which is the number that actually governs whether a release moved anything. Generation on the same set runs about 900 input and 350 output tokens per example, so a full end-to-end pass is closer to $30 and 17 minutes. Ten repeats of the end-to-end variant is about $300 and three hours of wall clock — a one-off cost, run once per pipeline configuration, that converts every subsequent comparison from an argument into arithmetic.

The third row is the one people are surprised by. Nothing about the pipeline changed; the set got smaller, and the band nearly doubled relative to the 2,000-example measurement. That is not an accident of these particular numbers. Averaging n independent per-example outcomes gives a standard error that scales as one over the square root of n, so shrinking a set by a factor of four widens the band by a factor of two. It is why a team that trims its eval set to keep nightly costs down often finds that its dashboard suddenly seems noisier, and why the correct reading of a five-point move on a 200-example set is usually "not measurable."

Two bands, and they answer different questions#

The band above measures repeated runs on the same examples, and it is the right yardstick for the comparisons in the next three chapters: a restore of an old dataset version, a crossed rerun of an old evaluator, a rerun after a code change. In all of those the example set is held fixed, so example-to-example sampling variation cancels — the same hard cases are hard in both runs, and only the pipeline's own nondeterminism separates the two numbers.

A different band applies when the two numbers were computed on different samples of examples, which is the situation whenever a set grows, shrinks, or is re-sampled. There the dominant uncertainty is which examples happened to be included, and it is approximated by the standard error of a proportion:

Conflating the two bands is the common mistake, and it fails in both directions. Applying the wider sampling band to a same-set comparison discards real regressions as noise; applying the narrower run-to-run band to a comparison across two different sets manufactures findings out of resampling. Chapter 6 deals with the second case directly.

For a same-set comparison there is a sharper instrument than either band, because you have the per-example scores rather than only the aggregates. Compare them pairwise and bootstrap the mean difference, which means drawing thousands of resamples of the per-example differences with replacement, taking the mean of each, and reading the middle 95 percent of those means as the range the true difference plausibly occupies:

import random, statistics

def paired_bootstrap(a: dict[str, float], b: dict[str, float],
                     iters: int = 10_000) -> tuple[float, float]:
    """a, b map example_id -> score, over the same example set."""
    ids = sorted(set(a) & set(b))
    diffs = [b[i] - a[i] for i in ids]
    n = len(diffs)
    means = []
    for _ in range(iters):
        sample = [diffs[random.randrange(n)] for _ in range(n)]
        means.append(statistics.fmean(sample))
    means.sort()
    return means[int(0.025 * iters)], means[int(0.975 * iters)]

lo, hi = paired_bootstrap(march_scores, june_scores)
# (-0.0812, -0.0588) — the interval excludes zero, so the drop is real

An interval that straddles zero says the two runs are consistent with no difference. An interval entirely below zero, as here, says the June pipeline is worse on these examples by somewhere between 5.9 and 8.1 points, which is a claim with a size attached rather than a direction. The pairing matters: examples that are hard for both runs contribute a difference near zero and shrink the interval, whereas an unpaired comparison of two aggregates spends its precision re-measuring the same difficulty distribution twice.

What a band is not#

It is not a statement about users. A faithfulness score inside its band means the two runs are indistinguishable on this dataset with this evaluator, and says nothing about whether either is good enough to ship. Product significance and statistical significance are separate questions, and a 2-point regression that clears the band may still be irrelevant to anyone, while a 0.5-point regression buried inside the band may be catastrophic if it is concentrated in the highest-value segment. Aggregate bands are blind to concentration by construction.

It is also not permanent. The band belongs to a specific pipeline: this judge model, this rubric, this set size, this concurrency. Change the judge model and you have a different band, usually a different one by a factor of two or more, because judge nondeterminism varies with how close the rubric puts the decision to a coin flip. Treat a measured band as an artefact with its own version — recorded in the run store, stamped with the evaluator configuration hash it was measured under, and re-measured whenever that hash changes. A band quoted from six months ago against a judge that has since been upgraded is exactly the kind of stale reference chapter 1 was about.

Finally, a band tells you when a difference is worth explaining and never what explains it. It converts "0.83 against a recorded 0.85" from an opinion into a decision with a stated basis, which is all it is for. Whether the version machinery those recorded numbers depend on actually works — whether an old dataset version can still be fetched, assembled, and scored at all — is a separate question that nobody answers by looking at a dashboard, and that most teams have never tested. The next chapter tests it.