Repeatability in an eval tool trial

An eval tool is the software that surrounds an evaluation: it holds a fixed set of cases, runs each one through the system you are testing, scores the answers with one or more graders, and keeps the results somewhere you can compare today's run against last month's. It exists because the check in the middle cannot be an equality assertion. A customer asking whether a worn pair of boots is still covered by a returns policy has a hundred acceptable answers and no single expected string, so the verdict has to come from somewhere else — either deterministic code of the kind topic 3.3 covered, on checks a machine can run without a model, or a second model reading the answer against a written rubric, an arrangement generally called an LLM-as-judge. Every team starts by running a handful of cases by hand and keeping the verdicts in a spreadsheet, and an eval tool is what that spreadsheet becomes once there are hundreds of cases, several people looking at them, and a reason to want last quarter's number back.

You meet these tools in three shapes, and the rest of this topic compares them: a hosted platform with a database behind it, a review interface for the people who label cases, and a per-seat or per-run price; a command-line tool that keeps cases, graders and configuration as files inside your own repository; and a runner you write yourself, which starts at a few hundred lines. Choosing between them starts with a measurement that no demo takes, which is how far a candidate tool's score moves when nothing else moves at all.

The reason a demo cannot answer that question is not that a demo is a sales event. It is that a demo is one run. A grader that consults a model produces a verdict by sampling, so the score it reports on any given afternoon is one draw from a distribution of scores the same suite could have produced. Two properties of that distribution matter and they are independent of each other: where it sits, which is a fact about how good the assistant is, and how wide it is, which is a fact about how firmly the tool can hold a verdict. A demo shows you one draw, and one draw locates nothing about width. It is not that the demo number was misleading — it was a real measurement of the first property. Nobody had taken a measurement of the second, and the second is the one a merge gate depends on.

That measurement is the subject here, and one set of numbers makes it concrete. Wiring an identical run into CI — continuous integration, the pipeline that runs your checks automatically on every commit — produced 0.84 on the first nightly build, 0.89 on the second, and 0.86 on the third. No commit in between touched the prompt, the retrieval index, the model id, or the case set. Those three numbers are not a fact about the assistant being tested. They are a fact about the tool, and separating the two is what the rest of this chapter is for.

What the first real run exposes#

Getting to those three numbers took four days, and almost none of that was spent on the score. Most of it went to the dataset format, which is where first runs stall. One tool's dataset schema wanted three fields per case:

{"input": "string", "expected": "string", "metadata": {"key": "value"}}

Real cases are not that shape. A returns conversation carries a customer turn, two prior turns of context, the four retrieved policy chunks that went into the prompt, the order record the tool call returned, and a reference answer written by someone in support. Flattening that into one input string is mechanical:

def to_vendor_case(case: EvalCase) -> dict:
    prompt = render_prompt(
        history=case.history,
        chunks=case.retrieved_chunks,
        order=case.order_record,
        question=case.question,
    )
    return {
        "input": prompt,                      # the fully rendered prompt
        "expected": case.reference_answer,
        "metadata": {
            "case_id": case.id,               # survives as an opaque string
            "locale": case.locale,
            "policy_version": case.policy_version,
        },
    }

It runs, and it costs two things that are not obvious until later. The grader now receives one blob of text, so a check like the answer must cite only facts present in the retrieved chunks has no separate chunk field to look at; it has to find the chunks inside the blob by convention. And case_id has become a metadata string rather than a primary key, which matters enormously and is the subject of the next chapter, on where the artifacts of an eval run have to live.

The rest of the four days went to the things nobody demos: getting an API key into the CI environment without it landing in logs, discovering that the tool's CLI — its command-line entry point, the thing CI actually invokes — exits 0 whether the suite passed or failed, so the gate, meaning the build step that blocks a merge when the score drops, needed --fail-under which turned out to be --threshold, and finding out that a run of 200 cases takes 11 minutes at the tool's default concurrency of 4, which is longer than an existing unit test suite. None of that is the problem. All of it is ordinary integration work, none of it is in anybody's quickstart, and any tool would have some version of it. The problem is the three numbers.

Measuring the grader instead of the assistant#

The right response to 0.84 / 0.89 / 0.86 is to stop reasoning about it and measure it, which takes one afternoon and a loop. Freeze the commit, freeze the case set, and run the identical suite ten times, keeping the per-case verdicts rather than only the aggregate:

for i in $(seq 1 10); do
  evaltool run \
    --dataset returns-v3 \
    --config eval.yaml \
    --output "runs/repeat-${i}.json"
done

jq -s '
  [.[] | .cases[] | {id: .metadata.case_id, pass: .passed}]
  | group_by(.id)
  | map({id: .[0].id, passes: (map(select(.pass)) | length)})
  | map(select(.passes > 0 and .passes < 10))
' runs/repeat-*.json

The result on a 50-case returns suite was that 47 of 50 cases returned the same verdict all ten times, and three did not. Case ret-0117 passed 6 times out of 10, ret-0142 passed 3 out of 10, and ret-0009 passed 9 out of 10. Three cases out of fifty is 6% of the suite, and because each case is worth 0.02 of the aggregate, those three alone move the headline number over a range of 0.06 — wide enough to contain the 0.84-to-0.89 spread that showed up in CI, arriving from a completely stable system.

The arithmetic in that sentence is worth doing slowly, because it is what converts a vague sense of flakiness into a bound. Fifty cases, each scored pass or fail, produce an aggregate that is the count of passes divided by fifty. Moving one case from fail to pass therefore moves the aggregate by one fiftieth, which is 0.02. Forty-seven cases never move, so they contribute the same amount to every run and cannot be the source of any difference between runs. The remaining three can each be on or off independently, so the aggregate's lowest possible value is the one where all three fail and its highest is the one where all three pass, and the distance between those two is three times 0.02. Every observed spread has to fit inside that interval, and the CI numbers do.

Reading the three cases makes the mechanism concrete rather than mysterious. ret-0142 is a Dutch refund case, where the assistant answers correctly but mentions the refund window in a way the rubric can read either as a policy citation or as an unrequested upsell. The judge is being asked a question that genuinely has two defensible answers, it is sampling at whatever temperature the vendor picked, and it lands on one or the other depending on nothing you control. ret-0009 fails once in ten, and that one is worse, because it fails for no legible reason at all — the judge's rationale on the failing run contradicts its own rationale on the nine passing ones.

Those two failure modes call for different responses, which is why the per-case data is worth more than the aggregate. An ambiguous case is a rubric problem: either the rubric gets a clause that decides the question, or the case is retired from the gating set and kept in a separate advisory set where a wobbling verdict costs nothing. An illegible flip is a grader problem, and no amount of rubric editing fixes it, because the same rubric produced both answers. A tool that reports only 0.86 gives you neither diagnosis, and a team working from aggregates alone will spend the next month editing prompts in response to noise.

Now put a gate on top. If the aggregate over fifty cases is roughly normal around 0.865 with a standard deviation of about 0.018 — which is what a spread of 0.06 across ten runs implies — then a threshold at 0.85 rejects a correct commit about 20% of the time.

Both halves of that sentence are derivations rather than assertions, and neither is long. The standard deviation comes out of the spread because the two are related by how many runs you took: across ten draws from a distribution of this shape, the gap between the largest and the smallest typically runs to around three standard deviations, so dividing an observed spread of 0.06 by about three recovers a standard deviation near 0.02. The rejection rate then comes from where the threshold sits relative to that. The centre is 0.865 and the threshold is 0.85, so the gap between them is a little under one standard deviation. Roughly a fifth of a normal distribution lies further below its centre than that, and a fifth is where the 20% comes from. Both steps are rough, and they are rough in the direction that matters: they establish the order of magnitude of the false-rejection rate, which is all a decision about gating needs.

One nightly build in five goes red with nothing wrong, and the fix each time is to press re-run. Three weeks of that and nobody reads the eval result at all, which leaves you paying for a gate that has been quietly reclassified as decoration.

The three controls that decide whether a tool can gate#

A judge that varies is not automatically disqualifying. What disqualifies a tool is varying with no exposed control over the variance, because then you cannot bound the spread and you cannot design a threshold against it. There are exactly three controls worth checking for, and they are cheap to check.

The third is the one that matters most and is offered least often. Repeats convert a coin flip into a measurement: running each case five times and scoring it by majority reduces the per-case flip rate substantially and, more importantly, hands you a standard deviation you can set a threshold against.

Why a majority is more stable than a single verdict is worth working out on the three unstable cases above, because the improvement is uneven and the unevenness is the useful part. ret-0009 passes nine times in ten. For a majority of five to come out as a fail, at least three of the five draws have to be the one-in-ten failure, and three independent one-in-ten events landing together is very much rarer than one of them landing on its own — so majority voting takes an already rare flip and makes it negligible. ret-0117 passes six times in ten, and a majority of five draws from something close to a coin lands on the pass side more often than not, but not reliably. ret-0142, at three in ten, is pushed the other way, toward a stable fail. The pattern is that repeats push every case away from the middle and toward whichever verdict it already favours, and they do very little for a case sitting at an even split. That is why repeats do not remove the need to fix an ambiguous rubric; they make a mostly-decided case behave as decided, and they leave a genuinely undecided one undecided while telling you it is.

The second thing repeats buy is the one that makes gating possible at all. Five verdicts per case is five aggregate scores per run, and five numbers have a spread you can compute. A threshold placed one or two of those spreads below the observed mean is a threshold with a known false-rejection rate, rather than a round number somebody picked. Without repeats there is one number and nothing to compare it against.

It costs five times the judge tokens, which on a 200-case suite with an 800-token judge prompt takes the judge's input to about 800,000 tokens a run, five times what a single pass sends — a few dollars at current mid-tier judge pricing, which is nothing against the cost of a gate people ignore. A tool that cannot do this natively can sometimes be forced into it with a shell loop like the one above, and that is a legitimate workaround as long as the tool at least gives you per-case verdicts in a machine-readable file. A tool that returns only an aggregate has closed even that door.

Seeds deserve one caveat so you do not over-trust them. A seed makes a provider's sampling reproducible in principle, and in practice identical seeded requests to a hosted model still disagree at a low rate because batching changes floating-point reduction order on the GPU. Seeds move you from "varies constantly" to "varies rarely", which is worth having and is not determinism.

The mechanism under that caveat is worth one paragraph, because it is the reason no amount of configuration removes the last of the nondeterminism. A model's next token is chosen from a set of scores, and those scores are computed by summing very large numbers of floating-point products. Floating-point addition is not associative: adding the same quantities in a different order can give a very slightly different total. A provider batches your request with whatever other requests arrive at the same moment, and the batch size changes how the sums are partitioned across the hardware. So the same prompt with the same seed can produce scores that differ in their last digits, and on a token where two candidates were nearly tied, that difference decides which one wins. Nothing you send in the request controls what else is in the batch, which is why the residual rate is low, not zero, and not yours to set.

Sampling error and grader variance answer to different remedies#

The vendor's support answer was not evasive. Increasing the sample size is genuinely good advice for a different problem — the sampling error that comes from having chosen fifty cases rather than five hundred — and at fifty cases that error is around ±0.10 at 95% confidence even with a perfect grader. What more cases will not fix is a grader that disagrees with itself on a fixed set, because that variance does not shrink with the size of the case set; it shrinks with repeats per case, and only if the tool will do repeats. The two sources of movement look identical in the headline number and respond to completely different remedies, which is one reason a tool that reports a single figure with no interval attached is hard to reason about at all.

That claim needs one condition attached, because as stated it is stronger than the arithmetic supports. If the wobbling cases are independent of one another, then averaging over more of them does steady the aggregate, at the same square-root rate that governs every other average — five hundred cases with 6% of them unstable produce a narrower headline range than fifty cases with 6% of them unstable. Three things stop that from being the remedy. More cases are not free, since each one has to be written and given a reference answer, whereas repeats reuse the cases you already have. More cases do nothing at all for the per-case verdict, and a per-case verdict is what an investigation into a specific regression needs. And a larger set changes what is being measured, whereas repeats improve the precision of the same measurement. Repeats attack the source; a bigger set dilutes it.

Nor does any of this say a deterministic grader is a correct grader. Topic 3.3 covered the deterministic checks in detail, and a regex that repeats perfectly while measuring the wrong property is worse than a noisy judge measuring the right one, because it fails silently. Repeatability is a precondition for gating and nothing more than that; a tool can clear this bar and still be the wrong choice on every other axis, and a tool that fails it can still be the right place to argue about twelve failures with a product owner.

And a single trial does not resolve the question a trial was supposed to answer. Knowing that one candidate cannot gate means running this exercise again against another tool, and possibly a third. The comparison guides are useful for narrowing the shortlist — LLM Evaluation Tools: The Complete Comparison Guide covers the current field feature by feature — but the shortlist is not the decision, and the second run on your own data is.

The disqualification this chapter argues for is deliberately narrow. A tool that varies without exposing a seed, sampling settings or repeats cannot be the thing that blocks a merge, because you cannot place a threshold against a spread you have no way to bound; the ten-run measurement that establishes this costs an afternoon and is the cheapest thing in a trial. Everything else about the tool — the interface, the failure browser, the price — is a separate argument, and it is one worth having on its own terms rather than through a number that turns out to be measuring the ruler.

The part worth noticing before the second trial starts is what four days produced besides a verdict on one vendor. There are now fifty cases in a structured form, ten runs of per-case verdicts, a reference answer for each case, and three cases annotated with the reason they are ambiguous — and every one of those things is currently sitting inside a vendor's dataset object, keyed by an identifier they minted. If the tool changes next month, the question is which of it comes with you.