Tests, evals and repeatability

Two kinds of quality check end up in the same repository and want completely different machinery, so the first job in this topic is naming them. A test is what the repository already contains: a fixed input, one correct output, an assertion, and a red build when the assertion does not hold. An eval, short for evaluation, is a measurement of how good a system's outputs are, taken over a set of saved example inputs rather than derived from one expected answer. An eval has three parts — a collection of cases, a way of running the system over each case, and a way of scoring each output — and it reports one aggregate number, usually a pass rate or a mean score, which carries meaning only next to the same number from a previous version.

Evals exist because generated text has no single correct value to assert against. Topic 3.1 works through why the usual testing instincts fail against a generative system, and the short form of it is that two runs of the same prompt come back in different words, both of them acceptable, so an equality assertion has nothing to hold on to. Quality therefore has to be estimated over a sample, the way a poll estimates an election: ask a few hundred and report a proportion. That comparison is worth holding loosely, because it breaks in one specific place. A poll is eventually settled by an election, so the population it was sampling declares itself and the poll can be scored afterwards. An eval has no election day. The only thing its number can ever be held against is the same number from a previous version of the same system.

The scorer can be a rule somebody wrote, such as a regular expression looking for a promised delivery date; it can be a second model prompted with a rubric and asked to grade the answer, which this topic calls a judge; and it can be a person reading transcripts. In practice an eval shows up as a directory of JSONL case files, one JSON object per line, and a rubric file committed next to the application code, plus a job that prints a line like order_assist v7: 84.2 (400 cases) and posts it somewhere a human is supposed to read. That line is the entire output: a name, a version of the case set, one aggregate score, and the count of cases behind it. The benchmark tables in a model provider's release notes are the same construction at a larger scale, and tools such as promptfoo, DeepEval and Langfuse are packaged versions of it. This chapter is about which of a team's checks belong in which pile, and about the one property that decides it.

The contract a merge check has#

A merge check makes two promises to the engineer who triggered it. Run the same commit twice and you get the same answer. And the answer arrives while the engineer still has the change in their head.

Both promises are load-bearing, and the first is the one that decides this chapter's question. A verdict that repeats on the same input can be wired to a red X on a pull request, because when it goes red something in the commit caused it. A verdict that has to be estimated from a sample cannot be, because a commit scoring 4.1 today can score 3.8 tomorrow with nothing changed, and a red build that carries no information about the commit trains the people looking at it to ignore it. Everything else people use to sort checks into these two piles, including the heuristics that mostly work, is a proxy for that property.

The "because" in the middle of that paragraph is carrying an inference, and it is the mechanism the rest of the chapter rests on. A check is a function, and what it is a function of decides what its output means. When a check reads nothing but the commit — the source tree, some files on disk, no clock, no network, no random draw — its verdict is fixed by the commit alone. Two runs on the same commit therefore agree, and that agreement is not merely convenient. It is what licenses the next step: if this commit's run is red and its parent's run was green, the only thing differing between the two runs is the commit, so the commit is the cause. A red X is an argument by elimination, and elimination only works when there is nothing else left to blame.

Add one input that moves on its own — a draw from the model, a different sample of cases, a person having a different afternoon — and the elimination fails. A red run now has two available explanations, the commit and the draw, and the run by itself cannot separate them. That does not make the number worthless. It makes it evidence rather than proof, and evidence is acted on through a decision rule with an error rate somebody chose deliberately: measure the run-to-run spread, decide how large a move has to be before it is worth investigating, and accept that a known fraction of the alerts will be wrong. A red X is not that shape. It is a binary carrying an implied claim of certainty, and a pull-request check has nowhere to record "this is probably the commit".

The second promise fails differently, and it does not degrade gracefully — it either holds or the check has no subject left. A verdict is computed against one state of a branch. If the answer takes long enough that the branch has moved on, the verdict describes a commit that is no longer the head, and the engineer is being asked to act on a measurement of something they have already replaced. Speed here is not about impatience. It is about whether the thing being judged still exists when the judgment lands.

The proxies and where they break#

Sorting a team's whole inventory of checks by repeatability and speed happens at a whiteboard in an afternoon, so people reach for shortcuts. Each shortcut below is a genuine correlate of repeatability, which is why it survives the room, and each fails on a case that turns up in practice.

The most common shortcut gets written into a CONTRIBUTING file in almost exactly these words: if it calls a real model it is an eval, if it doesn't it is a test. That rule sorts by where the work happens rather than by whether the verdict repeats, and it produces two specific mistakes.

It lets a human-rated check into the test suite, because no model was called — and a check whose scorer is a person reading a transcript is an eval by any useful definition, since two annotators handed the same transcript disagree and so does the same annotator on a different day. Inter-annotator agreement on a subjective 5-point usefulness scale typically lands somewhere around 0.5 to 0.7 on Cohen's kappa when the rubric is good, worse when it is not.

Cohen's kappa is agreement with luck taken out of it. Two annotators working the same five-point scale will land on the same box some of the time purely by accident, and the statistic subtracts the agreement that chance alone would produce before reporting what remains, on a scale where one is perfect agreement and zero is no better than guessing. Landing in the middle of that scale is not a sign of careless annotators. A rubric maps a continuous judgment onto five discrete boxes, and two people can agree entirely about what a transcript shows and still disagree about which box it belongs in, because the boundary between "mostly useful" and "useful" is a line the rubric drew rather than a property of the transcript. The same person disagrees with themselves for the same reason, which is why re-rating a batch a week later settles nothing.

The rule's second mistake runs the other way. It pushes checks out of the test suite that belong there, because a model was involved somewhere in producing the artefact being checked. A validator run against a recorded response grades model output and calls nothing:

# tests/test_order_response.py
import json
from pathlib import Path

from assistant.parsing import parse_order_answer, ParseError

FIXTURES = Path(__file__).parent / "fixtures" / "recorded_responses"


def test_parses_recorded_completion_with_trailing_prose():
    # captured from gpt-4o-mini on 2026-02-11, request id req_8f2a1c
    raw = (FIXTURES / "order_status_trailing_prose.txt").read_text()

    parsed = parse_order_answer(raw)

    assert parsed.order_id == "SO-449120"
    assert parsed.status == "in_transit"
    assert parsed.carrier_ref == "DHL-3391847722"
    assert parsed.freeform.startswith("Your order left the Rotterdam")


def test_rejects_recorded_completion_with_unclosed_json():
    raw = (FIXTURES / "order_status_unclosed_json.txt").read_text()

    try:
        parse_order_answer(raw)
    except ParseError as exc:
        assert exc.reason == "unterminated_object"
    else:
        raise AssertionError("expected ParseError")

Both of these run in under ten milliseconds, hit no network, and will return the identical verdict on the same commit for as long as the fixture files exist. The model produced the string being parsed, which is precisely why the fixture is worth having, and none of that makes the check an eval. The stochasticity was spent when the response was recorded; from then on the input is a file on disk.

The split is not new terminology invented for language models. It is the same distinction that separates a unit test from a load test or a canary analysis — the practice of sending a small slice of live traffic to a new build and comparing its metrics against the old one — both of which have existed in deployment pipelines for twenty years and neither of which anybody tries to run on every commit. A load test says the 99th-percentile latency was 340 ms on this build; run it again and it says 361 ms, and the discipline that grew up around it is to trend the number, alert on a threshold, and never wire it to a red X. Hamel Husain's "Your AI Product Needs Evals" makes the same cut for language models, calling the cheap deterministic layer level-1 assertions and putting model-graded and human-graded checks in a separate tier with separate machinery. The vocabulary differs across sources; the property being sorted on is the same one every time.

The other common proxy is cost: cheap means test, expensive means eval. That correlates well enough to survive most conversations, since deterministic assertions usually run locally and sampled evals usually buy tokens by the thousand. It breaks in both directions. A test that spins up a container, seeds Postgres and exercises a retry ladder against a fake transport can take ninety seconds and burn real minutes of build time, and it is still a test because it answers the same way every time. An eval can be cheap — a hand-labelled set of forty cases scored by a regex, with the generation responses cached from last week, costs nothing per run and is still an eval, because the pass rate being read is an estimate over a sample of a distribution, and a different forty cases would give a different number.

Sorting the cases people actually argue about#

Nobody argues about the clear cases. The argument is always over one of five or six shapes, and the criterion sorts all of them without needing a second rule:

The fourth row is the one that teaches the criterion, because temperature 0 is where people reach for an exception. Setting temperature to zero makes sampling greedy — the code that chooses the next fragment stops drawing at random in proportion to the scores and simply takes whichever candidate scored highest — which narrows the output distribution a great deal, and it does not make the verdict repeat. Greedy picking removes the randomness from the choosing step and leaves the scores alone, and the scores are not bit-identical from one run to the next.

Where that comes from takes three links, none of them exotic. The first is that producing a score is a very long chain of additions: every number the model emits is a sum of a great many products, accumulated across the width of the model and then again across its depth. The second is that addition on a computer is not quite associative. Each addition rounds its result to the nearest value the number format can represent, and which value that is depends on the sizes of the numbers being added, so summing the same list in two different orders can leave the totals differing in their last bits. A tiny number added to a huge one can round away to nothing, and whether it gets added before or after the huge number has been assembled is exactly the kind of reordering that changes an answer. The third link is that the order is not fixed by your request. A provider does not run your request on its own. It packs many callers' requests into one batch and computes them together, because that is what makes the hardware worth its price, and how many rows sit in that batch decides how the work is split across the parts of the GPU that run in parallel. That split decides the order in which the partial sums are combined. Batch composition is whoever else happened to be calling at that instant, which is not a property of anything in your repository.

A difference in the last bits of a score is normally invisible, and the reason it sometimes is not is the loop. Greedy picking asks the scores one question: which candidate is highest. When the leading candidate is ahead by any real margin, a rounding difference cannot change the answer and the run reproduces exactly. When two candidates sit within a rounding distance of each other — which is what it looks like when the text genuinely could go two ways — the rounding decides the comparison, and a different fragment gets appended. From that point the model is continuing a different string, so every score after it is computed on a different input, and the two runs can diverge completely. The perturbation is tiny. The loop is what makes it large.

That also explains which case in a set is the one that differs. Length is exposure: every fragment produced is another comparison that could land inside the rounding distance, so a long answer has more chances to hit one than a short answer does. Ambiguity is proximity: a case where the model is genuinely torn between two continuations is a case whose top two scores are close together by construction. Short unambiguous cases reproduce run after run, which is precisely why the belief that temperature 0 means determinism survives so long — it is true of every case anybody checks casually.

Mixture-of-experts routing makes this worse by a different mechanism rather than by more of the same. In a model of that design each fragment is handled by a few of many specialised sub-networks, picked per fragment by a small scoring step, and each expert takes only so many fragments from one batch before it is full. So which experts a token is routed through depends on what else is in the batch: when enough of the other callers' tokens want the same expert, yours is handled by its second choice. That is not a last-bit difference in a sum. It is a different set of weights doing the work, and the score difference it produces is large enough to flip comparisons that were not close.

Two conditions belong on all of that, because it is a claim about a hosted endpoint and readers carry it further than it goes. It describes shared, batched inference. A model served on hardware you control, with the batch size pinned and kernels chosen for reproducibility rather than for throughput, can be made to return identical bits run after run, and some inference stacks offer exactly that as an option with a speed cost attached. A provider is also free to change any of this without announcing it. So the step to make somebody demonstrate is not an argument about what the setting means but a measurement: send the same request to the endpoint you actually call, several hundred times, and count how often the text comes back identical. You will see agreement in the high nineties percent and then a run that differs, usually on the longest or most ambiguous case in the set.

The arithmetic of that residual disagreement decides the question, and it is worth doing rather than arguing about. Suppose a temperature-0 check over twelve cases agrees with its recorded expectation 99.4% of the time per case, which is optimistic. Per case means exactly that: take one of the twelve, run it once, and the text comes back matching what was recorded almost every time. The complement is what matters — six runs in a thousand, for that one case, come back different.

A run is clean only if every case agrees, because a build has no partial credit. One disagreement is a failed assertion, and one failed assertion is a red build. So the quantity to compute is the probability of twelve separate things all happening.

The probability of several things all happening is the product of their individual probabilities, and that step needs a licence rather than being a rule of arithmetic. Multiplying is allowed when the events are independent — when learning that one of them happened tells you nothing about whether another did. Here the twelve cases are twelve separate requests. Each lands in whichever batch the provider was assembling at the moment it arrived, alongside whichever strangers were calling then, so whether case three hit a near-tie in its scores says nothing about whether case seven did. Independence is a fair reading of that, which makes the probability that all twelve agree 0.994 multiplied by itself twelve times — 0.994 to the twelfth power — and that comes to about 0.93.

Two things would break the licence, and both are worth recognising because they push in the opposite direction from the one people expect. If the twelve cases share a cause of failure — sent in one batch, sharing a cached prompt prefix, or straddling the moment a provider swapped a model build — then their outcomes move together and the honest calculation is not a product. Correlation of that kind makes clean runs more common rather than fewer, because the failures clump into a small number of bad runs instead of spreading themselves one per run. And if the per-case rate is not the same for every case — the long ambiguous case being far shakier than the other eleven — then twelve copies of one number is the wrong model, and the run is really no cleaner than its worst case. Independence with a uniform rate is the middle assumption. It is the one to compute first, and the one to say out loud so somebody can argue with it.

What falls out of 0.93 is the figure that decides the question. If ninety-three runs in a hundred are clean then about seven in a hundred are not, and seven in a hundred is roughly one in fourteen, so one run in fourteen goes red with no code change behind it. A team merging thirty times a week triggers that check thirty times, and thirty divided by fourteen is a little over two: the team sees two spurious red builds a week from that one check.

What happens next is not a debate about determinism: somebody adds a retry, then somebody adds a second retry, then people stop reading the failure and press re-run on reflex, and from that point the check no longer distinguishes a real regression from noise even when it fires correctly.

That reflex deserves better than being called laziness, because it is the correct response to the check's own statistics. An engineer looking at a red build is choosing between two explanations: my commit broke something, or the check flaked. Which is more likely depends on how often each happens. Two flakes a week, against a genuine regression that this check catches far less often than that, means a red build is overwhelmingly likely to be noise, and pressing re-run is what the evidence supports. Every individual decision to re-run is defensible. Their aggregate is a check nobody reads.

The retries do not rescue it either, and the reason is worth stating because adding one feels like engineering rather than surrender. Running the check twice and accepting the second verdict does drive the noise down: the failure rate becomes roughly its own square, which is a large improvement. It buys that improvement by defining away the signal the check existed for. A real regression that reddens only some of the time is, to a retry, indistinguishable from a flake, and the retry passes both. The check ends up reliably green on exactly the failures that were hardest to spot by reading the diff. That is the same failure mode as any flaky test in an ordinary service, and it is why the criterion is about repeatability rather than about how sure anyone feels that the model will behave.

Moving a check across the line#

The most useful consequence of the criterion is that a check is not permanently one or the other. It can be moved by changing how it is measured, and the delivery-date constraint from the table shows both directions.

As specified — 150 sampled orders, model-graded — it is an eval and it cannot block a merge. Narrow it: fix a set of thirty adversarial prompts that historically provoked date promises, freeze the model responses to those prompts as fixtures, and assert with a regex over date patterns that none of the recorded responses contains one. That check is deterministic, runs in a second, and blocks a merge honestly. What it now guards is a regression in the parsing and post-filtering layer, rather than the model's behaviour changing underneath you — the frozen fixtures cannot notice a new model version that starts promising dates.

The other direction keeps the sampling and removes the stochastic grader. Replace the model judge with a deterministic date-extraction rule and run it against cached generations, and the verdict repeats for as long as the cache does. That version still measures behaviour across 150 real orders, and it goes stale on a schedule set by how often the cache is refreshed. Each version measures something narrower than the original, and the discipline is to write down what each gives up next to the check itself, because a narrowed check with its concession forgotten reads like full coverage a year later.

Sorting a check into the eval pile is a statement about the machinery it needs rather than about its importance, and it is worth being explicit about that, because the sorting exercise reliably produces a room where the business-critical properties all end up on the side that cannot block anything. That is the correct result and it is also an uncomfortable one. Deterministic tests do not measure whether the assistant is useful; the vendor rating does. The response is not to force the rating into a merge gate where it cannot work, but to build machinery that takes it seriously somewhere else, which is what the rest of this topic is about.

The criterion also stops short of telling anyone what to do with a check once it is sorted. Knowing that a rating is an eval says nothing about where it runs, how often, over how many cases, or what happens when the number drops. Those are separate decisions with separate costs, and a wrong one produces a gate that spends money and protects nothing.

What is settled is the sorting rule itself, and it is worth applying literally. For each check around a feature, ask whether the same commit produces the same verdict, and whether that verdict arrives in minutes. Two yeses make it a test and it belongs on every commit. A no on either makes it an eval, whatever it measures, however cheap it is, and whether or not a model is involved anywhere in it. Writing the reason down beside each answer is the part that survives contact with next quarter, because the reason is what says which of the two narrowing moves above could bring a check back across the line.

The next chapter does that sorting on a real feature — prompt rendering, retry behaviour, schema validation and answer quality — and produces the two piles this topic then has to find homes for.