Regression tests on model output

A regression test on a model call is written the way every other regression test is written — send a fixed input, compare against a stored expected output — and that shape is correct for a pure function and wrong for a generation endpoint. The measured divergence rates of the previous chapter say why: at a per-call divergence rate of a fraction of a per cent, an exact-match assertion fails a few times a month, which is often enough to be annoying and rare enough that re-running the job always works.

The repair everyone reaches for first is a retry decorator, and "flaky external dependency, add retries" is genuinely correct advice in most situations a backend engineer meets. It is wrong here, for a reason worth working through arithmetically rather than asserting, because the same reasoning determines what the assertion should have been instead.

Retries repair transience, and this is not transient#

A retry is the right tool when the failure is a sample from a process whose underlying state is fine. A connection reset, a 503 from a load balancer mid-deploy, a 429 you backed off from: in every case the system under test is healthy and the observation was corrupted in transit. Retrying re-observes the same healthy state, and the probability of three corrupted observations in a row is the cube of a small number.

Non-determinism in generation is not a corrupted observation. The variation is a property of the thing under test, not of the channel to it. Retrying does not re-observe the same output; it draws again from the set of outputs the deployment produces for this request, and the assertion passes if any draw matches. That changes what the test means, and the change cuts in both directions.

Stated as a difference between two models of error, it is the distinction that decides when a retry is ever correct. A transient fault is a noisy view of a fixed underlying state, so repeated observation converges on the truth and the extra attempts are buying information. Non-deterministic generation is a distribution rather than a state, so repeated observation does not converge on anything — each attempt is a fresh draw from the same set, and the extra attempts are buying chances rather than information. A test whose pass condition is "any attempt matched" has stopped asking what the system does and started asking whether the system can do it, which are different questions with different answers.

The false-failure side works exactly as hoped. At a divergence rate of 0.7% per call, three attempts brings the spurious failure rate to 0.007³ ≈ 3 in 10 million, which is effectively never. Three attempts all diverging is three independent unlucky draws, so their probabilities multiply, and a figure below one per cent multiplied by itself three times is small enough to disappear.

The detection side is the problem. Suppose a real regression ships — a prompt change, a model swap, a broken retrieval step — and the new configuration still produces the old golden string on some fraction q of attempts. A single-attempt test catches it with probability 1 − q. A three-attempt test passes if any attempt matches, so it catches the regression only with probability (1 − q)³.

The two expressions come from the same place, which is what makes the comparison fair. Catching the regression means the assertion fails, and with a retry wrapper the assertion fails only when every attempt fails to produce the golden string. Each attempt fails to produce it with probability one minus q, the attempts are independent draws, so all three failing is that quantity multiplied by itself three times. Run a middle row of the table below: at a q of 20% a single attempt catches four cases in five, and four fifths cubed is a little over half, which is the 51% recorded there. At a q of 30% it is seven tenths cubed, which is the 34%.

Put the two calculations side by side and the shape of the mistake is a single sentence: retries cube both numbers, and cubing is a favour to a quantity you want small and an injury to a quantity you want large. The false-failure rate is tiny and becomes negligible. The detection rate is large and shrinks toward nothing. There is no setting of the retry count that improves one without damaging the other, because it is the same exponent acting on both.

A regression that is total — the new system never produces the old string — sits at the top of that table and is caught either way, which is why the retry looks harmless in the cases people imagine when they approve it. The regressions that actually happen are partial. A prompt tweak that degrades one summary in three, a retrieval change that drops an account identifier half the time, a model revision that is subtly worse: these live in the middle rows, where the retry wrapper takes a test that would have caught the problem more often than not and turns it into one that mostly does not. The signal was noisy. Now it is quiet, and quiet is worse, because a test that never fails is indistinguishable from a test that cannot fail.

A close cousin of the retry deserves separating out, because it is a good idea in the wrong place. Calling the model three times and taking the majority answer — self-consistency, in the literature — genuinely does reduce the variance of a system's behaviour, and for a high-stakes classification or a numeric extraction it can be worth the tripled cost and latency. That is a product decision about how the feature behaves in front of users, and if you make it, the test should exercise the three-call system as the unit under test. What it is not is a way to make a single-call feature's test stop failing, which is what the retry decorator does: production still makes one call, and the test now measures something production never does.

The reason the retry looked reasonable when it was approved is worth naming rather than treated as carelessness, because the same reasoning will be offered again. Everyone in that review had seen retries fix a flaky test correctly, many times, and the mental model they were applying — a healthy system, an unreliable channel — is the right model for almost every other flaky test a backend engineer meets. Nothing in the pull request signalled that this case was different, and nothing in the outcome contradicted them either: the failures stopped, which is exactly what a correct fix looks like. What was missing is a question rather than a caution — what is the model of the error here — and asking it takes one sentence and separates the two situations completely.

There is a second cost worth naming. The retry also erases the evidence that the divergence rate exists. The previous chapter's numbers came from running the same request repeatedly and looking at the spread; a CI job that silently swallows two failed attempts per week is running that experiment continuously and throwing the results away.

What a regression test on a model output should assert#

The question that unsticks the exact-match habit is: what does the code downstream of this call actually depend on? Almost never the exact string. Usually a handful of properties, and those are what the test should pin.

Assert properties and invariants. The output parses as JSON. It has the four required keys. total equals the sum of the line items. The extracted account ID matches the one in the input ticket. The tool call was issue_refund with amount=41.50, whatever prose surrounded it. The summary is between 20 and 60 tokens and does not contain a phone number. Each of these is stable across the divergences chapter 2 produces, and each fails loudly when the behaviour you care about breaks.

import pytest
from app.summarise import summarise_ticket

TICKET = open("tests/fixtures/ticket_8841.txt").read()

def test_summary_carries_the_routing_fields():
    result = summarise_ticket(TICKET)

    assert result.account_id == "ACC-88213"          # exact: extracted, not generated
    assert result.category in {"billing", "technical", "account", "other"}
    assert result.category == "billing"
    assert 20 <= len(result.summary.split()) <= 60
    assert "ACC-88213" in result.summary
    assert not PHONE_RE.search(result.summary)

Note what stayed exact. Values the model extracted from the input — an identifier, an amount, a date — should be asserted exactly, because there is a single right answer and a near-tie in the logits is not a licence to return a different account number. Values the model generated get property assertions. Splitting the output along that line is most of the work.

The line is drawable because the two kinds of value differ in how many correct answers exist. An extracted value is determined by the input: the ticket contains one account identifier, so exactly one output is right and every other output is a defect, whatever produced it. A generated value has many acceptable forms, because a summary can be worded a dozen ways and all of them do the job the downstream code needs done. Asserting exact equality is appropriate wherever the set of correct answers has one member and wrong wherever it has many, and that is a question about the value rather than about the model. Working through a response field by field and asking how many correct answers each field admits is what produces the split, and it is a question a reviewer can check.

Assert bounded variation where a property is too coarse. If the thing you depend on is that the answer stays close to a reference answer, say so numerically: an embedding similarity above a threshold you calibrated on real pairs, where an embedding is the numeric vector a separate model assigns to a piece of text so that two rewordings of the same answer sit close together; a numeric tolerance (abs(actual - 41.50) < 0.005); set equality of extracted entities regardless of order. These are weaker than a property assertion and stronger than nothing, and they carry a maintenance obligation — a similarity threshold is a property of the embedding model you chose, so it has to be re-calibrated whenever that model changes. A similarity score has no meaning outside the space the embedding model defines, and a different model defines a different space with a different typical distance between unrelated texts. The number you calibrated is a statement about pairs of your own answers as scored by one specific model, and swapping the model leaves the threshold in the code looking exactly as authoritative as it did the day before while no longer corresponding to anything.

Record and replay where the exact text genuinely matters. Sometimes the thing under test is your parsing code, your prompt template, your streaming handler or your error path, and the model is a fixture rather than a subject. In that case call the model once, record the full response, and replay it from disk on every subsequent run. The recorded file is conventionally called a cassette, after the HTTP-replay libraries the pattern came from. The test becomes deterministic, fast, and free, and it tests exactly what it should.

import json, pathlib, pytest

CASSETTES = pathlib.Path("tests/cassettes")

@pytest.fixture
def replayed(monkeypatch):
    def _use(name: str):
        recorded = json.loads((CASSETTES / f"{name}.json").read_text())
        monkeypatch.setattr(
            "app.client.create_message",
            lambda **kwargs: recorded["response"],
        )
        return recorded
    return _use

def test_streaming_parser_handles_a_split_tool_call(replayed):
    recorded = replayed("tool_call_split_across_chunks")
    parsed = summarise_ticket(TICKET)
    assert parsed.account_id == "ACC-88213"
    # the cassette records what the request was, so a template change is visible in the diff
    assert recorded["request"]["messages"][0]["content"].startswith("You are a support")

Record the request alongside the response. A cassette that stores only the response silently keeps passing after somebody rewrites the prompt template, because nothing compares what you sent this time against what you sent when recording. The replay is a substitution: the client call is swapped for a function returning a stored value, and a function returning a stored value ignores its arguments. Whatever the code assembles and passes in is discarded, so a template that now produces different text produces the same recorded reply and the assertions downstream of it still hold. The defect is inherent in what replay is rather than an oversight in any particular library, which is why the repair is to assert on the recorded request as well. Storing both turns a template change into a visible diff in the assertion or in a checked-in file, which is the behaviour you wanted from the golden test in the first place.

Where the live model still belongs#

Moving the model out of the merge path does not mean never calling it. It means calling it on a schedule, in a job whose failure mode is a notification rather than a blocked pull request. A nightly job that runs the twenty-generation experiment from chapter 1 against a handful of representative prompts, records the first-divergence metrics from chapter 4, and writes them to a time series gives you two things a blocking test cannot: a divergence rate you can quote when somebody proposes an exact-match assertion, and an alert when that rate changes. A jump from 2% to 30% overnight is a real signal about the serving path, and it is invisible to a suite that has no live calls in it at all.

Cassettes need a schedule too, for the opposite reason. A recording is a snapshot of a model's behaviour on the day it was taken, and it stays valid until either the prompt-building code or the model moves. The prompt side is enforceable — store the request alongside the response, as above, and a template change fails the test immediately. The model side is not, so put re-recording on a deliberate cadence: regenerate the cassettes whenever the model identifier you pin changes, and review the diffs as part of that change rather than accepting them in bulk. A cassette directory that has not been touched in a year is testing your code against a model that no longer exists, which is fine for a parser test and misleading for anything else. A suite built entirely on cassettes is a trap for that reason: it will keep passing through a provider silently repointing an alias at a new build, because it stopped talking to the provider. That is not a failure the suite could be improved into catching. A test that makes no network call has no channel through which a change on the other end of the network could reach it, so its silence carries no information about the provider at all. It is the same trade the replay bought deliberately — determinism in exchange for coverage — and the only repair is a separate job that does make the call.

None of these are an evaluation suite. A property test tells you the JSON has four keys and the account ID survived; it does not tell you whether the summary is any good. Quality lives in a scored set with human-labelled references and a pass rate you track over time, and it belongs on a different cadence — nightly, or per model change — rather than in the path of every pull request. Confusing the two produces either a CI job that takes forty minutes and costs real money, or a quality suite that only checks structural properties and reports a comfortable number that means nothing.

Mixing modes is what works: replay for the tests that are really about your code, properties for the tests that are really about behaviour, exact matches only on values the model extracted rather than generated, and a separate scheduled job that calls the live model and reports its numbers rather than blocking merges on them. Each of those decisions follows from one question about the assertion — what does the code downstream depend on — and none of them requires the model to be deterministic.

A green property-based regression test does not mean nothing changed. It means nothing you asserted changed. When the pass rate on a nightly quality run drops by two points, or a customer reports an answer you cannot reproduce, you are back to the question the previous chapter left open: was that the serving variance you have already measured and priced, or did the thing on the other end of the connection actually become a different model? Nothing so far can tell those apart, because nothing so far has recorded anything about the model except a name — and a name is precisely the field designed to stay the same while what it points at moves.