Regression fixtures from an incident

Every assertion written in this topic so far has been evaluated on runs that were meant to pass. That is the normal condition of a test suite and it hides a specific defect: an assertion whose predicate cannot be false on any input it will ever see. It costs the same to run as a real check, appears in the same green summary, and is indistinguishable from the outside. The instrument that tells them apart is a fixture — a frozen run record plus the case that produced it plus the verdict the checks are required to return on it, replayed with no provider call at all.

This chapter is about building those fixtures and about the ordering discipline that gives them their value, which is borrowed from test-driven development with one adjustment: the failing input is not invented, it is recovered from an incident that already happened.

What an audit of assertions turns up#

One team instrumented their suite to record, for every assertion, the last time it had rejected anything. Six months of CI history over 312 assertions across 300 cases read: 41 assertions had failed at least once, 271 had never failed. The reasonable interpretation is that most assertions guard behaviour that has simply not regressed. The other interpretation is that some of them cannot fail, and the report gives you no way to tell which.

At least one of them could not. This had been in the tool-checking module for four months:

for call in record.tool_calls:
    assert call.name == "issue_refund"
    assert call.arguments_parsed["order_id"] == case.order_id

It reads like a check that the agent issued the right refund. On a run where the agent called nothing at all — refused, apologised, ended the turn — record.tool_calls is empty, the loop body never executes, and the case passes. The single behaviour most worth catching in a refund agent, doing nothing when it should have acted, was the one input this assertion was structurally incapable of rejecting.

Why it passes is worth stating as a property rather than as a quirk of one language, because the same shape appears in every suite that walks a list. Each assert in that loop states something about a value the loop is holding. A loop over an empty list never holds a value, so neither predicate is evaluated and neither can come out false. The check does not say what its author believed it said. What it says is that every tool call the agent made was issue_refund carrying the right order id — a claim that is true, and trivially so, of a run in which the agent made no calls at all. A statement of that form is true whenever there is nothing to check, and that is not a defect in the language; it is what "every" means.

Naming the gap that way supplies the repair. A claim about behaviour that should have happened is a claim that something exists, and existence is never established by iterating over a collection — iteration can only qualify whatever is already there. It has to be asserted about the count: exactly one side-effecting call, or at least one, stated before anything is said about what that call contained. Any assertion nested inside a loop is silent on the empty case by construction, so the cardinality is either a separate assertion or it is not made at all. The rule generalises past tool calls to citations, to retrieved documents, and to every list a checking module walks.

The instrumentation that produced that audit is worth building before the practice that fixes it, because it is what tells you where to spend the effort. Give every assertion a stable identifier, have the checking module emit one row per assertion per case with its verdict, and keep a rolling table of assertion_id, runs, rejections, last_rejection_at. On a suite that runs on every merge this is a few thousand rows a day and answers a question nobody could otherwise answer: which of our checks have ever done anything.

Red first, on a real artefact#

Every incident already produced the artefact needed to settle it — the run that went wrong, captured as the record of chapter 3. Freezing it turns an outage into a permanent, free assertion about your assertions.

evals/fixtures/2026-06-14-refund-off-by-100x/
├── case.yaml          # the input and expectations, as the suite would run it
├── record.json        # the RunRecord captured during the incident
├── expect.yaml        # verdict: reject; by: assert_refund; reason: amount_cents
└── README.md          # two paragraphs: what happened, ticket link

The replay harness never calls a provider. It loads the record, runs the assertion module against it, and compares the verdict to expect.yaml:

@pytest.mark.parametrize("fixture", sorted(FIXTURE_DIR.iterdir()), ids=lambda p: p.name)
def test_fixture(fixture):
    case = load_case(fixture / "case.yaml")
    record = RunRecord.from_json((fixture / "record.json").read_text())
    expect = yaml.safe_load((fixture / "expect.yaml").read_text())

    result = run_all_checks(record, case)      # no network, no model

    if expect["verdict"] == "reject":
        assert not result.passed, "fixture was accepted; the check has no teeth"
        assert result.failed_check == expect["by"], \
            f"rejected by {result.failed_check}, expected {expect['by']}"
    else:
        assert result.passed, f"good run rejected by {result.failed_check}"

The second assertion in the reject branch is the one people leave out and then wish they had. A fixture rejected by some check is weak evidence; the failing $0.42 record might be rejected because a schema field was also missing, and the tool-argument assertion you actually wrote would still be untested. Pinning which check must do the rejecting is what makes the fixture a test of that check rather than of the pipeline in general.

Underneath that pinning is a chain worth following link by link, because it is what separates a regression test from a keepsake. The replay harness calls no provider, so the model is not exercised at any point; what runs is the checking module, against bytes that cannot change. A fixture is therefore a test of a check and of nothing else. It establishes one sentence: given exactly this record, this named assertion returns reject. Getting from there to "the incident cannot recur unnoticed" takes a second link, and the second link is the one that breaks quietly — the check has to reject the record because of the property that made the run bad, and not because of some other property this particular record happens to carry alongside it.

An example makes the failure concrete. Suppose the incident was a refund issued at a hundredth of the right amount, and the fixture is rejected because the response also happened to be missing a citation. The file is committed, the suite is green, the outage is commemorated in the repository, and the next refund off by a factor of a hundred goes through untouched, because the assertion that fired was never looking at the amount. expect.yaml naming by: assert_refund and reason: amount_cents forces that second link to be verified at the moment the fixture is written, which is the only moment anybody still remembers what went wrong.

The same reasoning marks out what a fixture cannot promise. It preserves one recorded route to the failure, so it catches a regression that produces a record carrying that property again. A model that arrives at the same bad outcome by a different route — the wrong amount reaching the tool through a differently shaped call, or through a second call that undoes the first — produces a record the frozen one says nothing about. The fixture is not evidence that the incident cannot happen again. It is evidence that the check written after it still works.

The protocol that produces a fixture with both links intact is three steps in a fixed order. Capture the record and commit it with verdict: reject, at which point the fixture test fails, which is the evidence you wanted — the check under construction does not yet exist or does not yet work. Write the assertion until the fixture is rejected for the stated reason and nothing else changes. Then land the behavioural fix and add a second fixture, captured after the fix, with verdict: accept.

A check that rejects everything also rejects the bad record, so a reject-only fixture cannot distinguish a precise assertion from assert False. The pair — one run that must fail, one nearly identical run that must pass — is what pins the boundary.

The pair works because its two halves fail in opposite directions, and each catches the degenerate check the other cannot see. A check that never rejects anything, of which the vacuous loop above is one, sails through the accept fixture and is caught by the reject fixture. A check that rejects everything sails through the reject fixture and is caught by the accept one. Only a predicate that actually separates the two runs satisfies both at once, which is the sense in which neither fixture is worth much on its own.

How tightly the pair pins the check then depends on how alike the two records are, and this is where the practice is usually done carelessly. If the failing run and the passing run differ in one field, any check that rejects the first and accepts the second must be reading that field, or something computed from it. If they differ in twenty fields — a different customer, a different day, a different prompt version, because the post-fix record was grabbed from whatever traffic came along afterwards — then "rejects one, accepts the other" is satisfied by a check keyed on any one of the twenty, and the pair has stopped being evidence about the assertion. The word doing the work in the definition of the pair above is nearly: capture the accept fixture by re-running the incident's own inputs after the fix, so the two records differ in what the fix changed and in almost nothing else.

Recovering the record is the step that fails most often in practice, and it fails for an operational reason rather than a conceptual one. An incident happens in production, where traces are sampled — one percent is a common default — and retained for a fortnight, so by the time finance notices a pattern the evidence for the specific request may be gone. Two cheap adjustments prevent this: sample at one hundred percent on any request that invoked a side-effecting tool, and copy the full trace to durable storage whenever a request is flagged by a support ticket or an error. Neither is expensive relative to the model calls that produced them, and both convert "we think it sent 42" into a file you can commit.

One percent is worse odds than it sounds, and the arithmetic is what explains why those two adjustments are written as rules rather than as a higher sampling rate. Head-based sampling decides at the start of a trace, by a flip made independently for each request, whether that trace is kept. The chance one particular request survived is therefore one in a hundred, and the chance that none of several occurrences survived is ninety-nine in a hundred multiplied by itself once per occurrence — multiplication being licensed here by exactly the independence the sampler was built to have. A dozen occurrences of the same defect still leave close to nine chances in ten of there being nothing to commit, and you do not get to choose which occurrence is the one that survives. Raising the rate improves the gamble and leaves it a gamble. Sampling at one hundred percent on side-effecting requests, and copying flagged traces to durable storage, remove the flip altogether by making retention follow from a property the request already has.

The other prerequisite is that replay be deterministic, which requires the record to contain everything the checks read. If an assertion consults the current date to decide whether an ETA is in the future, the fixture will start failing in three weeks for reasons unrelated to the model, so the case's notion of "now" belongs in case.yaml and the check should take it as a parameter. The same applies to retrieved documents, exchange rates, and any policy table the assertion looks up: freeze them alongside the record, or the fixture is testing your environment.

Every incident leaves one behind#

The economics of the Permanent state are what make this worth institutionalising. A replay suite of 89 fixtures runs in 4.2 seconds on a laptop, makes zero API calls and costs nothing, against a live suite that takes eleven minutes and $10.80 per run. It therefore belongs in the pre-commit hook — the script git runs on your machine before it records a commit — rather than in the nightly job, and it gives an engineer editing the checking module immediate feedback on whether they have broken a check that somebody's outage paid for.

Dividing those figures turns the preference into a property. 4.2 seconds across 89 fixtures is under fifty milliseconds each, which is about what it costs to read a JSON file and run a few dozen predicates over the object it deserialises into. There is no network anywhere in that number, which is why it does not drift with the time of day the way the measurements in chapter 6 do. Set against eleven minutes, the replay suite is more than a hundred times faster and its marginal cost per run is zero — and fast and free are the two properties a pre-commit hook has to have, because an engineer disables a hook that costs a minute of their attention and a finance team asks questions about one that bills per commit. The live suite has neither property, which is the argument for leaving it where it is rather than an argument against it.

Assertions with no incident behind them#

Most checks in a new suite have never had an outage to inherit a fixture from, and waiting for one is a poor plan. The generalisation is to manufacture the failing record by mutating a good one, which is cheap because a run record is data. Take a passing case, apply a single targeted corruption, and require that a named check rejects the result.

Ten mutations, each one line, and between them they establish that every class of check written across this topic can actually reject something. The fourth row is the one that would have caught the vacuous loop on the day it was written, and it costs a single record.tool_calls = [].

This is mutation testing with the mutants applied to fixtures rather than to source, and it inherits the same caveat: passing all ten proves the checks reject these ten corruptions, not that they reject every corruption. It is a floor on the evidence, deliberately set low enough that a team will actually maintain it.

Frozen bytes, and a moving model#

A frozen record never changes, which is the source of both its value and its limit. It locks the checker, not the model. When a provider ships a new version and the agent starts phrasing refunds differently, the fixture keeps passing, because the bytes on disk are the bytes from June. Nothing in a replay suite will ever discover a new failure mode; it can only confirm that known failure modes remain caught. The live cases are the instrument that watches the model, and the fixtures are the instrument that watches the instrument.

Fixtures also go stale in a way that is silent unless you plan for it. A record captured against a tool signature that no longer exists is asserting about a world that has gone, and the check it exercises may have been deleted while the fixture continues to pass for uninteresting reasons. The startup contract validation of chapter 4, which compares every stored expectation against the live tool definitions, covers this if fixtures are run through it as well as live cases: an expectation naming a parameter the tool no longer has fails loudly, which forces a decision to update the fixture or retire it with a note. Retirement is a legitimate outcome and should be recorded rather than done by deletion, since "we stopped checking this in September" is exactly the fact a future postmortem wants. That is the governance half of the practice: the fixture directory is a register of which historical failures the suite still claims to catch, and a register is only worth reading if entries leave it by a recorded decision rather than by a git rm in a tidying commit.

Why that decision has to be forced from outside the replay loop is worth spelling out, because the mechanism is the same one that makes staleness silent. A fixture's verdict is a function of three inputs: the frozen record, the checking code, and the expectations committed beside it. When the checking code moves, the fixture reports it immediately — an assertion that stops rejecting the June record turns that fixture red, and that is the fixture's whole job. When the world moves, nothing inside the loop can notice, because the record is frozen by design and the expectations were written to match the record rather than the world. The suite goes on demonstrating that a check rejects a run the system is no longer able to produce, and it demonstrates it in green. Staleness is the absence of a signal rather than a signal, and no closed loop detects the absence of an input it never reads, which is why the instrument that catches it has to be the one that reads the live tool definitions.

Three neighbouring failures are caught, and knowing which they are keeps the gap drawn in the right place. Delete the assertion a fixture names and the bad record is accepted, so the fixture fails. Weaken it until it no longer keys on the amount and the same thing happens. Rename it and the result.failed_check == expect["by"] comparison fails on the name. What passes all three and still means nothing is a fixture whose subject matter has been retired from the product.

The subtler risk is what a large green replay suite does to a team's confidence. Eighty-nine fixtures all passing says that eighty-nine specific historical failures would still be caught, and it is easy to read it as saying the system is well tested. Regression fixtures are, by construction, a catalogue of things that have already gone wrong once, so the coverage they provide is exactly as good as your incident history and no better. They are also biased towards failures that were noticed — the ones that produced a ticket — which systematically under-represents the quiet kind, the answer that was subtly wrong for a customer who never complained.

The distinction the practice buys is between a check that guards behaviour and a check that executes, and no amount of green tells you which you have. A fixture answers it directly, once, and then keeps answering it on every commit for the cost of reading a JSON file. Where an outage supplied the failing run, the fixture is free evidence recovered from something you already paid for; where none exists, a one-line mutation of a passing record buys the same evidence at the cost of a line. Both produce the same artefact, and the ordering — red first, for a named reason, then the fix — is what stops a check from being written to fit the run it was written beside.

The last thing worth saying about a check with proven teeth is that it costs nothing to run, and that fact is about to become the organising principle for the whole pipeline. Every assertion in this topic — parse, schema, required fields, forbidden strings, tool identity, tool arguments, token budgets, completion length — executes in microseconds against a record you already have. The graders that come next do not.