A finding becomes a regression case, not a memorised example

The test written on the afternoon of 5 March, while the mitigation was still deploying, was seven lines long.

def test_calderwood_parental_leave():
    out = ask("t_calderwood", "how much paid parental leave do I get")
    assert out.text == (
        "You are entitled to 16 weeks of paid parental leave under the "
        "2025 benefits handbook."
    )

It passed that day. It failed on 12 March, when a prompt change added a leading acknowledgement to answers about leave, and the diff showed the assistant now saying "Under your 2025 benefits handbook, you're entitled to 16 weeks of paid parental leave." The behaviour was correct, the assertion was not, and somebody updated the expected string. It failed again on 2 April for a similar reason, and the third time it was marked @pytest.mark.skip with the comment flaky, wording changes. By June the assistant could have served the 2024 figure to Calderwood again and nothing in the suite would have objected, because the only test guarding that failure had been switched off for being annoying.

What that test was trying to be is a regression case: a test kept in the suite for no reason other than to fail if a defect that has already been diagnosed and fixed comes back, long after everyone involved has forgotten the incident that produced it. That sequence is the default outcome of pinning a literal output, and it is worth understanding why the pin is tempting rather than dismissing it. In ordinary backend testing, equality against an expected value is the correct assertion almost all of the time, since a function that formats a currency amount has exactly one right answer and any change to its output is a change you meant to make. A generated answer has an enormous space of correct forms, no stable serialisation, and a wording that legitimately changes whenever the prompt does. Equality against the string encodes hundreds of properties you do not care about along with the one you do.

Asserting what failed#

The failure in chapter 2 was not that the assistant produced a particular sentence. It was that the context assembled for the request contained chunks from a superseded document, and the answer therefore reported a figure that contradicted the version of the handbook in force on that date. Both halves of that are checkable without knowing what the sentence looks like.

CURRENT_VERSIONS = {"t_calderwood": {"benefits-handbook": "2025.1"}}

def test_no_superseded_chunks_in_context(case):
    """The retrieval half: nothing superseded reaches the prompt."""
    chunks = retrieve(case.tenant, case.query, k=4)
    stale = [
        c for c in chunks
        if c.doc_version != CURRENT_VERSIONS[case.tenant].get(c.doc_family, c.doc_version)
    ]
    assert not stale, (
        f"{len(stale)} superseded chunks retrieved for {case.query!r}: "
        f"{[(c.doc, c.doc_version, round(c.score, 3)) for c in stale]}"
    )

def test_answer_reflects_current_policy(case):
    """The answer half: the figure and the citation come from the live version."""
    out = ask(case.tenant, case.query, seed=7)
    assert extract_weeks(out.text) == case.expected_weeks
    for doc in out.cited_docs:
        assert doc.version == CURRENT_VERSIONS[case.tenant][doc.family]

The first of these needs no model call at all, which makes it fast, deterministic and cheap enough to run on every pull request; on this system the retrieval half of the suite runs in 4.1 seconds for 34 cases. The second calls the model and costs about $0.004 per case at current pricing, so 34 cases is roughly 14 cents a run and takes 82 seconds against a warm endpoint, which is a nightly job rather than a pull-request gate. Splitting them along that line matters more than it seems, because the retrieval assertion is the one that would have caught this incident on 13 February, and a check that is cheap enough to run continuously is worth more than a better check that runs weekly.

extract_weeks is a small parser over the answer text — it looks for a number followed by a duration unit within the sentence that mentions leave, and returns it normalised to weeks — and writing it takes about fifteen minutes. That is the actual work of property assertions: turning "the answer was wrong about the figure" into a function that can read the figure out of arbitrary prose. It is unglamorous and it is what makes the test survive rewording.

Before either assertion can run, the case needs the condition that produced the failure, and this is where incident-derived tests most often die quietly. The query alone reproduces nothing once the index has been cleaned up, so the case has to carry a corpus fixture: a small tenant with two versions of a handbook, one marked 2024.2 and one 2025.1, the older one written with a null version field to reproduce the exact defect class, and both indexed by the same embedding model the test asserts against. Building that fixture took about two hours and it is reusable — every subsequent staleness finding drops into the same tenant with another document pair. Teams that skip it end up with tests that point at the production index, which means the test passes because someone deleted some chunks in March rather than because the pipeline is correct, and it starts failing for unrelated reasons the first time a real customer document changes.

Sampling is the other thing that has to be handled before the case is trustworthy. With temperature at 0.3 the answer half is not deterministic, so a single call can pass while the underlying behaviour is wrong a third of the time. Fixing the seed, which is the number that determines the sequence of random draws the sampler makes and so pins one particular run out of the many available, makes the test reproducible and narrow, since it only ever exercises one path through the sampler; running the case five times and requiring all five to hold the property costs five times as much and measures something closer to what you care about. The compromise this suite settled on is a fixed seed on pull requests and n=5 with a requirement that at least four pass on the nightly run, with the failure message printing all five extracted figures so a partial failure reads as "3 of 5 returned 16, 2 returned 8" rather than as a mysterious flake.

The comparison that decides how brittle the suite will be#

The last row is the one that surprises people. A groundedness judge, of the kind that scored this traffic at 0.88 while it was failing, evaluates whether the answer follows from the context it was given, and in this incident the answer followed from its context perfectly — the context was simply the wrong document. A judge is a reasonable instrument for hallucination and a poor one for staleness, and picking it here would produce a test that passes throughout the exact failure it was written for.

The substring row is the compromise most teams reach for after being burned by exact equality, and it is worth being clear about what it buys. Asserting that "16 weeks" appears somewhere in the answer survives reformatting and takes thirty seconds to write, which is genuinely better than the string comparison it replaces. It also passes on an answer that says "the 2024 handbook allowed 8 weeks, though some employers offer 16 weeks", and it says nothing at all about which document was cited, so the mechanism from this incident can return in a form the test approves of. Substrings are a reasonable holding position on the afternoon of an incident and a poor place to leave the coverage a week later.

The third and fourth rows have a property worth naming: their expected values are derived from the source of truth rather than written down. case.expected_weeks is read out of the current handbook fixture at test time, so when Calderwood raises the entitlement to twenty weeks in 2027, the test updates itself and continues asserting the same property. A test whose expected value is a constant becomes stale in the same way the index did.

Variants, and what they are protecting against#

One case is coverage for one customer's Tuesday. The reason a suite needs variants is that any single input can be satisfied by a fix far narrower than the defect — deleting Calderwood's null-version chunks makes the original case pass while eleven other tenants stay broken, and adding a special case for the phrase "how much paid parental leave do I get" would too. Variants are chosen to break each of the narrow fixes you can imagine.

- id: FM-2025-014/origin           # the literal triggering input
  tenant: t_calderwood
  query: "how much paid parental leave do I get"
- id: FM-2025-014/paraphrase-1     # same intent, different wording
  query: "what's the parental leave policy here"
- id: FM-2025-014/paraphrase-2     # oblique, no policy vocabulary
  query: "my partner is due in June, how long can I take off paid?"
- id: FM-2025-014/other-tenant     # different corpus, same mechanism
  tenant: t_harlow
  query: "how many weeks of paid parental leave am I entitled to"
- id: FM-2025-014/other-doc-family # same mechanism, unrelated document
  tenant: t_calderwood
  query: "what is the mileage reimbursement rate"
- id: FM-2025-014/negative-control # no superseded version exists
  tenant: t_calderwood
  query: "who do I contact about a dental claim"
  expect_no_stale: true
- id: FM-2025-014/historical       # the old figure is the right answer
  tenant: t_calderwood
  query: "what was the parental leave entitlement in 2024"
  expect_cited_version: "2024.2"

Three of those deserve comment. The oblique paraphrase exists because the retrieval failure in this incident was driven by embedding similarity, and a query phrased in the vocabulary of the 2024 document is exactly the query most likely to pull it back, so a variant that avoids policy vocabulary tests a different point in that space. The other-document-family case checks that the fix was made in the ingestion pipeline rather than in a handler for parental leave. The historical case is the one that stops the suite from teaching the system something false: it asks about 2024 deliberately, and it should retrieve the superseded document and cite it as superseded, so a fix implemented as "never retrieve anything not current" fails here and gets caught before it ships.

Six or seven variants per finding is a reasonable working number. Fewer and the suite is memorising; many more and you are building an eval set rather than regression coverage, which is a different artefact with a different purpose — the sampled, stratified sets from topic 3.6 estimate how often something happens across traffic, while these cases assert that one known mechanism has not returned. Confusing the two produces either a regression suite too slow to run on every change or an eval set whose distribution has been quietly warped by twenty copies of one customer's incident.

What the case cannot do#

The regression case is not the mitigation and does not prevent anything on its own; it detects a return. In this incident the prevention was the assertion inside reindex_tenant that fails the job when a corpus change deletes nothing, and the regression case exists because that assertion could itself be removed, refactored around, or bypassed by a second ingestion path added next year. Treating the test as the fix is how teams end up with a green suite and a live defect in a code path the suite does not exercise.

Property assertions also run out where the failure is a judgement. If the complaint had been that the answer was technically correct and unhelpfully terse, there is no extract_weeks equivalent to write, and the options are a judge with its own error rate or a human reading a handful of outputs on a schedule. Some findings genuinely convert into a monitoring change and a note rather than a test, and forcing a brittle assertion onto them produces a case that fails for reasons unrelated to the finding until someone skips it, which is where this chapter started.

The narrower limit is that variants are written by the person who just spent a day inside one incident, so they cluster around the mechanism that person now understands. This suite would not have caught a stale document reaching the prompt through the summarisation path, which does not call retrieve at all. Coverage grows by incident, unevenly, and the map of what is covered is only as good as the record of which findings produced which cases — which is a problem about organising thirty findings rather than writing one, and the register that has to hold them is the subject of the next chapter.