Passing the sixty cases you tuned on is not evidence

The pull request is titled doc_qa@23: fix effective-date and citation failures, and it is green. Sixty cases were pulled out of production over three weeks, given expected outcomes, and added to the suite; the prompt was edited eleven times over four days until all sixty passed. The diff shows what those four days produced. Seven of the eleven new instructions name something from the sixty directly — when the document is a Schedule B addendum, the effective date is the addendum's own date and not the master agreement's; when a clause is quoted, include the section number; if the document contains a table of amendments, read the last row — and the prompt has grown from 1,100 tokens to 2,900. Every one of those instructions was written by a person looking at a case that was failing, and each one made it pass.

That process has a name in any other part of engineering, which is fitting to the test set. The sixty cases stopped being a measurement the moment someone started editing the prompt in response to seeing them fail, because from that point on the prompt was being optimised against them, and the score they produce is a training score. A model that scores 60/60 on the set it was tuned against tells you the tuning worked, and it says nothing about the 300,000 responses a week that were not in the set.

Three artifacts, and the different question each one answers#

The discipline that makes shipping decisions possible is to hold three separate things and never let them merge. They are separated by who is allowed to touch them and when, not by what is in them.

The frozen set is the part teams skip, and its rules are what give it value. It is drawn before the cycle begins, nobody looks at individual failures inside it while iterating, and it is scored exactly once, at the end, against the same rubric version and the same judge version — the scorer, usually a model, that decides whether each case passed — used at the start. If it drops, the change is not shippable regardless of what the sixty say, because the sixty measure the intended effect and the frozen set measures the unintended one. A prompt that gained 1,800 tokens of case-specific instruction has a large surface for unintended effects: those instructions apply to every request, including the 96% that have nothing to do with addenda.

The live metric closes the third gap, which neither offline artifact can. Chapters 2 and 3 established which behavioural signals this product has and what they are worth; the release gate uses them in the narrow way they support, as a post-release comparison on the same traffic mix rather than as an absolute level. On the contracts assistant that is the edit-distance residual above the 0.12 floor, sliced to the clusters the change targeted and to everything else separately, over the two weeks after release.

The frozen set cannot stay frozen forever, because production drifts and a set drawn eighteen months ago stops representing the traffic. The refresh that preserves comparability is an overlapping one: each quarter, retire the oldest quarter's cases and draw a replacement cohort, so the set is always four cohorts deep and any two consecutive quarters share three of them. A change in the overall pass rate can then be decomposed into the part that comes from the shared cohorts, which is a real change in behaviour, and the part that comes from the new cohort, which is a change in what is being asked. Refreshing the whole set at once destroys that decomposition and produces a step in the pass rate that will be attributed to whichever release happened nearby.

The gate, written down#

A gate that lives in someone's head gets skipped under deadline, so it is a check that fails the build and a record that survives the release.

# release/doc_qa@23.yaml — written by CI, stored with the deploy
change: "doc_qa@23"
working_set:
  dataset: "cycle-2026-03"
  version: "v7"
  pass: 60
  total: 60
regression_set:
  dataset: "frozen-2026-Q1"
  version: "v3"              # unchanged since 2026-01-06
  rubric_version: "contracts-quality@4"
  judge_version: "judge@2026-01-02"
  pass_before: 351           # of 400, measured on doc_qa@22
  pass_after: 349            # of 400, measured on doc_qa@23
  delta: -2
  gate: FAIL                 # tolerance is -1
rollback:
  previous_template: "doc_qa@22"
  previous_dataset_hashes: "sha256:9c41…"
  restore_command: "deploy --template doc_qa@22 --index contracts@2026-03-11"

TOLERANCE = -1     # one case of noise; anything worse is a stop

def gate(before: dict, after: dict) -> None:
    if after["rubric_version"] != before["rubric_version"]:
        raise SystemExit("regression set graded with a different rubric; not comparable")
    if after["dataset_version"] != before["dataset_version"]:
        raise SystemExit("regression set changed during the cycle; not comparable")
    delta = after["pass"] - before["pass"]
    if delta < TOLERANCE:
        newly_failing = set(before["passing_ids"]) - set(after["passing_ids"])
        raise SystemExit(f"regression: {delta} cases, first failures {sorted(newly_failing)[:5]}")

The two version equality checks matter more than the arithmetic. The most common way a regression gate produces a false green is that somebody refreshed the frozen set or bumped the rubric mid-cycle, so pass_before and pass_after were produced by different instruments and their difference means nothing. Making that a hard failure rather than a warning is what keeps the comparison real over the eighteen months during which everything else in the system changes.

The rollback block is there because the useful moment for it is three weeks later, when the live metric has drifted and nobody can remember which index version was current at release. Preserving the previous template and the dataset hashes turns a rollback from an archaeology exercise into one command.

The six-month review#

Now the harder case. Over two quarters the suite went from 200 cases to 1,800 and its pass rate from 71% to 94%. Production thumbs-down rate is unchanged. Support ticket volume is unchanged at about 4,200 a month, within ±3% month to month. The team presents the pass-rate curve as the flywheel working and asks for three more engineers to expand the capture pipeline.

Both sets of numbers can be true at once, and there are at least six mechanisms that produce exactly this pattern.

The measurement that would settle it is a fixed, randomly drawn set of production responses, labelled by humans against a pinned rubric at both dates — the quarterly sample from chapter 1, run at month zero and month six. That comparison is immune to every mechanism in the table, because the sample was not chosen by the pipeline, not edited, not re-graded by a different judge, and reflects real traffic frequencies. It was not available here because nobody drew it at month zero, and it cannot be reconstructed retrospectively: the traffic exists in the trace store, but labelling six-month-old responses against today's rubric measures the rubric change as much as the system change, and the cheaper proxy of re-running old requests through the current system produces answers to questions nobody asked in their original context.

The support ticket number is the part that should not be waved away. Thumbs-down is sparse enough to be plausibly insensitive, and chapter 3's arithmetic showed how much movement it takes to clear the noise band. Tickets are neither sparse nor subtle: 4,200 a month, with a tagging pass showing roughly 15% — about 630 — attributable to something the assistant got wrong. If defect rate had genuinely fallen the way the suite implies, from 29% failing to 6% failing, a substantial share of those 630 should have disappeared, and even half of it would be a 7-8% drop in total ticket volume, well outside the ±3% band. The flat line is not the absence of evidence. It is evidence, and the honest reading is that whatever improved was concentrated in behaviours that were rare in production, which is precisely what a pipeline fed by complaint volume from a few heavy tenants produces.

Which is the answer on headcount. Three engineers added to capture will produce more cases at the same rate of unmeasured effect, and the constraint is not case supply. What is missing is the instrument: a frozen production sample drawn now and re-labelled quarterly, case-level version pinning so that a pass rate has a comparable predecessor, production frequency weights so the suite's headline number reflects the traffic rather than the queue, and the release gate above wired to the frozen set. That is roughly one engineer for a quarter. Fund it first, and revisit the expansion when a pass-rate movement can be shown to correspond to something a user experienced.

What the suite is for once you stop asking it the wrong question#

None of this makes 1,800 cases worthless, and the failure mode of a team that has just absorbed this argument is to conclude that offline evaluation does not work. A regression suite is a detector: it catches the reintroduction of failures you have already seen and paid to fix, which is a real and continuous benefit in a system where a prompt edit can revive a bug from March. It is not an estimator of quality, and the numbers it produces do not convert into a statement about the user population, because the cases were never sampled in a way that would support that conversion. Two artifacts, two jobs, and most of the confusion in that review came from asking the detector to do the estimator's work.

The operating shape that falls out of these six chapters is a weekly cadence rather than a project. Candidates arrive from the signals — ratings for their pointers and comments, edits for their localisation, retries only after the residual has been taken — and get compressed by the caps and the dedup rules into a handful of cases with agreed expected outcomes, stored where a deletion request can reach them. Work happens against those cases, ships behind the frozen-set gate, and gets checked against a live metric a fortnight later. Once a quarter someone draws a few hundred random responses and labels them, which is the only number in the whole system that can be compared to itself a year from now. Start with that quarterly sample, because every other part of the loop is calibrated against it, and a loop with no fixed point is a loop that will report progress for as long as anyone is willing to fund it.