Retire synthetic cases class by class, not all at once

Eight months after launch the eval suite had two halves that nobody had reconciled. The synthetic half was 800 cases built during cold start across twelve classes, and it reported 91%. The real half was 3,000 user questions with adjudicated labels, accumulated at roughly 400 a month by two support engineers doing a Friday afternoon pass, and it reported 74%. Both ran in CI. The release dashboard showed the average of the two, which was 82.5% and was a number describing nothing at all.

The 74 is the number to report. It is a measurement of the system's behaviour on the questions users actually ask, and every argument in this topic points at it: it can fail for reasons nobody anticipated, its labels came from people reading the answer against the docs, and it is not correlated with the system's blind spots by construction. The 91 is a measurement of behaviour on a set of situations somebody enumerated in month zero, and it is not the product's score. Averaging them produces a quantity with no referent, which is worse than either input, because it moves when either half moves and means something different depending on which one did.

Deleting the 800 is the move that follows naturally and should not be made.

What 3,000 real cases do not contain#

The synthetic classes exist because certain situations are rare in traffic, and rarity is exactly the property that survives eight months of accumulation. Here is the coverage audit that decided what went.

That table is the whole argument in one place. Two hundred and seventy cases were retired because real traffic now covers those situations better than the generator ever did, and the multi_intent cases were retired with some relief, since chapter 2 marked them low-fidelity from the day they were written and 143 real multi-intent messages are strictly better material. Five hundred and thirty stayed, and the reason they stayed is legible in the middle column: three real contradictions in 3,000 questions, one stale-fact case, zero for freshness and permissions. The last two cannot be sampled at all — a freshness case requires two corpus states and a clock, and a permission case requires asking as a user whose data you are not allowed to put in an eval store. Deleting those wholesale converts a documented blind spot into an unmeasured one, and nothing in the dashboard announces the loss.

The retirement is also not a delete. Each retired class gets a record naming what replaced it, which slice of the real set now carries that coverage, and how large that slice is, so a traffic shift that empties the slice can be noticed. When the product added a self-serve tier in month ten, identifier_missing in real traffic dropped from 208 to 34 within a quarter, because the new tier authenticated users automatically, and the record was what made the resurrection of those 40 cases a five-minute decision rather than a rediscovery.

Check for contradictions before anything else#

The first pass over the 800 was not retirement, it was a correctness audit against what had since been learned, and it is the step people skip. A synthetic case carries an expected answer written by a generator in month zero, and eight months of product change and adjudicated real cases will have proved some of those expectations wrong.

Twenty-two of the 800 asserted that audit logs are retained for 30 days — the case from chapter 1, where the generator dropped the Enterprise conditional. Those were rewritten with the conditional rather than deleted, because the class is still needed and the fix is a one-line edit to the reference answer. Nine cases referenced documentation pages removed in a migration, and their retrieval assertions could never pass; those were deleted. Four cases had expected answers that the support team now considered wrong on policy grounds, which had nothing to do with the generator and everything to do with the policy having changed.

Running an audit like that is cheap when the manifest from chapter 4 exists, because the corpus commit recorded there gives you a diff. Comparing e91b774 against the current docs produced 340 changed paragraphs, and mapping cases to source chunks narrowed the manual review to 51 cases out of 800.

Regenerating from real patterns is the wrong repair#

The proposal that comes up whenever the two numbers differ is to regenerate the synthetic set from real question patterns, so that the synthetic score converges toward the real one. It sounds like a fidelity improvement and it is a coverage loss.

Fitting a generator to the distribution of observed questions puts synthetic mass where real mass already is. The classes that survived the audit survived because real mass there is near zero, so a generator trained or prompted to match the real distribution produces almost none of them. What you get is 800 cases that are an inferior copy of the 3,000 you already have, the two numbers converge, and the convergence is caused by the synthetic set no longer measuring anything the real set does not. Convergence was never the goal. A synthetic set is valuable to the extent that it covers situations the real set cannot reach, which means a well-functioning pair of sets reports two different numbers, and the difference is informative rather than embarrassing.

There are two uses of real questions that are sound and worth separating from this one. Real questions make excellent seeds for controlled perturbation, which topic 3.4 established: take a real case the system handles, vary one property, and see what breaks. That inherits real distribution at the level of the individual case without fitting the generator to the aggregate. Real questions also feed the class list, in the manner chapter 3's discriminator sorting showed — read the real messages least like anything synthetic, and turn the recurring shapes into new classes. Both take from real data and neither reshapes the synthetic set toward the average.

Decomposing the 17 points, and what 300 real cases can resolve#

The gap between 91 and 74 is a quantity to explain rather than to close. Restricting both sets to the classes present in each gave synthetic 91% against real 79% on comparable classes, which puts roughly 12 of the 17 points down to case mix — real traffic contains situations the synthetic set does not have at all — and the remaining 5 down to difficulty within shared classes, mostly surface form. That second part is checkable directly: perturbing 100 synthetic single_hop cases with real typo patterns and stripped identifiers dropped their score from 93% to 84%, which accounts for most of it. A decomposed gap has owners; a single number labelled "the synthetic set is optimistic" does not.

The real set stays the gate even when it is small, and small is the ordinary case early on. At 300 real cases and a 74% pass rate, the 95% Wilson interval runs from about 68.8% to 78.7%.

from statsmodels.stats.proportion import proportion_confint
for n in (300, 900, 3000):
    lo, hi = proportion_confint(round(0.74 * n), n, alpha=0.05, method="wilson")
    print(f"n={n:>4}  74.0%  [{lo:.1%}, {hi:.1%}]  width {100*(hi-lo):.1f}pp")
# n= 300  74.0%  [68.8%, 78.7%]  width  9.9pp
# n= 900  74.0%  [71.0%, 76.7%]  width  5.7pp
# n=3000  74.0%  [72.4%, 75.5%]  width  3.1pp

Reading that as "300 cases cannot detect anything under 10 points" is too pessimistic, because release decisions compare two runs over the same cases rather than two independent samples. Run the same 300 before and after a change and only the cases that flipped matter: 18 cases moving from fail to pass against 5 moving the other way is a real improvement at n=300, and a paired test says so at a difference the unpaired interval would call noise. What 300 genuinely cannot do is support per-class claims, since a class with 22 real instances has an interval wide enough to swallow any plausible change, and it cannot support fine-grained comparisons between two candidate models that differ by two points.

Somebody has to own this review, and the failure mode is that nobody does. The audit above happened in month eight because a new engineer asked what the two numbers on the dashboard were, which is not a process. What worked afterwards was attaching the coverage audit to an existing cadence rather than creating one: the same quarterly session that reviews the class list runs the real-instance counts per class, and the counts come out of a query over the labelled set rather than out of anyone's memory. It takes about ninety minutes a quarter. The labelling itself is the standing cost — two engineers, one Friday afternoon a month, roughly 400 cases at four to five minutes each including adjudication of disagreements — and it is the line item most likely to be cut when a deadline arrives, which is worth defending explicitly, since every synthetic case that stays in the suite is there because that budget did not stretch to covering its situation with real data.

The second job the synthetic set has earned#

Kept cases do work that the gate does not. Running 530 synthetic cases takes about four minutes and two dollars of tokens, which means they can run on every commit, while the real gate involves human adjudication of new cases and runs against release candidates. The synthetic suite in that role is a screening signal: it is looking for large, structural breakages, per class, against its own recent baseline.

The rules that keep it honest are narrow. Its absolute level is never reported as the system's score and never appears on the release dashboard next to the real number. Only per-class deltas against a rolling 7-day baseline trigger anything, with the threshold set at 10 points, which fires roughly once a month and has caught real things — a reranker change that dropped contradiction from 78% to 41% while the overall synthetic score moved by 3 points and the real gate, running four days later, would have caught it after the release branch was cut. And no change ships on a green synthetic run alone, which is chapter 1's rule surviving into month eight in its final form: the generated set can tell you something broke, and only the real set can tell you the system works.

Applying this to a suite that already exists starts with a sorting exercise rather than a rewrite. Take the cases you have, label each one generated or sampled, assign each to a class, and count how many real adjudicated instances of each class you hold. That table is usually uncomfortable — most teams find two or three classes with hundreds of synthetic cases and thousands of real ones, and one or two with a dozen synthetic cases holding the entire coverage of a situation that would be a serious incident. The first has cases to retire and the second has a manifest to write, an absence to declare, and a real holdout to go and build.