Refreshing a golden dataset

An eval set is an instrument, and instruments drift out of calibration. The difference is that a drifting set gives no sign of it — the number it produces stays inside the same band, the gate keeps passing, and the release notes keep improving. A stable eval score is usually read as evidence of a mature system, and it is at least as often evidence that the instrument stopped tracking the thing.

This chapter is about keeping it tracking. It names the four mechanisms by which a set falls behind, which are distinct and fail differently: coverage that never extends to new behaviour, population weights measured on a traffic mix that has since moved, ground truth written against a policy that has since changed, and saturation, where every case that could ever catch a bug already caught it. It then sets out the triggers and detectors that fire without anyone remembering to look, and the refresh procedure itself — which is additive, versioned and re-baselined, because the obvious alternative of drawing a fresh sample destroys every number the team has. The chapter ends on ownership, because none of the above happens without a name against it.

Four mechanisms, all of them quiet#

Coverage drift is the one people name first. New behaviour ships, traffic flows to it, and the set has zero cases there, so the new behaviour contributes zero to the score whether it works or not. A subscription tier launched in May and carrying 15% of conversations by June contributes 0% of a set built in March, which means the score is computed over the 85% of traffic that existed in March, weighted as though that 85% were everything.

Weight drift is quieter and biases the aggregate rather than omitting a slice. The population weights that turn per-slice rates into one number came from the traffic mix of a specific three-week window. When subscription questions take 15% of traffic, they take it from somewhere — order-status volume falling from 31% to 24% is a realistic shape — and every weighted number reported after that shift is reweighting the same per-slice rates with stale multipliers. The error is small when the mix moves a little and it compounds silently, because nothing in the pipeline knows the weights have an expiry date.

What that error is made of is easiest to see with two slices rather than thirteen. Take a set with an easy slice the assistant nearly always passes and a hard slice it passes about half the time, and let traffic move so that the hard slice takes some of the share the easy one used to have. The true population rate falls, because more of the traffic now sits in the half-passing slice. The reported number does not move at all, because it multiplies the same two per-slice rates by the same two stale shares. Nothing has gone wrong with the per-slice rates; they are still exactly right. The aggregate assembled from them is wrong by the size of the shift multiplied by the distance between the two slice rates, and that product is the whole error term. It is small when the shift is small, it grows in direct proportion to the shift, and no part of the pipeline computes it or could notice it growing.

Ground-truth decay is the mechanism that actively causes harm rather than merely omitting information. When a returns policy changes from 30 days to 14 for sale items, every case carrying an expected answer written against the 30-day rule — along with a frozen copy of pol_returns_v7 in its retrieval fixtures — now passes when the assistant states the old policy and fails when it states the new one. The set is applying downward pressure on exactly the behaviour the company needs, and it will keep doing that until someone reads the fixtures.

Calling that downward pressure rather than a wrong number is deliberate, and the step in between is what makes this the mechanism that does damage. A gate is not a passive reading. It is an optimisation target with a team attached to it, so a change that lowers the score gets investigated and usually gets reverted. A case whose expected answer still says thirty days therefore does more than score the corrected assistant incorrectly. It produces a red gate on the very release that fixed the policy string, an engineer hunting for a regression that does not exist, and a plausible-looking fix that restores the old behaviour to production. Coverage drift leaves a set blind. Ground-truth decay leaves it hostile.

Saturation is the fourth and it explains the flat band. An audit of one 120-case set found 31 cases that had never changed verdict across 90 releases, and that fraction only grows, because every case that catches a bug gets the bug fixed and then joins the population of cases that always pass. After a year, a set that is not being added to consists mostly of settled questions, and the handful of cases that can still move are not enough to shift an aggregate by more than a couple of points. A set that has scored between 0.92 and 0.95 on every run for a year, while support tickets tripled, is what saturation looks like from the outside.

Why that fraction only ever grows is one causal step and it makes saturation predictable rather than surprising. A case earns its keep by catching a bug. Catching the bug is what causes the bug to be fixed. After the fix, the case passes — and it goes on passing on every subsequent release, because the behaviour it tests is now behaviour somebody deliberately implemented and defends. So every case that does its job moves permanently into the population of cases that cannot do it again, and no case moves back the other way unless somebody adds one. A set with no additions has a sensitivity that only decays, and its score rises as it decays, which is why the symptom is a flat high band and a set quietly disconnected from the product rather than an obvious failure.

Triggers, because a cadence alone is not enough#

A quarterly review is a floor and it is not sufficient on its own, because the events that invalidate a set do not arrive on a quarterly rhythm. The arrangement that works pairs a slow cadence with fast triggers, each attached to a detector that fires without anyone remembering to look.

refresh_triggers:
  - name: product_launch
    detect: release note tagged assistant-behaviour
    action: add a slice with >= 20 cases
    sla: 2 weeks after launch

  - name: traffic_shift
    detect: monthly job — any intent whose share moved > 5pp,
            or any new intent above 2% of conversations
    action: re-measure weights; add cases if the slice has < 8
    sla: 2 weeks

  - name: policy_change
    detect: version bump on any document referenced by a fixture
    action: re-derive affected fixtures; re-check ground truth on those cases
    sla: 1 week

  - name: incident
    detect: any Sev-1 or Sev-2 involving assistant output
    action: add >= 10 cases of the incident pattern, oversampled
    sla: 5 working days after postmortem

  - name: cadence_floor
    detect: 90 days since last refresh
    action: review coverage, retire stale cases
    sla: —

The traffic-shift and policy-change detectors are the ones worth building properly, because both are cheap and both catch things nobody would otherwise notice. The traffic detector is a monthly comparison of the live intent distribution against the set's slice shares, and it emits a work item rather than a chart:

def coverage_gaps(live_shares, slice_cases, slice_weights):
    for intent, share in live_shares.items():
        cases = slice_cases.get(intent, 0)
        if share >= 0.02 and cases < 8:
            yield f"{intent}: {share:.1%} of traffic, {cases} cases — add cases"
        old = slice_weights.get(intent, 0.0)
        if old and abs(share - old) > 0.05:
            yield f"{intent}: weight {old:.1%} -> {share:.1%} — re-weight"

# June run:
# subscription_billing: 15.0% of traffic, 0 cases — add cases
# order_status_id: weight 31.0% -> 24.0% — re-weight

The policy detector is a join between the fixture table's doc_version fields and the document store's current versions, and it surfaces affected cases the week a policy changes rather than the following January.

Rebuilding is the expensive answer#

The instinct when a set is badly behind is to delete it and draw 220 fresh cases from last month's traffic. The coverage problem does go away, and so does every number the team has. A score is only interpretable relative to other scores on the same instrument, so a wholesale rebuild makes every historical figure incomparable, and the first question anyone asks after the rebuild — is this release better or worse than the last one — becomes unanswerable until enough releases have accumulated on the new set to form a trend, which is months.

The reason is the hold-one-thing-still argument the fixtures chapter turned on, and it is worth writing out rather than asserting. A score is a pass rate over one particular collection of cases, and two collections drawn from the same traffic still differ in difficulty by a few points, purely through which conversations happened to be drawn. So the difference between a score on one set and a score on another mixes two quantities: how much the system changed, and how much the two sets differ. Nothing in either number separates them, and no amount of care in drawing the second sample removes the problem, because a difficulty difference of a few points is exactly the size of the release movements anyone cares about.

Metrology, the discipline concerned with measurement itself and with keeping instruments comparable over time, solved this a long time ago, and the practice is worth borrowing directly. A calibration lab with a set of reference weights does not discard them when the standard is updated; it sends them for re-certification, records the offset between the old certificate and the new one, and keeps every prior measurement usable by applying that offset. The offset is the artefact that preserves the history, and producing it costs far less than starting over.

Borrowed as an analogy that has one limit worth naming before it does damage. A lab's reference weights do not stop being relevant: a kilogram is still a kilogram after the product line changes. An eval case can stop being relevant completely, because the behaviour it tests may have been deleted from the product. Re-certification alone is therefore not enough here, and a refresh has to do two things a calibration lab never does — add cases for behaviour that did not exist before, and retire cases for behaviour that no longer does. The part of the comparison that carries over is the offset, and that part carries over exactly.

The equivalent here is an additive refresh with a re-baseline. Existing cases stay where they are, new strata are added alongside them, the set gets a version, and — this is the step that does the work — the previously released system is re-run against the new version so that both the old and new sets have a score for the same system.

That re-run is what separates the two confounded quantities, and it does it by holding the system still instead of the set. One fixed release measured against both versions gives two numbers whose only difference is the version, so the difference between them is the ruler change with the system's contribution removed by construction. Subtract that from the raw movement in the gate and what is left is the system's contribution. It costs one extra run of a set you already have, which is minutes, and it is the difference between a version table that can be read and a column of numbers that cannot.

Every score in that table carries the version it was measured on, and every row where the version changed carries a re-run of the prior release, so the honest comparisons are within a row rather than down a column.

The v5.1 row is the informative one and it repays being read slowly, because every number in it is doing a different job. That refresh landed in July, when the returns policy went from thirty days to fourteen for sale items and eighteen cases had to be relabelled against the new policy. The current release scored 0.66 where the previous release had scored 0.69 on the older version of the set, which read down the column looks like a three-point regression. The re-baseline column says the previous release, re-run against v5.1, would have scored 0.64. So the same unchanged system lost five points purely to the relabelling, and that five points is the ruler change measured directly rather than estimated. On the new ruler the comparison is 0.66 against 0.64 and the release gained two points. Both readings are true of the same two runs; only one of them is about the assistant. Without the re-run the release looks like a regression, and somebody spends a day investigating a change that happened in a labelling guide.

Retirement is the part teams skip and it matters as much as addition. The v6.0 row in September is the one where it happened. Twelve cases testing a checkout flow removed from the product in August were passing, contributing to the denominator, and describing nothing. Cases are retired rather than deleted — moved to an archive table with the reason and the date — because a retired case is still evidence about what the system used to do, and because the archive is what lets you reconstruct why an old score was what it was.

Someone's name, on a schedule#

None of this survives without an owner, in the same sense that a production service has an on-call rather than a hope. The set needs one named person per quarter who receives the trigger alerts, runs the refresh, publishes the version table, and is asked about coverage in release review. The measurable version of ownership is a number in the version table's date column: a set whose last refresh is six weeks after the most recent trigger has an owner in name only, and the gap between trigger date and refresh date is the honest health metric for the whole practice.

A version change also has to be announced, because the gate is not the only consumer of these numbers. Thresholds configured against v5.0 do not automatically mean the same thing on v5.1, so a refresh that adds a hard slice will start failing releases that would have passed, and the first team to hit that will reasonably assume something broke. The convention that avoids the argument is to publish both numbers for one release cycle — the score on the retiring version and the score on the new one, side by side — along with the re-baselined figure for the previous release, and to move the per-slice thresholds in the same change rather than a week later. Slice thresholds are where most of the gate's value sits anyway: a data-exposure slice carries an absolute floor a release cannot cross, while the aggregate carries a relative rule about how far it may fall from the previous release, and the two are set independently because they protect against different things.

The direction this is moving is toward closing the loop automatically. Teams with mature pipelines mine candidate cases continuously — conversations with a thumbs-down, an escalation, a retry, an abandoned session — cluster them, and put the cluster representatives into a review queue where a human decides whether each becomes a case. That reduces the discovery problem stratified sampling leaves open, since the clusters surface failure modes nobody enumerated, and it does not remove the human step, because the labelling rules and the retirement decisions are judgement calls the pipeline cannot make.

What none of it removes is the underlying tension: a set that changes often tracks the product and has a short history, and a set that changes rarely has a long history and tracks something that no longer exists. Versioning and re-baselining make that trade manageable rather than making it disappear. The refresh is additive so the old cases keep their meaning, the version is stamped so two numbers are never compared across a ruler change, and the re-run of the previous release is what converts the ruler change from a mystery into a figure — two points here, five points there — that a release review can read. About once a year you will find that so much has been added and retired that the set shares few cases with its original self, which is the point to say so plainly in the README rather than to keep quoting a two-year trend line.

The set is now a maintained artefact: sampled from real traffic, stratified around the failure modes that matter, frozen so it replays deterministically, sanitised so it can be stored, labelled by rules that are written down, and versioned so its numbers mean something over time. What it does not yet do is produce a verdict without a human reading the output, and 179 cases read by hand on every pull request is not a gate anyone will run twice. Turning these labels into an automatic scorer — and knowing how far to trust it, given the 0.64 kappa the labels themselves carry — is the next thing this set is for.