Governing and retiring a held-out set

Governing a held-out set means writing down five things: how big it is, where the cases came from, who may run it, how often, and what happens when it has been used up. That sounds like process for its own sake until you notice what the alternative depends on. The alternative is four people remembering a rule during a launch week, with no artefact that records whether the rule held, and the failure it guards against produces no error, no diff and no alert at the moment it happens.

This chapter writes that policy and then prices it honestly, because the strongest objection to holding cases back is a good one and comes from people who have spent hours labelling.

What a burn looks like in the log#

A read log is a plain append-only file, one line per read, and it is the only artefact that records the history the set's independence consists of. The fortnight before a launch reads like this.

$ tail -n 6 evals/holdout/READS.log
2024-06-11  extract@v58  rc-1 for launch                          0.812
2024-06-12  extract@v59  rc-1 regression check after judge fix    0.809
2024-06-12  extract@v61  comparing v59 and v61, picked v61        0.824
2024-06-13  extract@v63  rc-2                                     0.821
2024-06-14  extract@v64  v63 vs v64 on table-cell headers         0.833
2024-06-17  extract@v66  final pre-launch check                   0.828

Two of those lines record a variant chosen because the held-out number came out higher, and those two are the ones that matter. A read spent on a release candidate is a read the set exists to serve; a read that decides between v59 and v61 is the set participating in selection, which is exactly the condition that made a tuned-against score untrustworthy in the first place. Counting reads against a budget catches the volume problem. Reading the reasons catches the kind, and the kind is what retires a set.

What separates the two kinds is not care, and it is not who took the read. It is whether a decision followed that the read could have gone the other way on. A release-candidate read has one outcome that changes an artefact — ship or do not ship — and when the answer is ship, the thing that ships was chosen by the tuned-against set and merely confirmed by this one. A read comparing v59 with v61 has two outcomes and each of them produces a different artefact. That second shape is what makes the shipped prompt a function of the held-out cases, and it arrives after one read rather than after twenty. A team that fails a release candidate and then picks its replacement with another read has taken the second kind in two steps, which is worth naming, because the log shows it as two innocent lines.

Retiring is the cheap half#

The verdict is administrative rather than dramatic. The set becomes a tuned-against set, and the most useful thing to do with it is merge it into the daily one, which takes a 200-case set to 320 and improves its standard error from 1.2 points to about 0.95. That improvement is the inverse-square-root law applied once: 200 cases to 320 multiplies the count by one and six tenths, so the standard error is divided by the square root of one and six tenths, which is a little over a quarter more than one, and 1.2 points divided by that is about 0.95. It is a modest gain for 120 cases of labelling, and it is worth having because it is permanent and because it applies to every comparison the daily set will make from then on. Weeks of labelling work are not lost; they change role. Growing the daily set this way, once per burn cycle, is the main compensation for the coverage that sits idle while a set is held out, and it means the regression guard gets steadily stronger over the life of the product.

Building the replacement follows the sampling and labelling process from topic 3.4, on building a golden dataset, with three constraints carried over from the old set. Match the size, so that transfer ratios computed against the new set are comparable to the ones computed against the old — a jump from 120 to 200 cases changes the standard error and makes the historical numbers non-comparable. Match the stratification, meaning the same proportions of contract types, clause categories and document lengths, because a set with different composition will diverge from the daily set for reasons that have nothing to do with fitting and will look exactly like overfitting when it does. And draw from recent traffic rather than from the pool the old set came from, since a fresh draw buys a check on distribution shift at no extra cost.

# evals/holdout/MANIFEST.yaml
set_id: holdout-2024Q3
created: 2024-06-24
created_by: r.okonjo
supersedes: holdout-2024Q1        # burned 2024-06-17, merged into evals/dev/
n_cases: 120
drawn_from: production traffic, 2024-05-24 to 2024-06-21, stratified sample
stratification:                    # matched to holdout-2024Q1 within one case
  supply_agreement: 44
  saas_msa: 31
  employment: 26
  nda: 19
labelled_by: [r.okonjo, t.baptiste]
inter_annotator_kappa: 0.81
owner: r.okonjo                    # the only person who runs it
reads_budget_per_quarter: 20
reads_used: 0
retire_when:
  - reads_used > reads_budget_per_quarter
  - any read is used to choose between two prompt variants
  - age > 2 quarters

The inter_annotator_kappa line is the two labellers' agreement with the agreement they would have reached by chance already subtracted out, on a scale where 1.0 is perfect and 0 is coin-flipping; 0.81 is high enough that the labels describe the task rather than one person's reading of it. It is worth knowing what that line buys, because agreement between labellers is a ceiling on everything measured against those labels. A set whose labellers agree at 0.81 can support finer claims than one whose labellers agree lower down, and the difference is not a matter of pride — it decides whether a four-point improvement is a claim these labels are able to carry at all. Recording the figure in the manifest is what makes the set's limits legible to somebody who arrives after everyone who built it has moved on.

The manifest sits next to the cases, is reviewed like code, and the reads_used counter is incremented by the runner rather than by hand. The creation date matters more than it looks: without it, a set that has been in the repository for three years reads as authoritative when it is in fact describing traffic that stopped being representative around the second product launch.

Sizing the set against the job#

The objection worth taking seriously comes from whoever paid for the labelling. Holding 45 of a 150-case golden set out of daily use means 45 cases of real, expensive coverage detecting nothing: a regression that only affects employment contracts, if that is where most employment contracts live, ships and stays shipped until the next scheduled read. The cost is concrete and recurs every day; the benefit is a statistical correction nobody ever sees happen.

The rest of that argument is quantifiable and it comes out in the objector's favour, so it is worth conceding in full. Splitting 150 into 105 and 45 raises the standard error on the daily set from 2.9 points to 3.5, so every comparison made while iterating gets about 20% noisier, which means more variants tried to reach the same confidence, which burns the daily set faster. That is the same square root once more: keeping 105 of 150 cases leaves seven tenths of them, one divided by the square root of seven tenths is about one and a fifth, and 2.9 points multiplied by that is 3.5. The consequence then chains, which is what makes the objection more than an accounting complaint — a noisier daily set resolves less, so more variants get tried before anything looks convincing, and more variants is more selection gain. Holding cases back badly makes the set you kept worse at the job you kept it for. And a 45-case held-out set is nearly useless at that size.

The first column assumes two prompts that disagree on about 8% of cases, which is typical for successive versions of a working prompt. It is not a table to be taken on trust: it is the paired standard error from the chapter on comparing two prompts, evaluated at each size. Take the 120-case row. Eight in a hundred of 120 cases is about ten discordant cases; the spread of the paired difference is twice the square root of a quarter of that count, divided by the number of cases, which comes to a little over two and a half points; two of those either side is the 5.1 in the table. Do the same at 45 cases and the discordant count drops below four, the spread rises, and the smallest resolvable difference is 8.3. The column deteriorates so sharply at the small end because the discordant count and the divisor shrink together and do not cancel: resolution improves only with the square root of the cases.

The read budget in the third column is a working rule of thumb — roughly one read per six cases — calibrated so that accumulated selection gain across the quarter stays under about one point. It scales with the set for the same reason everything else here does. More cases means a smaller standard error, which means each read supplies less selecting power relative to the size of the differences anyone is arguing about, which means more reads can be afforded before the accumulated gain matters.

Both columns get worse quickly as the set shrinks, and a 45-case set that can only distinguish differences wider than 8 points and can absorb seven reads a quarter is a bad trade.

Where the objection goes wrong is the alternative, which is keeping all 150 in daily use and relying on people to remember not to tune against a subset. The resolution is to size the set against the job instead: decide the smallest difference between what the daily set claims and what actually generalises that would change how the team works — five points, for most teams — read 120 off the table, and go and label 90 more cases. Eight hours of labelling turns an argument about splitting a scarce resource into a question about whether an independent estimate is worth eight hours, which is a much better question and usually has a clear answer.

The set's life, written as states#

Governance is easier to follow when the set has a small number of states and explicit transitions between them, because then "is this set independent right now?" is answered by looking at one field.

The transition worth arguing about is the second route into Burned, because it fires after a single read. One comparison read is enough, since the prompt that shipped was chosen with the set's help and no later read can undo that.

The irreversibility is worth understanding rather than accepting, because it is what makes the harshness defensible. Every other resource in an evaluation setup can be rebuilt: cases can be redrawn, labels re-derived, rubrics rewritten, runners reimplemented from the manifest. Independence is not something the files hold. It is a claim about a sequence of past events, and no operation performed afterwards changes what happened, so the counter can only ever move one way. Teams find this harsh and then discover that the harshness is the point: a rule with exceptions gets an exception every launch, and the counter that would have caught it is the one nobody incremented. The last transition is the operational trick that keeps this affordable — draw and label the successor while the current set is still active, so the gap between burning one and having another is days rather than the three weeks it takes to accumulate traffic, sample, label and reach agreement.

Three checks on the policy at six months#

Governance documents accumulate whether or not they do anything, so the policy needs its own evaluation, and three checks cover it.

The first is whether reads stayed inside budget, which is a one-line audit of the read log against the manifest. Twenty budgeted and nineteen taken is a policy that held; twenty budgeted and thirty-four taken is a policy people routed around, and the interesting question then is whether the budget was wrong or the process was. A team that overshoots because every release needs a read has a budget problem, and should either raise the budget and grow the set to match or release less often.

The second is whether the gap between the two sets stopped widening. What you want to see over two quarters is a gap that oscillates in a band a few points wide rather than climbing. A gap growing from 2 points to 6 to 11 across three checkpoints means tuning continued regardless of what the policy says, and the read budget was spent watching it happen rather than preventing it.

The third is the retrospective check, and it is the only one that tests the policy's actual claim. Take the last four changes that shipped on the strength of a held-out result and re-measure them on the current held-out set, which is fresh and had no part in any of those decisions. If a change approved on a predicted +4 points comes back at +3.5, +5, +1 and +4, the policy is producing estimates that hold up. If the four come back at +0.5, −1, +2 and 0, the reads were being taken but the decision rule around them was too loose — most likely accepting any positive number, which approves roughly half of all worthless changes. Six months of that data also gives a calibration curve: predicted improvement on one axis, realised improvement on the other, one point per shipped change. A slope near 1.0 means the process is honest; a slope near 0.4 means every number the team reports should be multiplied by 0.4 before it reaches anyone outside.

That instruction needs its conditions attached, because a slope is a fragile thing to compute from four points. A slope means nothing unless the predicted values span a range — four changes all predicted at plus 4 points give a vertical scatter and no slope at all — and quoting one to a decimal place off a handful of points is unreasonable. What the check demonstrates at four points is the sign and the order of magnitude: whether realised gains land near the predicted ones, systematically below them, or nowhere near them. Six months of shipped changes is where it becomes a curve worth reading. Before that it gives a direction, and a direction is enough to decide whether the decision rule around the reads is too loose.

Governance is not correctness#

Writing the policy tends to produce a feeling that evaluation is now handled, and the boundary is worth being specific about. A manifest makes the held-out score an honest estimate of performance on cases drawn like the held-out cases, judged by the rubric the team wrote, labelled by two annotators who agreed 81% of the time above chance. None of the governance touches whether that rubric measures what the legal team cares about, and a perfectly independent estimate of the wrong quantity is still the wrong quantity. If the annotators share a misreading of what counts as an indemnity clause, the fresh set inherits it exactly as the burned one did.

It is also not a substitute for watching production. A held-out set answers one question — how much of a measured gain is real — and it answers it every few weeks on 120 cases sampled from a month-old window. Latency regressions, a customer whose documents arrive in a format nobody sampled, a retrieval index that silently stopped updating: none of those appear on it, and all of them will appear in complaints. The sensible division is that the held-out set governs claims about improvement, the daily set governs claims about breakage, and production telemetry governs claims about what users are experiencing right now.

One mitigation for the waste argument is worth adopting because it costs nothing. Rotate which slice is held out: if this quarter's set is drawn to match the overall traffic mix and next quarter's is drawn the same way from a different window, no particular clause type ends up permanently outside daily use, because each retired set is merged back in when it burns. Over a year the daily set grows by roughly one held-out set per quarter, and coverage that was idle in one quarter is working every day from the next.

What all this policy protects is a single fact about a directory of files: nothing in it has ever participated in a decision. That fact cannot be restored, recomputed or backed up, which is why the mechanisms around it are budgets, counters and logs rather than tests. Everything else in an evaluation setup can be rebuilt from the cases and the rubric; independence can only be spent.

From here the discipline stops being a topic and becomes part of how a change gets shipped. The question to ask before opening the eval runner is which set you are allowed to look at today, and the question to ask before quoting a number is which set produced it and how many decisions that set has already made. Neither question has a universal answer, because both depend on circumstances a manifest records and no textbook can: how big the set is, how often it is read, what it was drawn from, and who is allowed to spend it.