Held-out sets and read budgets

A held-out set is a second eval set, drawn the same way as the set you tune against and then deliberately left unread while the prompt is being worked on. Leaving it unread is what keeps its score an estimate of quality rather than the quality-plus-fit that a tuned-against score becomes once selection leakage has had three weeks to accumulate. You meet it as a second directory beside the first, run on a schedule or before a release instead of on every commit, and quoted in the places where the tuned-against number cannot honestly go.

That description contains the whole fragility. The remedy works because of a property the data does not have and a process does, which means the remedy can be destroyed without anybody editing a byte. This chapter is about how that happens and what it costs. It counts the information a single read returns and compares it against the information a variant search actually consumes, works through the access modes that slow the leak and the two that meaningfully stop it, and then puts the rule somewhere it survives people who have never heard of it.

The difference lives in the process, not in the files#

Two sets drawn identically from the same population have identical statistical properties: same expected difficulty, same expected score for any given prompt, same variance. Open a tuned-against set and a held-out set side by side and they are indistinguishable — same document types, same categories, same rate of ambiguous edge cases. If you swapped the directory names, nothing about either set would change. Their difference is entirely a difference in how they have been used, and two consequences follow.

The first is that a held-out set can be destroyed without editing a byte of it. Corruption here is not a change to the data; it is an accumulation of decisions made while looking at the data. There is no diff to review, no checksum that fails, no test that goes red. The set on disk after two weeks of daily plotting is bit-for-bit the set that was committed, and it no longer answers the question it was created to answer.

The second is that "held out" is a claim about history rather than a property you can verify by inspection. If you inherit a repository with a directory called holdout, you cannot tell from the contents whether it is independent. The only evidence that would settle it is a record of every time it was read and what was decided afterwards, and most teams have no such record, which means most inherited held-out sets have unknown status and should be treated as burned until proven otherwise.

Counting what one read returns#

The usual defence at this point is that nothing really leaked, because nobody opened the held-out cases. All that was seen was one number per run. It feels like the difference between reading the exam paper and being told your grade.

Counting the information makes the size of that difference concrete. A held-out run on 120 binary-scored cases produces at most 120 bits if you read every case outcome. Collapsed to a single score, the same run yields one of 121 possible values, which is 6.9 bits. Reading only the aggregate cuts the per-read leak by a factor of about seventeen, and seventeen is a much smaller factor than it needs to be, because of how little information a selection decision actually consumes. Choosing the best of thirty variants requires distinguishing thirty options, which is 4.9 bits total, spread across the whole campaign. A single aggregate held-out score carries 6.9 bits, so one read supplies more choosing power than an entire thirty-variant search needs.

All three of those counts come from the same operation and are worth doing rather than accepting. Reading every case outcome hands over 120 separate yes-or-no facts, and a yes-or-no fact is one bit by definition, so 120 is the ceiling. Collapsing them into a score discards which cases passed and keeps only how many, and a count of passes out of 120 can be anything from none of them to all of them — 121 possibilities, counting zero. Pinning down one of 121 possibilities takes about seven yes-or-no questions, since six halvings bring 121 down to about two and a seventh finishes it, which is the 6.9. Identifying one variant out of thirty takes about five halvings — thirty, fifteen, eight, four, two — which is the 4.9.

The 6.9 is a ceiling and a real read usually carries less, which is worth saying because that gap is where most of the reassurance people feel actually lives. It would be the exact figure only if all 121 scores were equally likely, and they are not: a set that came back in the low seventies last week will come back in the low seventies this week, so most of the possible values carry almost no probability and the read tells you correspondingly less than the count allows. The bound is still the right quantity to compare against, because what it is being compared with — the choosing power a selection needs — is a bound too, and comparing a ceiling against a ceiling is the only honest version of this argument.

Bits are an upper bound on what could have travelled, and the mechanism by which it actually travels is easier to see than the counting suggests. Whenever the two numbers disagree — the tuned-against score up half a point and the held-out score down, or the reverse — a human resolves the disagreement, and the resolution is a decision in which the held-out score participated. It does not matter that nobody can name which cases drove it. A prompt that survived thirty-four such filters is a prompt the held-out set helped select.

The formulation that stops the leak entirely is a rule about decisions rather than a rule about eyeballs: a set is independent for exactly as long as no artefact you ship was chosen with its help. Under that rule, whether anybody looked at individual cases or only at a line on a chart is beside the point, because both routes end with a kept edit that would have been discarded had the held-out number gone the other way.

Two engineers make the difference sharp. The first opens the held-out directory, reads every case in it over an afternoon out of curiosity, and then ships whatever the tuned-against set recommended. The second never sees a case, watches only a line on a chart, and reverts three edits because the line dipped. Under a rule about eyeballs the first has committed the offence. Under the rule that matters, the second has spent the set's independence and the first has not, because independence is a property of how the shipped artefact was chosen rather than of what anybody has seen. Reading is dangerous mainly because it usually comes just before deciding.

The last two rows of the table are the only access modes that meaningfully slow the leak, and they do it by refusing to answer most questions. The bottom row is the reusable-holdout idea from the adaptive-analysis literature made operational: perturb the held-out score, compare it against the tuned-against score, and return the perturbed value only when the two disagree by more than a margin, otherwise return the tuned-against score unchanged. Because the answer is usually a repeat of something you already knew, most reads transmit almost nothing.

import numpy as np

class Thresholdout:
    """Bounded-leak access to a held-out set (Dwork et al., 2015)."""

    def __init__(self, tolerance: float = 0.04, budget: int = 20, seed: int = 0):
        self.tolerance = tolerance
        self.budget = budget
        self.rng = np.random.default_rng(seed)

    def report(self, dev_score: float, holdout_score: float) -> float:
        noisy_gap = abs(dev_score - holdout_score) + self.rng.normal(0, self.tolerance / 4)
        if noisy_gap < self.tolerance + self.rng.normal(0, self.tolerance / 2):
            return dev_score  # they agree; the read told you nothing new
        if self.budget <= 0:
            raise RuntimeError("holdout budget exhausted; this set is now a dev set")
        self.budget -= 1
        return holdout_score + self.rng.normal(0, self.tolerance / 2)

Two mechanisms inside that method do the work, and the second is the one that gets dropped from re-implementations. The first is that the usual reply is not the held-out score at all: it is the tuned-against score, handed straight back. When the two agree, the caller is told something they already knew, nothing about the held-out cases has travelled, and the budget is not charged. The leak per read collapses because most reads are not answers.

The second is that noise is added to the comparison as well as to the returned value. Without it, a caller could recover the held-out score to arbitrary precision by walking the threshold — asking repeatedly with slightly different tuned-against scores and watching where the reply flips from a copy to a report. The location of that flip is itself a function of the held-out cases, so a sharp boundary is a channel in its own right, and one that leaks faster than the value it was protecting. Randomising where the boundary sits is what closes it, which is why the code perturbs the tolerance and not only the number it returns.

Two things about that class are worth saying plainly. It is genuinely used in adaptive-analysis research and it is genuinely rare in production eval tooling, partly because the noise it adds makes the reported number harder to explain to anyone outside the team. Most teams get most of the benefit from the crude version, which is a hard cap on reads and a rule that the cap is the whole defence. What the class illustrates regardless is the shape of the answer: the budget is finite, and the mechanism has to know when it has been spent.

Moving the rule out of people's heads#

Because the rule is a rule about behaviour, the only durable place to put it is in the tooling that the behaviour goes through. Three changes cover most of it, and none requires new infrastructure.

# 1. The iteration script physically cannot reach the held-out data.
#    evals/holdout/ is not a sibling of evals/dev/ — it lives behind an owner.
$ cat .github/CODEOWNERS
evals/holdout/**   @eval-owner

# 2. The only entry point that can read it demands a reason and logs the read.
$ python -m evals.holdout_run --prompt prompts/extract@v31.md
ERROR: --reason is required. Held-out reads are budgeted: 12 remaining this quarter.

$ python -m evals.holdout_run --prompt prompts/extract@v31.md \
      --reason "release candidate for 2024-04-11 ship"
holdout score: 0.743  (n=120, run 9 of 20 in budget)
appended to evals/holdout/READS.log

# 3. CI fails any change that imports the held-out loader outside that entry point.
$ rg -l "holdout" --glob '!evals/holdout_run.py' --glob '!evals/holdout/**' src/ evals/
evals/iterate.py
FAIL: evals/iterate.py references the held-out set outside the budgeted runner.

READS.log is append-only and committed, one line per read with date, prompt version, reason and result, because the count is the only thing that tells you how much independence is left. A new engineer who tries the obvious thing hits an error message that explains the rule, which is the only form of documentation that reliably gets read.

Independence is not correctness#

A held-out set does not make your evaluation correct. If both sets were labelled by the same two people using the same guidelines, a systematic labelling error — say, treating a clause reference in a schedule as out of scope when the legal team considers it in scope — sits in both sets identically. Independence protects against fitting to the noise in your labels. It offers nothing against a shared bias in them, and a confident held-out score on wrongly-labelled data is a confidently wrong number.

Why the split helps against one and not the other comes down to where the error sits relative to the split. Label noise varies from case to case, so a prompt fitted to the noise in one set gains nothing on another set whose noise fell differently — and detecting exactly that gain is what the held-out score does. A shared misreading of the guidelines is not noise; it is a constant. It applies the same way to every case in both sets, so it passes through the split untouched and both scores agree while both are wrong. Splitting data can separate a signal from noise. It cannot separate a signal from a definition, because the definition was applied to both halves before the split existed.

It also does not measure distribution shift, which is the thing people most often assume it does. Two sets drawn from the same month are equally stale; if the traffic mix changes because a large customer onboards with a different document format, both will agree with each other while disagreeing with production. Detecting that requires a set drawn from recent traffic, which is a different axis entirely from held-out versus tuned-against.

And it is not a harder set. A held-out set that scores systematically lower from the start usually means the two were not drawn the same way, which is a bug in how they were constructed. The two should start at the same score for the same prompt, and that shared starting point is what later licenses reading a divergence between them as fitting rather than as difficulty.

That licence is an inference and worth stating as one, because it is the whole basis of the next chapter. Difficulty is a property of the cases, so two sets that differ in difficulty differ in it on the first day and on every day after. A measured equality on the first day therefore rules difficulty out for all the later days as well, without anyone needing to re-check. A divergence appearing afterwards has to come from something that changed in between, and since the files did not move there is exactly one candidate left standing.

The last limit is the one that follows from the budget. A set that may be read twenty times a quarter cannot answer twenty questions a quarter, because most of those reads have to be reserved for release decisions. Deciding what a read is for, in advance, is part of operating the set, and a read spent confirming something you already believed is a read you cannot take later.

What makes a held-out set work is not its contents and not its size. It is an unbroken history of not having participated in any decision, and that history is the only asset here that cannot be rebuilt from a backup. Everything practical follows: the read count is a budget, the budget belongs in a runner rather than in somebody's memory, and the log of reads is the only evidence that will survive the people who took them.

There is a question this chapter has been deferring. Suppose everything had been done right — one read, at the end, on a single release candidate, properly logged — and it came back at plus 0.4 points. That is a legitimate read of an independent set, spent exactly as intended. It is also, as the next chapter works through, a number small enough to be entirely consistent with no improvement at all, which means the read was spent on a question the set was never given enough cases to answer.