Selection leakage in prompt tuning

An eval set — short for evaluation set, and usually called the dev set, for development, once a second set exists alongside it — is a fixed collection of inputs paired with the answer somebody has decided is correct. For a system that extracts clauses from contracts, that means a few hundred real contracts, each marked up by a lawyer with the clauses a correct extraction should return. Running the set feeds every case through the current prompt, counts how many extractions match their labels, and yields one number between 0 and 1. Such a collection exists because there is no other practical way to tell whether a prompt edit helped: a single edit changes behaviour across every input at once, reading a dozen outputs by hand gives an impression two people will report differently, and impressions do not compare across versions or across weeks. Assembling one is the subject of topic 3.4, on building a golden dataset; this topic is about what happens to a set as you use it.

What happens is eval-set overfitting: the score on those particular cases climbs while the quality anyone outside the team experiences stays where it was, because the prompt has been shaped, edit by edit, to the specific cases in the set rather than to the work those cases were meant to stand for. This chapter is about the mechanism that produces it, which is neither carelessness nor copying. It is selection leakage — the accept-or-reject decision that follows every measurement, carrying a little of the set into the artefact each time. The chapter shows that loop as a training loop, counts how much information one score returns, simulates a search in which every variant is stipulated to be equally good and watches the number climb anyway, and then separates this failure from the two contaminations that teams already know to look for.

The optimiser is in the chair#

The usual objection to the word "overfitting" here is not unreasonable. Nothing was trained. There is no optimiser, no loss function, no backward pass, no gradient anywhere in the system; the prompt is a text file in a repository and the model behind it is a hosted endpoint whose weights are frozen and identical on every call. All of that is true, and the conclusion drawn from it is wrong, because training is not the only way for information to travel from a set of examples into a system that is later judged on those same examples. The general name for that travel is train/test leakage: something from the data used to build a system has reached the data used to judge it, so the judgement comes back flattering.

Consider what happens on each iteration of a week of prompt work. Somebody changes some text, runs the cases, looks at a number, and makes a decision — keep this edit or throw it away. The decision is determined by the score, and the score is determined by those particular cases and their labels. Whatever survives into the final prompt survived because of how those examples behaved.

Written as a chain it is three links, and each one can be checked on its own. The keep-or-discard decision is a function of the score. The score is a function of those particular cases and the labels attached to them. Therefore the prompt that comes out at the end is a function of those cases — not of the subject matter in general, but of these specific documents with these specific labels. Once the prompt is a function of the set, the set's score on that prompt has stopped being an independent measurement of it, because the thing being measured and the thing doing the measuring now share a history.

That is a training loop whose parameters happen to be English sentences and whose gradient estimate happens to be computed by a person reading a dashboard. Prompt tokens are the parameters, the dev score is the objective, the accept-or-reject decision is the update rule, and the number of variants tried is the number of steps taken. A training loop proposes a parameter update, measures the loss on a batch, and accepts the update in the direction that lowers it. The two differ in resolution and speed and they do not differ in kind, because in both cases the data is what selects the surviving configuration.

The channel by which information moves is narrow, which is what makes the whole thing feel innocent. A weight update absorbs a large amount of information about a batch in one step. An accept-or-reject decision carries a single bit, the amount of information in one yes-or-no answer, since all that comes back is up or down, and n bits are enough to pick one option out of 2^n. Thirty iterations therefore transfer on the order of thirty bits from the set into the prompt file, spread over three weeks, arriving one at a time through a human hand. Thirty bits is enough to specify one item out of a billion, which is considerably more choosing power than a prompt-tuning session actually needs.

The doubling is where that comes from and it is worth counting rather than taking on trust. One yes-or-no answer splits a collection of candidates in half, so it can single out one of two. A second answer halves what is left, so two answers single out one of four, and three answers one of eight. Every additional answer doubles the number of candidates that can be told apart, and thirty doublings starting from one goes past a billion. The reason that figure sounds absurd is that it is absurd: a week of prompt work needs nothing like that much selecting power, which is precisely why the leak stays invisible while it is happening. There is no moment at which a budget visibly runs out.

Two qualifications belong with the count and they pull in opposite directions. A bit is an upper bound on what could have travelled rather than a measurement of what did: the decisions are not independent of one another, most iterations go the same way as the one before, and a run of thirty accepts carries far less than thirty bits. In the other direction, the channel is wider than one bit per iteration whenever the engineer opens the failing cases before writing the next edit, because then the content of the edit carries information about those cases as well as the accept-or-reject decision does. The chapter on burn rate comes back to the second, which turns out to matter more than the first.

Thirty coin flips will buy you four points#

The claim worth pinning down is that this selection produces measurable score improvement even when none of the variants is better than any other. A simulation checks it, and the shape of the result is what you carry into every future decision.

Set up 120 cases the way real eval sets are usually shaped. Some fraction is easy: every plausible prompt gets those right, and they contribute no variance. Some fraction is beyond the model at the current state of the world, and every variant fails them. In between is a band of borderline cases where the answer depends on things nobody controls — where a sentence break landed, whether the clause used "shall" or "will", whether the sampler chose one token over another. Call it 72 free passes, 24 hopeless, and 24 borderline cases that come out right about half the time regardless of the prompt.

import numpy as np

rng = np.random.default_rng(7)

N_CASES, N_VARIANTS, N_TRIALS = 120, 30, 1000
ALWAYS_RIGHT, BORDERLINE = 72, 24  # remaining 24 cases every variant fails

def simulate_week() -> tuple[float, float]:
    # every variant is genuinely identical in quality: the same free passes,
    # the same hopeless cases, and 24 coin flips no prompt edit controls
    flips = rng.random((N_VARIANTS, BORDERLINE)) < 0.5
    dev = (ALWAYS_RIGHT + flips.sum(axis=1)) / N_CASES
    winner = int(dev.argmax())

    # re-run only the winner on 120 fresh cases drawn exactly the same way
    fresh = (ALWAYS_RIGHT + (rng.random(BORDERLINE) < 0.5).sum()) / N_CASES
    return float(dev[winner]), float(fresh)

runs = np.array([simulate_week() for _ in range(N_TRIALS)])
print(f"true quality of every variant : {(ALWAYS_RIGHT + BORDERLINE / 2) / N_CASES:.3f}")
print(f"best-of-30 dev score          : {runs[:, 0].mean():.3f}")
print(f"that same winner, fresh cases : {runs[:, 1].mean():.3f}")

# true quality of every variant : 0.700
# best-of-30 dev score          : 0.742
# that same winner, fresh cases : 0.700

Four and a bit points, produced by a process in which every variant was stipulated to be exactly as good as every other. The winner is not better at extracting clauses; it is the variant whose coin flips happened to land well on those particular 24 borderline cases. Run it on fresh cases drawn identically and it reverts to 0.700, because the thing that made it the winner was a property of the 120 cases rather than a property of the prompt.

The size of the free gain follows a rule you can carry in your head. The best of k noisy measurements sits roughly a fixed number of standard errors above the truth, where the standard error is how far a score typically wanders from the true quality simply because you measured on this sample of cases rather than another one drawn the same way. That number grows with the logarithm of k: about 0.8 standard errors for 3 variants, 1.2 for 5, 1.7 for 14, 2.0 for 30, and 2.2 for 50. The standard error itself depends on the set, and for the borderline band above it is 24 coin flips over 120 cases, or roughly 2 points. Two points of standard error times two standard errors of selection is the 4.2 that came out of the simulation, and nothing in that arithmetic required a real improvement to exist.

Both numbers in that multiplication can be derived rather than looked up, which matters because your own set will have different ones. Take the standard error first. The 72 always-right cases and the 24 hopeless ones contribute nothing that varies between variants, so every scrap of run-to-run movement lives in the 24 coin flips. The count of heads in a run of fair coin flips has a standard deviation equal to the square root of the number of flips, times a half, times a half. Twenty-four times a half times a half is six; the square root of six is a little under two and a half cases; two and a half cases out of a hundred and twenty is about two points. The variants in the simulation are fictional and that calculation is not — it is the same one you would run on a real set, using the number of cases that actually sit on the boundary.

Then the multiplier. The best of k noisy measurements lands, on average, near the point where the upper tail of the distribution holds about one draw in k, which is what being the largest of k means. Getting further out into that tail divides the remaining mass by larger and larger factors for each additional standard error travelled, so the distance needed grows with the logarithm of k rather than with k. That is why the list climbs so slowly, and it is the part with a practical consequence: doubling the search from 14 variants to 30 moves the multiplier by three tenths of a standard error, so halving your search breadth removes much less of the free gain than the effort saved would suggest.

Statistics named this before machine learning did#

The phenomenon predates prompts by about forty years. In 1983 David Freedman published a short paper showing that if you regress a response on fifty predictors that are all pure noise, keep the ones significant at the 0.25 level, and refit on just those, the second regression comes back with a highly significant overall F statistic and a respectable R-squared. Nothing in the data meant anything. The selection step was the entire effect, and the second model's diagnostics had no way to report that its predictors had been chosen using the same rows it was now being scored on.

The count inside that result is what makes it stick. Screening fifty pure-noise predictors at the 0.25 level passes about a quarter of them by chance, so roughly a dozen survivors go into the second regression, and every one of them was chosen precisely because it happened to line up with this sample's response. The second regression's diagnostics are then computed under the assumption that its predictors were named before the data was seen, which is what an F statistic and an R-squared mean. There is no field anywhere in that output where the screening step could have been declared, and no reader of the output could detect it. The model is not lying; it is answering the question it was asked, and the question omitted the part that mattered.

Statisticians call the general case selection bias or the winner's curse, and it is why pre-registration exists in clinical trials.

Machine learning rediscovered it on public leaderboards. A Kaggle competition gives everyone a labelled training set and a hidden test set, and lets competitors submit predictions against a public leaderboard several times a day. The public leaderboard is scored on a slice of the hidden data, and the final ranking is scored on a different slice nobody has ever seen. The gap between the two rankings is routinely large enough to reorder the top twenty, and the reordering is not random noise: teams that submitted three hundred times fall further than teams that submitted twenty, because each submission was a selection decision made against the public slice. The count is doing the same work it did in the simulation: three hundred submissions is three hundred selections against one slice of data, and the maximum over three hundred noisy readings sits further above the truth than the maximum over twenty. No claim about the teams' skill is needed anywhere in that explanation, which is what makes a leaderboard such a clean demonstration — the private ranking is a single measurement on a slice that took no part in any of the selections, so the drop between the two rankings is the selection effect with almost nothing else mixed into it.

The theory caught up in 2015, when Dwork, Feldman, Hardt, Pitassi, Reingold and Roth published work on adaptive data analysis: the setting where each question you ask of a set of examples depends on the answers to previous questions. Their central observation is that standard statistical guarantees assume the analysis was fixed before the data was seen, and every guarantee evaporates the moment the analysis is chosen adaptively. The adaptivity is the whole of the problem, and it is worth noticing that it is also the whole of the value: an analyst who never let the data influence the next question would be running a fixed script and learning nothing from failures. Their proposed fix, the reusable holdout, adds calibrated noise to answers from a reserved set and only reports them when they differ meaningfully from the answer on the set being tuned, which buys back a bounded number of honest queries. Under those bounds the number of queries n examples can safely absorb scales with n and with the square of the accuracy demanded, so wanting a score accurate to within 2 points on 120 cases puts the safe query count in the low tens.

The shape of that bound is worth carrying even without the proof behind it. Wanting an answer accurate to within some tolerance costs a number of examples growing as the inverse square of that tolerance, which is the same square-root law that governs any sample estimate, seen from the other side: twice the precision costs four times the data. Turned around for a set of fixed size, it says twice the precision costs a quarter of the queries. That is why 120 cases at two points of tolerance land in the low tens of safe questions rather than in the hundreds, and why phrasing a query more carefully does not move the number much — the constraint is on how much the answers can vary, not on what they are about. The theory and the simulation land in the same place, and neither offers a way to keep iterating on one set indefinitely.

Pasted cases and contaminated pretraining#

There is a cruder form of eval contamination that most engineers already guard against: putting eval cases into the prompt as few-shot examples, or pasting a tricky labelled case into the system prompt to fix it. That is direct copying, and it is easy to spot in a diff. Selection leakage leaves no such trace. Every individual edit in a thirty-variant history looks legitimate and generalisable, and the leakage lives in the choice of which such instructions survived rather than in their wording, so it cannot be reviewed out of a pull request.

It is also distinct from the training-data contamination discussed around public benchmarks, where a model's pretraining corpus happens to contain the test items. That is a property of the model, it arrives before you touch anything, and no amount of care in your own workflow prevents it. Selection leakage is a property of your process, it accumulates in your repository, and it is entirely within your control.

None of this says that iterating against a dev set is bad practice, since a set nobody looks at teaches nothing. Reading failures case by case, forming a theory and editing the prompt is how prompt work gets done, and a golden dataset is built precisely so that work has something to grip. What follows is narrower: the score on the set you iterated against stops being an estimate of quality and becomes an estimate of quality-plus-fit. Somebody who tried three variants has a nearly honest number. Somebody who tried three hundred has a number that describes their eval set.

One more distinction changes what you do about it. Selection leakage is not fixed by making the set harder, by relabelling it more carefully, or by improving the judge. Those are all good things and they address different problems. A perfectly labelled, perfectly representative 120-case set still has sampling noise in it, and the maximum over thirty variants still climbs that noise. The only thing that removes selection bias from an estimate is a set that took no part in the selection.

A dev score, then, is two quantities added together and reported as one. The first is how well the system does the work; the second is how well the prompt fits the particular examples that chose it, and that second quantity is manufactured by the search itself, at a rate set by the noise in the set and the number of variants tried. It cannot be reviewed away, argued away, or removed by better labelling, because it is produced by the honest, necessary act of looking at a number and deciding what to keep.

That is why the remedy is a second set, and the remedy comes with a property that makes it harder to operate than it first appears. The second set works because it has never been consulted, which means every consultation degrades the thing that made it work. A dev set can be run a thousand times and lose nothing it had not already lost. A set held in reserve is a consumable.