The dev-to-holdout gap

The dev-to-holdout gap is the difference between the score of the current prompt on the set that has been guiding every accept-or-reject decision and its score on a set that has taken no part in any of them. On its own a single reading of that gap says very little, because two sets can differ for reasons that have nothing to do with tuning. What makes it readable is a baseline: the same prompt scored on both sets before any iteration happened. If the two agreed then and disagree now, the sets have not changed and the process has, and the size of the disagreement is the size of the fit.

This chapter reads one such trajectory end to end, eliminating the four explanations that have to be ruled out before a widening gap can be called overfitting, and converting what survives into a point estimate with an interval.

Reading the trajectory#

Six weeks of prompt work on a contract-clause extractor, roughly thirty iterations, and two reads of the reserved set — one at week three and one at week six — produce a table like this.

The two rows that carry the argument are week 0 and week 6. At week 0, with the same prompt on both sets, the scores were 0.72 and 0.71 — a one-point difference on sets of 200 and 120 cases, well inside sampling noise and exactly what two samples from the same population should look like. Six weeks later the same prompt scores 17 points higher on one set and 3 points higher on the other. The sets did not change. What changed is that one of them was consulted about thirty times and the other twice.

Writing the two scores as sums is what turns that observation into an estimate. A score on the set you tuned against is the system's real quality, plus whatever fit to those particular documents the tuning bought, plus sampling noise. A score on the reserved set is the same real quality plus its own sampling noise, and no fit term at all, because no decision was ever made with its help. Subtracting the second from the first cancels the quality term — it is one system measured twice — and leaves the fit term plus the difference of two noises. That cancellation is the whole method, and it is available only because of the week-0 row. Had the two sets started apart, there would be a fourth term for the difficulty difference, it would not cancel, and nothing measured later could separate it from the fit.

Run it on the table. Seventeen points of movement on the tuned-against set, three points on the reserved one, and the difference is about fourteen points of fit. Neither of those movements is noise-free, which is why the section after next puts an interval around the three rather than reporting it bare.

Two properties of that table are worth noticing before the explanations. The gap grows monotonically with cumulative variants rather than with calendar time, which is the signature the mechanism predicts, since it is decisions and not days that move information into the prompt. And the reserved set is not flat — it moved 3 points, which is the part of the work that transferred, and a trajectory where it moves nothing at all is a different and worse finding.

Ruling out the explanations that are not fitting#

A widening gap has more than one possible cause, and the case for overfitting is only as good as the alternatives you can eliminate. Four are worth checking, and three of them are cheap.

The first is that the reserved set is simply harder — more ambiguous documents, more clause types the model struggles with. That is ruled out by the week-0 row. A harder set scores lower from the beginning, and this one did not; it started one point below, which is the wrong sign and the wrong magnitude for a difficulty difference. Any explanation requiring the reserved set to be intrinsically more difficult has to account for why that difficulty was invisible in week 0 and worth 15 points by week 6.

The second is that the sets were drawn or stratified differently, which would show up as a difference in composition rather than in overall difficulty. Checking it is a matter of counting: contract types, clause categories, document lengths, source customers. If one set is 40% supply agreements and the other 22%, the two are not exchangeable and the comparison needs reweighting — scoring each category on its own and recombining those scores under one shared set of proportions — before it means anything.

Two categories are enough to see what reweighting does. Take one set at 40% supply agreements, which the extractor handles well, and the other at 22%, and suppose that within each category the two sets score identically. The overall figures will still differ, purely because the easy category carries a larger share on one side, and the difference will look exactly like a gap. Reweighting scores each category separately, where the two agree, and then recombines both under a single set of proportions — either set's, or production's, as long as it is the same one on both sides. The recombined numbers come out equal, which is the right answer, and the apparent gap is revealed as a fact about how the samples were composed rather than about the extractor.

The third is distribution shift. Two sets drawn from the same month are equally fresh, so shift cannot produce a divergence between them; it can make both of them stale relative to production, which is a real problem and a different one. That is a structural argument rather than an empirical one, which is why it costs nothing to check: shift reaches both sets through the same channel, the calendar, and a cause acting equally on two quantities cannot open a gap between them. Anything that can open a gap has to be something the two sets do not share, and in this whole arrangement there is exactly one such thing.

The fourth is drift in the measuring apparatus — a judge model updated by the provider, a rubric edited mid-project, cases added along the way. This is the one that needs an experiment rather than an inspection, and the experiment is to check out the week-0 prompt and run it against both sets today.

$ git checkout prompts/extract@v1.md
$ python -m evals.dev_run --prompt prompts/extract@v1.md
dev score: 0.715  (143/200)   # week 0 reading was 0.72

$ python -m evals.holdout_run --prompt prompts/extract@v1.md \
      --reason "baseline re-measurement, overfitting investigation"
holdout score: 0.708  (85/120)  # week 0 reading was 0.71
appended to evals/holdout/READS.log   (run 3 of 20 in budget)

The baseline reproduces on both sets to within half a point, so the instrument is stable and the judge has not drifted. With difficulty, composition, shift and drift all accounted for, what remains is that thirty rounds of accept-and-reject decisions were made using one of the two sets, and about 14 of the 17 points is fit to those particular 200 documents.

The number that replaces seventeen#

The defensible estimate of real improvement is 3 points, and it comes with an interval that has to travel with it. Run as a paired comparison of the week-0 and week-6 prompts on the same 120 reserved documents, the two prompts disagree on 14 of them, with the newer prompt correct on 9 of those. That is a standard error of about 3.1 points, so the estimate is 3 points with a 95% interval running from roughly −3 to +9. Zero is inside it.

Every figure in that sentence is the paired calculation from the chapter on comparing two prompts, run on 120 cases rather than 200. Fourteen discordant cases give a coin-flip spread of a little under two cases. Doubling it — because shifting one case from one column to the other moves the difference by two — gives about three and three quarter cases. Dividing by the 120 reserved documents turns that into 3.1 points. The observed difference is nine cases against five, a net of four cases in 120, which is the 3 points. An interval of about two standard errors either side of a 3-point estimate reaches from roughly −3 to +9. The same formula and the same steps as before, on a smaller set, and a much wider answer: the price of holding back 120 cases instead of 200 is paid here, in resolution, and it is the reason a reserved set has to be sized against the smallest difference it is expected to settle.

That is a genuinely awkward number to communicate, because it says both that the honest estimate is far below what has been reported and that the honest estimate is itself uncertain. What works is separating the point estimate, the single number that is your best guess, from the interval that says how far the truth could sit from it, and saying both. An interval that contains zero is not the same as a measurement of zero: the best available reading is still that the work helped somewhat, and the honest addition is that a set of this size read twice cannot distinguish somewhat from not at all.

Splitting the difference is not a method#

The response this situation attracts is that the truth must be somewhere between the two measurements, so 10 points is a fair compromise. It has the shape of prudence and it is a fabrication.

Two measurements bracket the truth when they are two noisy estimates of the same quantity. These are not. The dev figure is an estimate of quality-plus-fit and the independent figure is an estimate of quality, so they measure different things, and the gap between them is the fit component rather than a disagreement about a shared quantity. Averaging them produces a number that is half a quality estimate and half a contamination estimate, which nothing in the world corresponds to.

Averaging is a real technique with real conditions attached, and both of them fail here. Two readings bracket a truth when they are noisy measurements of one quantity, and averaging then helps for a definite reason: the two errors are independent, so they partly cancel. Here the errors are not independent — the fit term is present in one number and absent from the other by construction, which is a systematic difference rather than a coincidence of sampling — and there is no single quantity for the two to be errors about. Averaging estimates of the same thing reduces noise. Averaging estimates of different things produces an estimate of a mixture, and nobody proposing 10 points has said what the mixture is a mixture of.

The interval around 3 points already carries the honest uncertainty and it does not extend to 10; quoting 10 asserts a precision no run supports, in the direction that is socially easier, and produces a figure that cannot be defended by pointing at any measurement anyone took.

There is a version of the midpoint instinct that is defensible, and it is worth naming so it does not get thrown out with the rest. If the two sets genuinely were drawn differently — different customers, different document mix — then reweighting one to match the other's composition and reporting the reweighted number is a real method with a real justification. It requires the composition difference to exist and to have been measured, which is the second of the four checks above, and it is the one case where a number between the two is the answer rather than a compromise between them.

The questions a burned set can still answer#

Declaring a tuned-against set burned invites the conclusion that it should be deleted, which would throw away six weeks of accumulated failure knowledge along with the contamination. The set is still valuable, and what changes is which questions it is allowed to answer.

The row that most needs a reason under it is the first one. If the score is inflated by six weeks of fitting, why is a ten-point drop still trustworthy? Because fitting raises the level and does not blunt the sensitivity. The cases the prompt was fitted to are cases it now passes, and a change that breaks the extractor makes them fail exactly as it would on any other set. What contamination destroys is the comparison of two numbers near the top, where the fit component is the same size as the effect being argued about. It leaves untouched the question of whether a large number of cases have just stopped working, because no amount of fitting to a set makes that set blind to a break in the thing it exercises. A regression guard has to be sensitive rather than independent, and those are different properties that happen to be supplied by the same file.

Running the set in continuous integration on every commit remains correct, with the gate set as a floor rather than a target: fail the build if the score drops by more than 5 points, and stop treating increases as news. The set has become a regression guard, which is a durable role it can hold indefinitely, because a regression guard does not need to be independent — it needs to be sensitive to breakage, and a contaminated set is fully sensitive to breakage.

It is also the right place to keep growing coverage. Adding a check for a property no case scored before, or a new slice of cases for a document type that started appearing in production, makes it a better failure catalogue without pretending it is a better estimator. New cases inherit the same status as the old ones the first time somebody tunes against them, and that is fine as long as nobody is quoting the total.

A gap between two sets that started level is the closest thing this work offers to a direct measurement of its own contamination. It requires a baseline read taken before any tuning, which costs one read and is the single most valuable read in the set's life — valuable because it is the only read that can never be taken later. Every other read measures the current prompt, and a current prompt can always be measured again next week. The week-0 read measures a prompt nobody will ship and a state of the process that has already ended, and without it the difficulty term never cancels, so every gap measured afterwards carries an unknown constant nobody can size. It also requires the discipline to eliminate difficulty, composition, shift and drift before drawing the conclusion. What comes out is one number with an interval, reportable, and one instruction: the movement on the set you tuned against has stopped meaning anything, so stop generating more of it.

Six weeks and thirty variants produced roughly 14 points of fit on a 200-case set. That is one observation from one team, and the obvious next question is whether thirty is the number to watch for, or six weeks, or something else entirely — because a team with a 40-case set and an aggressive search can burn through the same amount of independence in four days.