0.91 on the searched set is a fitted score, not an estimate

The structure of the mistake is familiar from ordinary machine learning, where nobody would report the score a model achieves on the data used to select it and call it accuracy. Prompt search hides the same structure behind a different surface, because the artefact is English rather than a weight matrix and the run looks like an experiment rather than like fitting. The tuning cases play exactly the role a training set plays: the search proposes, measures against them, keeps what scored well, and repeats.

A number computed on the data that did the selecting is therefore a property of the fit, which is what fitted score means here. Topic 3.5 works through what that costs a held-out set over weeks of human tuning, one accept-or-reject decision at a time; an optimiser is the same mechanism at several hundred decisions an hour, which changes the size of the effect and not its nature. What follows is the arithmetic of that effect at machine speed, and the split that gives you a number to put beside the fitted one.

How much of twelve points can noise manufacture on its own#

This is worth doing with actual arithmetic, because the intuition that "it's a bit optimistic" undersells it badly at 400 candidates.

The unit is worth pinning before a section is spent on it. The grader hands back a score between zero and one for each case, and a prompt's score is the mean of those across the cases it was run on. A point, throughout this topic, is a hundredth of that scale. So a winner reported at 0.91 against a hand-written prompt at 0.79 differs by twelve hundredths, and twelve hundredths of a scale whose whole width is one is twelve points. That is the number in the heading, and it is the number that got a shipping recommendation into a team channel.

Suppose, for the sake of the calculation, that every one of the 400 candidates is exactly as good as the baseline — the search found nothing real at all. Per-case rubric scores on this task have a standard deviation of about 0.33, and each candidate is scored on 200 cases, so the standard error of a candidate's mean score is 0.33 / √200 ≈ 0.023. Each candidate therefore lands somewhere around 0.79 with a wobble of roughly ±0.023. Now take the maximum of 400 such draws. The expected maximum of n independent standard normal draws is close to √(2 ln n), which for n = 400 is about 3.5, and simulation puts the practical figure near 2.9 standard errors once you account for the distribution's shape.

Two of those steps are doing real work and neither should be taken on faith. The first is the division. The spread of the individual case scores is 0.33, the square root of 200 is a little over fourteen, and 0.33 divided by that is the 0.023 — the wobble on a mean rather than on any one case, smaller than the per-case spread because averaging lets errors in opposite directions cancel. The second is the √(2 ln n), and it is the counter-intuitive one, because it says that taking a maximum over more and more draws helps less and less. A normal distribution's tail thins extremely fast, and an extra draw only changes the maximum if it beats every draw before it, so the record improves at the pace of a logarithm — and the square root flattens even that. Doubling the candidate count adds a fixed amount to ln n and then takes a square root of the result, which is close to no movement at all. For n = 400, ln n is a little under six, twice that is a little under twelve, and its square root is the 3.5 quoted above; the 2.9 is what simulation returns once the idealised tail is replaced by the one this distribution actually has.

That is enough to predict what the simulation below will print, which is the point of doing it by hand first. Each candidate's mean wobbles by 0.023. The best of 400 sits roughly 2.9 of those wobbles above the truth. Multiply and the inflation is about 6.7 points, and 0.79 plus 6.7 points is 0.857.

import numpy as np

rng = np.random.default_rng(0)
per_case_sd, n_cases, n_candidates = 0.33, 200, 400
se = per_case_sd / np.sqrt(n_cases)          # 0.0233

# 20k runs of "400 identical candidates, all truly worth 0.79"
best = (0.79 + rng.normal(0, se, size=(20_000, n_candidates))).max(axis=1)
print(f"mean best-of-400: {best.mean():.3f}")   # 0.857
print(f"95th percentile:  {np.quantile(best, 0.95):.3f}")  # 0.869

A search over 400 candidates that improves nothing will still typically report about 0.857, and one run in twenty will report 0.869 or better. Against a baseline of 0.79 that is a headline gain of roughly 6.7 points conjured out of nothing except the act of taking a maximum. Two caveats pull in opposite directions and neither rescues the reported figure. Candidates within a lineage are highly correlated, so the effective number of independent draws is smaller than 400 and the inflation is smaller than the simulation suggests; against that, the search is not sampling blindly, it is actively steering towards whatever the 200 cases reward, which finds case-specific quirks far more efficiently than random sampling does. The honest reading is that some material fraction of any twelve-point gain is selection, and the reported number cannot tell you which fraction.

The same effect explains something that surprises people the first time they see it: running the optimiser for longer makes the reported score go up and the shipped quality go down. Every additional candidate is another draw, so the maximum keeps rising, while the marginal real improvement was exhausted a long time earlier. A run that reports 0.91 after 400 candidates and 0.94 after 4,000 has probably learned nothing between the two and has certainly become more optimistic.

Three numbers and what each of them estimates#

The fix is mechanical. Split the cases before the run, give the optimiser one part, and keep the other part out of reach of every component of the loop — the proposer, the minibatch sampler, the selector, and any early-stopping rule. Score the winner on it once. That produces three numbers, and being precise about what each one estimates is most of the skill here.

Biased in that last column carries its statistical sense rather than a moral one: the tuning number is wrong in a known direction, systematically too high, and repeating the run or averaging several runs would not remove the error because the error is produced by the selection itself. An unbiased estimate can still be noisy — the holdout figure certainly is — and its errors go in both directions and shrink as the set grows.

Notice that the baseline scores the same on both sets, and that this is the check that tells you the split itself is sound. A hand-written prompt was not chosen by looking at either set, so any large gap between its tuning and holdout scores means the two sets are not drawn from the same distribution — usually because the split was done by row when the cases come in correlated clusters. If your 200 cases include eight conversations from the same enterprise customer with near-identical phrasing, a row-wise split puts some of those in each side, and the holdout stops being held out in any meaningful sense.

The damage there is worth following one step further, because it is a leak that does not look like one. A prompt tuned on one of eight near-identical conversations has, in effect, been tuned on all eight. When a twin of a tuning case turns up on the holdout side, it behaves like a tuning case: it rewards the same quirk of phrasing, for the same reason, despite never having been shown to the search. The holdout's mean is then a blend of honest cases and smuggled ones, and it reports a figure partway back toward the fitted score. No error is raised and no check fails. The only symptom is the one named above — the hand-written baseline, which was tuned on nothing, scoring differently on the two sides.

import hashlib

def split_key(case) -> str:
    # split on the entity, not the row: all turns of a conversation,
    # all questions about one document, stay on the same side
    return case["conversation_id"]

def is_holdout(case, salt="4.19-2024-11", frac=0.30) -> bool:
    h = hashlib.sha256(f"{salt}:{split_key(case)}".encode()).hexdigest()
    return int(h[:8], 16) / 0xFFFFFFFF < frac

tuning  = [c for c in cases if not is_holdout(c)]   # 200 cases
holdout = [c for c in cases if is_holdout(c)]       #  86 cases

Every property anyone wants from a split is visible inside those two functions, and reading them from the inside takes a minute. split_key returns the conversation identifier rather than anything about the row, so all the turns of one conversation produce the same key. That key is hashed, which turns a meaningful string into a value bearing no relationship to anything about the case — arbitrary, but fixed, since the same key always hashes to the same digest. Taking the first eight hexadecimal characters of the digest and dividing by the largest value eight hexadecimal characters can hold converts it into a number between zero and one, spread evenly across that range because the digest itself is. Comparing that number against frac therefore sends about that fraction of the distinct conversations to the holdout — and sends every turn of a given conversation to the same side of the split, because every one of them arrived through the same key.

Hashing rather than shuffling matters for a reason specific to this workflow. Cases get added to an eval set continuously, and a shuffle-based split reassigns every existing case each time the set grows, which silently moves cases from holdout to tuning between runs and destroys the comparability of last month's numbers. A stable hash of a stable key keeps every existing case on the side it started on, and new cases land on their side independently of what already exists.

A holdout equal to baseline, and what one still cannot see#

Suppose the winner had come back at 0.80 on the holdout, exactly level with the baseline. There is a strong temptation to read this as a near-miss, keep the optimised prompt because it is "at least as good", and mention that it did better on the tuning set. That reading is wrong on both halves. The whole of the twelve-point tuning gain would have been case-specific fitting, so the search found nothing that transfers, and shipping the winner would mean taking on a longer, unreviewable prompt in exchange for zero measured quality. Ties go to the artefact a human can maintain, which is the baseline.

The more valuable conclusion in that case is about the evaluation set rather than about the prompt. A search that can move a metric twelve points on 200 cases while moving nothing on unseen cases has found exploitable structure, and that structure is information you now have for free. Read the winner and identify what it exploited. Common answers: the tuning cases share a phrasing quirk that the winner keys on; a handful of cases have wrong reference answers that a specific hedge happens to match; the grader rewards a surface feature such as answer length or the presence of a citation marker. Each of those is a defect worth fixing regardless of whether you ever run another search, and the optimiser located it in six hours for $47, which is cheaper than a human finding it.

There is a limit on how many times you get to ask, and it is the read budget of topic 3.5 rather than anything specific to optimisers: every comparison against the holdout leaks a little information into your decisions, and a set consulted after every re-run has been used for selection too, at which point it is quietly a tuning set with no successor, because nothing else was held back to replace it. The figures that make it concrete are the replacement cost. Hand-labelled cases run at roughly 5–15 minutes of a domain expert's attention each, so a 300-case set is on the order of 40 person-hours, which is why the count of holdout scorings belongs in a log rather than in somebody's memory.

A holdout also answers exactly one question: did the winner learn something about the task, or only about these particular cases. It does not tell you whether the winner learned something about the grader, because both sets are scored by the same grader. If the judge model quietly prefers confident, unhedged answers, a prompt that suppresses hedging will score better on tuning cases and on holdout cases alike, and no split will reveal it. The number will be clean and the behaviour will be worse for users. Nor does it say anything about cost, latency, prompt length, or whether a human could edit the result under pressure.

The 0.036 quoted at the close of that scenario is the same division run on a smaller set, and it is the figure that decides whether any comparison in this chapter can be acted on at all. The square root of 86 is a little over nine, and the per-case standard deviation of 0.33 divided by that is 0.036 — 3.6 points. The gap under discussion is three. When the standard error is larger than the difference it is measuring, repeating the whole exercise on a different 86 cases would routinely hand back a gap of zero, or a gap twice the size, without either prompt having changed. That arithmetic is also what prices a holdout: halving the standard error means quadrupling the cases, because the square root is in the denominator, and at 5–15 minutes of expert attention per case that multiplication is the whole cost of the set.

What the split buys is a number that means something, and it costs almost nothing: a stable hash, a salt in the run config, and the discipline of scoring once. What it does not buy is a decision. A three-point gap on 86 cases with a standard error of 3.6 points is not a result you can act on, and no amount of care in constructing the split makes it one.

Whether that gap is a finding or a coin flip is not something one run can answer. The way to find out is to run the search again.