Splitting queries but not the corpus leaves the leak in place

A retrieval sweep is scored against a golden set: a fixed list of queries, each one carrying the passages an annotator has marked as answering it. Scoring a configuration means running every query through it and asking how high those marked passages come back. The metric used throughout this topic is nDCG@10 — it rewards a configuration for putting the marked passages near the top of its first ten results, and rewards it more the higher they sit. The settings being searched are chunk size, chunk overlap, k, rerank depth and whether a query is rewritten before it is sent. Behind all of them sits a corpus, meaning the entire collection of documents that have been cut into chunks, embedded and stored in an index, and every score any configuration earns is produced by those settings acting on that corpus while those queries are asked.

The standard discipline is to hold some of the queries back. Shuffle the golden set with a fixed seed, tune on most of it, seal the remainder, and score the winning configuration exactly once against the sealed part, so that the number which gets reported is not the number that was maximised. That discipline is imported from supervised learning, where it is correct and where it works for a reason that can be stated precisely. The reason does not survive the journey, because the object being fitted in a chunking sweep is not a list of questions.

What a chunk size is fitted to#

A chunk size is a claim about document layout. Setting 384 tokens with 64 tokens of overlap says that the meaningful units of these documents are roughly 384 tokens long and that a 64-token window is enough to catch a fact straddling a boundary. When a sweep prefers 384 over 768, it has discovered something about the average section length, heading density and table formatting of the documents that answer the evaluation queries.

Calling that "fitting" is not a metaphor, and the steps behind it are worth laying out one at a time, because each of them is checkable and the conclusion is not obvious once they are skipped. A score is a function of three arguments: the settings, the queries and the documents. A sweep varies the first argument and maximises over it while the other two are held fixed. Any property of a fixed argument that happens to raise the score is therefore baked into whichever setting wins, in exactly the way a training row is baked into a model's weights — the optimiser cannot tell the difference between a setting that is right in general and a setting that suits the particular material it was scored on.

The document side of that is where the sweep's information actually comes from. Chunking runs once per document, before any query exists, so the boundaries a chunk size produces are a function of the document text alone. A query then scores well or badly depending on whether the passage an annotator marked survives those boundaries as one retrievable unit rather than being cut in half or buried inside a much longer block of unrelated text. Whether it survives is decided by that document's section lengths measured against the chunk size. So the value of chunk size that wins a sweep is the value best matched to the section lengths of whichever documents happen to carry the gold passages, and nothing about the wording of the queries enters that chain at any point.

Those documents are the same on both sides of a query split. Of the 60 held-out queries in this team's split, 41 had at least one gold passage — a passage an annotator marked as answering that query, which is what the metric scores the retrieval against — in a document that also served one of the 240 selection queries; the popular runbooks answer many questions each. Widen it from documents to document families, meaning pages generated from the same template or maintained by the same team with the same house structure, and the figure is 58 of 60. A chunk size fitted to how the platform team writes runbooks transfers perfectly to a held-out question about a different platform runbook, because the layout it was fitted to is the layout it is being tested on.

The right-hand side of that diagram is where the generalisation actually fails. Production queries hit the API reference pages and the incident write-ups far more often than the golden set does — the golden set was written by four engineers who reached for the documents they knew — and a chunk size fitted to 300-token runbook sections is a poor fit for 90-token API entries, where it merges eight unrelated parameters into one passage, and for 1,200-token incident narratives, where it slices a causal chain into four pieces that each look irrelevant on their own.

That mismatch is measurable without any new labelling. Classify the gold documents of the golden set by source system and compare that mix against the mix of documents retrieved for a month of production queries: the golden set is 64% runbooks, 21% architecture pages and 9% API reference, while production retrieval lands on 28% runbooks, 17% architecture and 44% API reference. Read those as two distributions rather than as six separate numbers and the useful thing is where each one has its mass. The golden set's mass sits on runbooks, production's sits on API reference, the two peaks are on different rows, and a score computed over the golden mix is therefore a statement about runbooks that says almost nothing about the largest slice of production retrieval — again not weak evidence, but no evidence, since the comparison contains barely any API reference queries to be evidence from.

Two honest qualifications belong with those percentages. Neither column sums to a hundred: a few percent of the golden set and around a tenth of production retrieval fall into source systems the breakdown does not name, so this decomposition has a residual and cannot be read cell by cell as though it were complete. That remainder is far too small to explain a gap running from 9% to 44%, and it is there all the same. The second qualification is that the two columns come from different instruments — the golden column counts documents an annotator marked, the production column counts documents the current configuration actually retrieved — so the production column is partly a readout of one pipeline's behaviour rather than of what users are asking for. The cleaner version of the measurement classifies production queries by the source system that should answer them, and a team leaning hard on this comparison should do that first.

The two worlds a held-out score cannot separate#

The belief that has to be named here is not a slip in the arithmetic. It is the belief that a defence works: we held out a fifth of the queries, the winner scored the same on them, therefore the result generalises. The reason an 80/20 split feels rigorous is that it is rigorous somewhere else, and the somewhere else is worth stating precisely rather than waved at. In supervised learning the model has millions of parameters, the training data is the thing being fitted, and removing rows from it genuinely severs the path by which memorisation of a row could inflate the score. Anybody who has watched a held-out score collapse a model's apparent accuracy has seen that mechanism work, and the shape of the ritual is identical here.

What makes it wrong is that the confirmation is comparing a quantity against a copy of itself. The property that decided the winner is the layout mix of the answering documents; both the selection half and the held-out half are supplied that property by the same corpus, unchanged. The split varies the query argument of the score function. The fitting consumed the document argument. A check that varies one input while the dangerous input is held constant will return a clean reading whatever the state of the dangerous input, which is the general form of a defence that cannot fail and therefore cannot inform.

Written out as two worlds, the emptiness of the reading becomes concrete. Suppose first that 384 with 64 tokens of overlap is genuinely the right setting for technical documentation in general; then the held-out 60 agree with the 240 and the setting is right for them too. Suppose instead that 384 is fitted to nothing more than the section lengths of these particular runbooks and transfers to no other material; then the held-out 60 still agree with the 240 — the same runbooks answer them either way. Both worlds produce +4.2 next to +3.9, so observing that pair rules neither out. An observation identical in both worlds is not weak evidence about which world you are in — it is no evidence at all, and treating it as confirmation is where the write-up goes wrong.

Separating the two worlds requires varying the document argument, and no partition of a query list can do that. This is a different defect from the one in chapter 5 — where a golden set selected against forty times gradually became the objective. There the queries had been optimised against too many times; here the queries were drawn from the wrong part of the corpus in the first place. A team can have both at once, which this one did, and the two failures need different remedies.

What holding out queries does block#

A random query split is a real instrument aimed at a real leak, and dismissing it wholesale is the opposite error. Of the settings in this sweep, one acts on the query text: the rewrite step, which reformulates the user's wording before retrieval. A rewrite prompt tuned against the exact phrasings in the selection set — the vocabulary, the average length, the habit of asking complete questions — is fitted to the queries themselves, and holding queries back does sever that path, because the sealed 60 carry wordings the prompt was never adjusted against. To the extent the reported delta comes from the rewrite, the held-out score is genuine evidence.

The other settings are decided before a query exists. Chunk size, overlap and the index parameters are applied at build time, so a query split cannot reach them, and their contribution to the delta is confirmed by nothing. That gives a usable test for telling a real defence from one in costume: ask which argument of the score function the split varies, then ask which argument the fitting consumed. Where they are the same argument, the split is doing work. Where they differ, the split will produce an approving number in every world and should be reported as what it is.

Measuring the leak before trusting the split#

The overlap is computable from the labels you already have, and it takes a few lines. Anything above roughly 20% of held-out queries sharing gold documents with the selection set means the split is not doing what the write-up will claim it does.

from collections import defaultdict

def gold_docs(queries) -> dict[str, set[str]]:
    return {q.id: {p.doc_id for p in q.gold_passages} for q in queries}

def leak_report(select, holdout) -> dict[str, float]:
    sel_docs = set().union(*gold_docs(select).values())
    hold = gold_docs(holdout)
    shared = [qid for qid, docs in hold.items() if docs & sel_docs]
    fam = {d: family_of(d) for d in sel_docs}
    shared_fam = [qid for qid, docs in hold.items()
                  if {family_of(d) for d in docs} & set(fam.values())]
    return {
        "holdout_queries": len(hold),
        "share_gold_doc_with_selection": len(shared) / len(hold),      # 0.683
        "share_gold_family_with_selection": len(shared_fam) / len(hold),  # 0.967
    }

The two fractions in the comments are the two counts from the paragraphs above, arrived at by a different route: 41 of the 60 held-out queries is 0.683, and 58 of 60 is 0.967. The prose counted by inspecting which documents answered which queries; the function intersects sets of document ids and never looks at a document. That the two land on the same pair of numbers is worth a moment, because it means the function is measuring the quantity the argument is about rather than something adjacent to it, and it can therefore be run on a split nobody has inspected by hand. It also says how badly this particular split fails: the document figure is more than three times the rough threshold, and the family figure is above nineteen-twentieths, so this is not a marginal call.

It is worth separating the requirement from the design decision that is usually mistaken for it. The requirement is to obtain an estimate of the winner's quality that is not inflated by whatever the winner was fitted to. Partitioning the query table is a design decision, and it is the one the supervised-learning habit supplies for free, which is why nobody stops to check whether it meets the requirement. Grouping queries by gold document is a second design, grouping by document family a stricter third, and holding out a slice of time meets the same requirement by a different mechanism entirely. Choosing among the last three is an engineering trade about cost and convenience; the requirement is not negotiable, and a team arguing about which design to adopt has at least agreed on what is being defended.

A grouped split fixes the first number and costs labels. Partitioning by gold document means every query touching a given document lands on the same side of the line, and because the distribution of queries per document is heavily skewed — 8 documents account for 71 of the 300 queries in this set — the partition does not come out at 240/60. The unit being allocated stops being a query and becomes a group of queries, group sizes here run from one to dozens, and a target ratio that is easy to hit with three hundred interchangeable items is often unreachable as a sum of a few large integers and a long tail of ones. A greedy grouped split of these 300 queries yields 212 selection and 48 held-out with 40 queries unassignable without violating disjointness, and those 40 are simply dropped.

Those three counts are worth adding up, because they close: 212 selected plus 48 held out plus 40 dropped is exactly the 300 the set started with. There is no fourth bucket, no queries counted twice and none quietly lost, so the cost of disjointness on this set is known rather than estimated — it is those 40 labelled queries, and nothing else. Getting back to a 240/60 split with document disjointness means labelling roughly 55 new queries against documents not already represented, at about 12 minutes each including passage-level relevance marking, which is six hundred and sixty minutes, or 11 hours of annotator time before the sweep can start. The 55 exceeds the 40 that were dropped, and the surplus is real rather than slack: a replacement only helps if it lands on a document neither side already holds, and freshly written queries arrive attached to documents with the same skew, so a fraction of them will land on documents already in play and count for nothing.

from sklearn.model_selection import GroupKFold

groups = [family_of(sorted(gold_docs([q])[q.id])[0]) for q in queries]
splitter = GroupKFold(n_splits=5)
for fold, (tr, te) in enumerate(splitter.split(queries, groups=groups)):
    print(fold, len(tr), len(te))   # 5 folds, no document family on both sides

Grouped k-fold, which divides the queries into five folds so that each one takes a turn as the held-out set while the other four are used for selection, and which assigns whole document families to a fold rather than individual queries, is often the better shape than a single 80/20, because it recovers the labels you would otherwise waste: every query serves as held-out exactly once, the sweep gets five estimates of the delta instead of one, and the spread across folds is a direct read on how layout-dependent the winning configuration is. The reason that spread reads as layout dependence is that each fold's held-out set is, by construction, a different mix of document families, so moving from fold to fold changes the document argument of the score function and only that argument, with query-sampling noise on top. A configuration that is +4 on four folds and −1 on the fold containing the API reference pages has told you something no single split could, because a single split returns one number and one number has no spread to read. The cost is that every trial runs five times, which multiplies the dominant judging line item from chapter 2 by five, so in practice this runs on retrieval metrics for the full grid and on the end-to-end judge for the finalists only.

The split designs, and what each one actually blocks#

Read down the middle column and the four designs form a ladder rather than a menu, because each one severs a path the row above it leaves open, and the label cost in the right-hand column is what that extra severing is bought with. The time slice is the closest thing to an honest estimate of what will happen in production, since it holds out both the questions people started asking and the documents written since, and those are the two things that actually change. It is also the least convenient, because you cannot manufacture it — waiting eight weeks is the price — and because a slice covering one month of a seasonal system carries its own bias.

One thing no design on that list can do is remove the answering documents from the index. The held-out queries must be answerable, so their gold documents have to be present when the winner is scored. What disjointness controls is which documents the tuning was allowed to see evidence about, not which documents exist. That distinction is why this is not the same problem as train/test leakage in supervised learning, where the fix is to remove rows from the training data. Here nothing is trained, and the leak runs through summary statistics of the corpus — section lengths, heading density, table formatting — that a handful of parameters have been fitted to.

Fitting to the judge, the guidelines and the query mix#

A corpus-aware split does not catch fitting to the judge. If the same judge model and prompt score both splits, and the sweep has been selecting configurations that produce answers this judge likes — longer answers with more quoted context, for instance — then every split scored by that judge inherits the bias. This is the same structure as before with a different argument held constant: the judge is supplied identically to both halves, so comparing the halves compares the judge with itself. The check for it is a human-labelled sample, periodically, on a subset small enough to afford.

It does not catch fitting to the label guidelines either. Passage-level relevance labelled by one annotator against one rubric encodes a view of what counts as answering the question, and if that view says a passage must contain the fact verbatim, then every split rewards configurations that retrieve verbatim matches over configurations that retrieve the surrounding explanation the generator would use better. That is a systematic difference between the evaluation and the product, and it is invisible to any partition of the same labels.

It also does nothing about the query mix. If the evaluation queries over-represent runbooks by a factor of two, a document-family-grouped split of those queries produces two samples that are each over-representative in the same way, and the winner is confirmed against the same distortion it was selected under. Fixing that is a sampling problem rather than a splitting problem: draw evaluation queries from production traffic in proportion to the traffic, or stratify deliberately and report per-stratum results so that a gain concentrated in 28% of traffic is visible as such.

And it does not, by itself, make a result trustworthy. A document-disjoint held-out score is still a single measurement with the noise floor and interval from chapter 4 attached, where identical reruns of one configuration establish the scale any delta has to be read against. A +3.9 on 48 held-out queries has a 95% interval of roughly ±5 points, which is wide enough to contain both a real improvement and nothing. There is also a structural trade to state rather than hide: a grouped split discards labels and leaves fewer held-out queries than a random one would, so the design that removes the bias leaves less data to measure with. Combining the two disciplines is the point: a corpus-aware split removes a bias, an interval quantifies the remaining uncertainty, and neither substitutes for the other.

The sweep is now designed, costed, recorded, measured against a noise floor and confirmed on a split that does not leak. The discipline that got it there is a single test applied to every guarantee in the design: name the input the check varies, name the input the fitting consumed, and require them to be the same input. A random query split fails that test and produces an approving number anyway, which is the most expensive kind of check to own, because it costs nothing to run and it removes the appetite to run the one that would have told you something.

What remains is the part that determines whether any of it changes anything, which is the document that goes to the people who did not run it. A result that cannot be adopted, reproduced or argued with by somebody who was not in the room is where a great deal of this work quietly ends.