Every item on this list is the right design for some corpus

A design checklist for a retrieval system reads as a set of prohibitions: chunks no smaller than some size, a reranker in the path, a lexical arm beside the dense one, only curated sources ingested. Each item on such a list was written down because it fixed a real failure on a real corpus, and none of them is a property of retrieval systems in general. A chunk size is correct or incorrect relative to how long the passages that answer questions actually are, which is a fact about documents rather than about chunking, and the same holds item for item down the list.

That distinction matters because the two readings recommend opposite actions on the same evidence. Read as prohibitions, a checklist tells a team with a short-unit corpus to enlarge its chunks and adds cost while removing accuracy. Read as conditional claims, the same checklist tells that team which measurement would settle the question, and the measurement is cheaper than the change.

The condition is a property of the corpus, not a preference#

Each item on the anti-pattern list is a defect under a condition and correct under its negation, and the condition is measurable in every case. Writing them out as pairs is what turns a list of prohibitions into something a reviewer can use.

The last row is empty because it has no negation that makes it correct. The other five are claims about a corpus that could be true or false, and the measurement in the third column decides which. Shipping without evaluation is what removes your ability to answer any of the first five questions, so it is the one item on the list that is a defect unconditionally. That asymmetry is worth stating in reviews, because it changes what to insist on when a team is short of time.

The empty cells are doing something more precise than marking an exception. The first five rows each name a corpus property, and a corpus property is only knowable through a measurement taken on a query set with known answers — which is the thing the sixth row is about. So the sixth row is not a sixth condition sitting alongside the others; it is the instrument the other five conditions are read with. A team that has skipped it has not failed one of six checks, it has made the remaining five unanswerable, which is why the column that would hold its condition is blank rather than strict.

That structure is also what makes the list usable by a reviewer who does not know the domain. Each of the first five rows converts into a question with a numerical answer, and the sixth converts into a question about whether the other five answers exist at all. None of the six requires an opinion about the design.

Measuring the chunk-size condition takes an afternoon#

The two numbers that decide chunk size are the length of the spans that actually answer questions, and how often an answer needs material from more than one contiguous unit. Both come out of the labelled query set built in chapter 1, and both are cheap.

def corpus_shape(labels: list[Label], docs: dict[str, Document]) -> dict:
    """labels: gold character spans per query, as built in chapter 1."""
    span_tokens = [count_tokens(l.text) for l in labels]
    spans_per_query = Counter(l.query_id for l in labels)
    multi_span = sum(1 for q, n in spans_per_query.items() if n > 1)
    gaps = [l.max_gap_tokens for l in labels if l.n_spans > 1]
    return {
        "span_p50": quantile(span_tokens, 0.50),
        "span_p90": quantile(span_tokens, 0.90),
        "multi_span_share": multi_span / len(spans_per_query),
        "gap_p90": quantile(gaps, 0.90) if gaps else 0,
    }

Read the four outputs against each other rather than one at a time. The two quantiles describe how long a single piece of evidence is, so they answer "could one chunk hold the answer". multi_span_share counts queries needing more than one piece at all, and gap_p90 measures how far apart those pieces sit, so together they answer "would one chunk have to be enormous to hold them all". A corpus can pass the first pair and fail the second, and the failure is invisible if you only look at span lengths. The two are worth keeping apart in your head because the fixes differ: a short span with a wide gap is an argument for overlap or for retrieving neighbours, and a long span is an argument for a bigger chunk.

One detail in the code changes how the outputs are read. span_tokens is built over labels rather than over queries, so a query with three gold spans contributes three entries to the quantiles. On a corpus where nearly every query has one span that difference is nothing; on a corpus where half of them have several, the span distribution is weighted toward multi-span queries and the median describes pieces of answers more than it describes answers.

On an FAQ corpus of short question-and-answer entries that function returns a median span of 74 tokens, a 90th percentile of 148, a multi-span share of 0.06 and a gap of essentially zero, because the rare multi-span answer is two adjacent entries. A chunk of 128 to 192 tokens covers nine answers in ten in a single unit, and nothing is gained by making the unit larger.

That claim is worth pinning to the exact number, since the range straddles it. Nine answers in ten fit inside a unit at least as large as the 90th-percentile span, so it is the top of the 128-to-192 range that delivers the figure comfortably and the bottom that sits just under it. And with no overlap, a span short enough to fit in a chunk can still straddle a boundary between two chunks, so containment measured this way is a ceiling on what retrieval will actually see rather than a guarantee. What makes the ceiling safe here is the gap of essentially zero: with almost no answers spread across a distance, nothing is being cut except by bad luck at a boundary.

On a runbook corpus of multi-step procedures, the corpus from chapter 3, the same function returns a median span of 210 tokens, a 90th percentile of 890, a multi-span share of 0.55 and a 90th-percentile gap of 640 tokens. The claim those four numbers support is worth stating carefully, because two of them are about different populations. The multi-span share says half the questions need material from more than one place. The gap is computed only over those multi-span questions, so a 90th percentile of 640 tokens says that for the widest tenth of that half, the pieces sit more than six hundred tokens apart.

The load-bearing number for chunk size is the plainer one beside them. A 90th-percentile span of 890 tokens means nine answers in ten do not fit in a 128-token unit at all, and that single figure condemns the setting without any help from the other three. The same setting, 128 tokens with no overlap, is correct on the first corpus and a defect on the second, and the difference is four numbers that took an afternoon to compute.

The k condition works the same way and is easier to get wrong, because the deciding factor is downstream of retrieval, in whatever consumes the candidates — the reader, in the language of the table above, which may be a single generation prompt or something else entirely. A pipeline where the retrieved set is fed to a cheap extraction model that runs once per candidate and votes does not care about context precision at all; twenty candidates with three relevant is fine, and adding a reranker costs latency to improve a number that does not affect the output. A pipeline where all twenty go into one prompt does care, for the reasons chapter 2 established. Before defending k=20, look at what consumes the candidates.

All three conditions can hold at once, which is the case people find hardest to believe. Consider an assistant over six months of structured log lines and alert descriptions, where each unit is a single event of 30 to 60 tokens, complete in itself, and the questions are of the form "have we seen this failure signature before, and what was done about it". The retrieval unit is the log line, so 128 tokens with no overlap is generous rather than aggressive. Twenty candidates go to a cheap long-context reader that summarises recurring patterns across them, so a candidate set that is half noise costs a few hundred tokens and changes the summary very little, while a reranker that trimmed it to five would throw away the repetition that makes a pattern visible. If the alert text is prose written by humans and the identifiers are already structured fields that queries filter on rather than search, dense-only is right too. That system runs 128-token chunks, k=20 and no reranker, and every one of those is the correct decision for its corpus and its workload. What would make it a defect is a team that arrived at the same configuration by copying a tutorial and never checked any of it.

Those three settings are not independent of one another, and the log-line case shows why the list has to be read as a whole. Small chunks raise the number of units a given answer is spread over, which is an argument for a larger k; a large k dilutes the prompt, which is an argument for a reranker. Two items on the list can therefore cancel, and a configuration that trips three of them at once may be coherent while a configuration that trips one of them is not. That is the strongest reason a checklist scored item by item gives the wrong answer: the items were never orthogonal, and a score computed as though they were is measuring something the design does not have.

The arithmetic behind those five rows#

The entry counts in that scenario are worth checking, because the arithmetic is not the arithmetic most readers do in their heads. Dividing 512 tokens by an average entry of 90 gives close to six entries, not the four or five the paragraph claims. The paragraph is right and the division is the wrong sum: a chunk boundary falls wherever 512 tokens run out rather than at an entry boundary, so a window holding close to six entries' worth of text typically holds the tail of one, four or five whole ones, and the head of another. Whole entries is what "glued together" is counting. The 20% share of a chunk given to the query's own topic is the same figure read the other way, since one entry in five is a fifth of the chunk, and 20 and 80 accounting for the whole chunk says the chunk is entries and nothing else — no headings, no boilerplate, no padding.

Two of the five rows in that table close on prose elsewhere in the chapter, and the other three carry residuals worth naming. Accuracy falling from 0.88 to 0.71 is exactly the seventeen points the last paragraph of the scenario claims, and the cost rising from $340 to $1,190 is the "tripled" of the same sentence with a little to spare. The residuals are in the other three. Tokens per query rose by more than five times while the chunk size rose by exactly four, so something beyond chunk size is adding context — which is what three simultaneous changes look like from the outside, and it is precisely the difficulty the next chapter takes up. Latency rose by two hundred and seventy milliseconds, of which the previous chapter's budget for a fused lexical arm accounts for a few tens; the bulk is the cross-encoder, so the most expensive change by latency is also the one the scenario says made accuracy slightly worse.

The precision row carries the subtlest residual of the three. A chunk that went from one entry to five dilutes the relevant share of retrieved text by about five times, while context precision fell only to a little over half its former value. Those two are not in conflict: a precision measured per chunk still counts a five-entry chunk as relevant when the answer is somewhere inside it, so it registers only the chunks that stopped being relevant at all. The thirty-five-point fall is therefore a floor on the dilution rather than a measurement of it, and a reader who expects the two to match has assumed a denominator the metric does not use.

Every reading in this section was available before the change was made. The entry lengths, the span distribution and the query log all existed, and running corpus_shape over the labelled set would have returned the four numbers this chapter quotes for an FAQ corpus: a 90th-percentile span of 148 tokens and a multi-span share of 0.06. Those say the answering unit is short and stands alone, which is the first row's condition, and they say nothing that recommends 512 — a size that puts four or five entries into a unit where the answer is one of them. The FAQ team could have known in advance.

The base rate is still against you#

None of this means the conditions are commonly met. Most internal corpora are wikis, and wikis are full of multi-step procedures, superseded pages and documents whose sections depend on each other, so the runbook shape is the usual one and the FAQ shape is the exception. Most query logs from technical users contain identifiers. Most shared drives contain three copies of the expenses policy. If you had to bet without measuring, betting on the checklist would win more often than it lost, which is exactly why it hardened into a list of prohibitions in the first place.

The reason not to bet is that the losses are asymmetric. Following the checklist on a corpus that meets the conditions costs what it cost the FAQ team: seventeen points of accuracy, an extra service, and three times the bill, for changes that could not have helped. Measuring first costs an afternoon of computing span distributions and an hour of regex over query logs. The measurement is cheap enough that "we did not have time" is rarely the real reason it was skipped.

There is also a failure mode in the other direction that this chapter can produce if read carelessly. A team that has absorbed the idea that everything is conditional will write "our corpus is like an FAQ, so 128-token chunks are appropriate" in a design document and consider the matter handled. That sentence is an assertion, and it is the same assertion that produced every failure in the preceding chapters, just with the conclusion the reader now prefers. The corpus either has a 90th-percentile answer span under the chunk size or it does not, and the number takes an afternoon to produce.

That failure mode has a quantity behind it that stays perfectly still while the thing it is supposed to track moves. A design scored against the checklist gets the same score on the FAQ corpus and on the runbook corpus, because the checklist scores the settings and the settings did not change — only the corpus did. A team watching that score sees nothing happen as their corpus fills with superseded pages or their query log fills with error codes, and reads the steadiness as safety. It is the same defence-that-does-not-defend as believing a green pipeline means a correct index: the instrument is stable because it is not measuring the variable.

Six questions with checkable answers#

Used as a set of prohibitions, the list blocks designs that are correct and approves designs that are wrong in ways it does not name. Used as a set of claims requiring evidence, each item becomes a question with a checkable answer, and a reviewer who cannot evaluate the design's domain can still evaluate whether the evidence exists.

The six questions to ask of someone else's design are what the middle column of that table implies. What is the distribution of answer-span lengths in your corpus, and how does it compare to your chunk size? What consumes the candidate set, and does context precision affect its output? What is the score spread across your top k, and does rank correlate with relevance at that depth? What share of your query log contains identifiers, and how do those queries perform? How many canonical-id groups in your ingest scope have more than one member? And what query set will you measure all of this on, before launch rather than after?

Those six questions and the six rows of the table are the same list, one row to one question, which is the check that nothing has been quietly added or dropped between the two. Five of them ask for a number about a corpus. The sixth asks whether the apparatus that produces such numbers exists, and it is the only one whose absence makes the other five answers unavailable rather than merely missing.

That last question is the one with no acceptable substitute, and it is also the one where the answer is most often a plan rather than a number. Which raises a problem the next chapter has to solve: a team that does have an evaluation set, and does change two things at once, still needs to know what to report so that the change can be judged. Removing a reranker's absence and cutting k are two interventions, and a single number will endorse the pair whichever way they actually moved.