Chunk size and k interact, so one-at-a-time lands on a path optimum

Standing up a retrieval-augmented generation pipeline — RAG for short, meaning that instead of answering from whatever the language model remembers, the system searches a body of documents for passages that look relevant to the question and pastes them into the model's prompt, so the answer is written from text the model has just been shown — forces a series of choices that no theory settles. The documents are cut into retrievable pieces first, which is chunking, where the chunk size is how many tokens go into one piece and the overlap is how many tokens each piece repeats from its predecessor so that a fact straddling a boundary survives intact somewhere. Each piece is stored as an embedding, the vector of numbers a model computes for a span of text, and searching means finding the pieces whose vectors sit nearest the question's. k is how many of those come back.

Two more stages sit downstream. A reranker, a second and more expensive model that scores a passage against the query directly, reads the top rerank depth candidates and reorders them, and the best k of that reordering are what reach the generator. A query transform, often just a rewrite, is an optional step that rephrases the question before any of this begins, turning "why is it slow" into wording that a document about request latency would actually contain. That is five knobs, and none of the five has a value anybody can derive. The right chunk size is a fact about how a particular set of documents is laid out and the right k a fact about how much evidence the generator needs before it starts filling gaps on its own, and both move when the documents or the mix of questions move.

So teams settle them by measurement. A parameter sweep, or a systematic pipeline parameter search, is that measurement made deliberate: choose candidate values for each knob, run the pipeline once per combination, score every run against the same set of labelled questions, and pick from the scores. You meet one the first time quality plateaus and somebody suggests trying 512-token chunks, and again when a proposal for several hundred trials arrives for your approval. What follows is one support-assistant team's sweep over exactly those five knobs, and the reason it is worth a chapter is that every individual step in it was sound and the endpoint was not.

Their document set is 210,000 internal support and engineering documents. Their baseline pipeline chunks at 1,024 tokens with 128 tokens of overlap, retrieves k=5, reranks the top 25, and does not rewrite the query. On the 300-query golden set it scores nDCG@10 of 0.512. They then tuned it the way a careful person tunes anything with five knobs. Hold everything fixed, sweep chunk size across {256, 384, 512, 768, 1024}; 384 wins at 0.547. Keep 384, sweep k across {3, 5, 10, 20}; k=10 wins at 0.560. Keep both, sweep rerank depth across {25, 50, 100}; 50 wins at 0.566. Keep all three, turn the query rewrite on; 0.571. Four sweeps, four improvements, no step where the metric went backwards.

Later somebody ran the full grid — all 360 combinations of the five axes — and the best cell scored 0.604 at chunk 768, overlap 128, k=20, rerank depth 100, rewrite on. The gap between the disciplined sequence and the grid is 3.3 points, which is larger than three of the four gains that produced the sequence in the first place. Nothing in the sequence was done wrong. The sequence searched a path through the space, and the path went the wrong way at the first step.

The first sweep asked the question at the wrong k#

Here is the slice of the grid that explains it: nDCG@10 with overlap fixed at 128, rerank depth at 50 and the rewrite off, varying only chunk size and k.

Read down the k=5 column, which is the column the team's first sweep actually measured. 384 is the clear winner at 0.547, and 768 is the worst value in the column at 0.505 — the single setting that sweep had the most reason to discard. Now read down the k=20 column and the ordering has inverted. 768 is the best value there, 2.2 points ahead of 512, which is itself the chunk size that wins the k=10 column; and 384, the sweep's pick, has fallen to fourth. The argmax over chunk size, meaning the value that scores highest, is not a property of chunk size. It is a property of chunk size at the k you happened to be holding fixed, and the team held k at the baseline value of 5 while deciding which chunk size to carry forward for the rest of the study.

The quantity the two knobs share#

The reason those two columns disagree is that chunk size and k are not two independent questions. They meet in a quantity neither of them names, and the quantity is worth deriving rather than asserting, because the derivation is what tells you which other pairs of knobs will behave the same way.

A retrieved passage arrives at the generator as a block of text about the size the chunker was told to produce. k of them arrive. The text the generator has to work from is therefore the sum of k blocks of about that size each, and a sum of k terms that are all the same size is that size multiplied by k. Call it the evidence budget. The step from a sum to a product is what needs justifying, and what justifies it here is that the chunker targets a fixed size, so the terms really are close to equal; where a chunker produces ragged output — cutting at headings, say, so pieces vary in length — the same argument goes through with the average piece size in place of the target, and the product is then an average rather than an exact total.

Putting the baseline's own numbers through that product gives two instances of it. At 1,024-token chunks and k=5 the model sees roughly 5,100 tokens of retrieved text; at 384 tokens and k=5 it sees roughly 1,900. Neither figure is the argument. The argument is the product, and what the product implies is a whole family of statements the two computed cases do not reach. Fix any budget and ask which pairs of settings achieve it: every pair whose product is that budget will do, so the settings holding the budget constant lie on a curve where one knob falls as the other rises. Halving the chunk size halves the evidence per passage, so k has to double to stay on that curve; quarter the chunk size and k has to quadruple. Read the other way round, that is why a k which was generous before a chunking change becomes stingy after it, without anybody having touched k.

That curve is not parallel to either axis, and that geometric fact is the whole of the problem with sweeping one axis at a time. A sweep moves along a line parallel to an axis, holding the other knob still. The structure it is trying to find — the band of settings where the generator gets enough evidence without drowning in it — runs diagonally across those lines rather than along them. So a sweep crosses the structure instead of following it, and where it crosses is decided entirely by where the other knob was pinned.

The mechanism reads back out of the table cleanly. When the team's first sweep punished 768-token chunks, it was partly measuring the fact that five long chunks retrieved at low precision put a lot of irrelevant text in front of the reranker; when the grid rewarded them at k=20, it was measuring that twenty long chunks with a deep rerank recover the precision and keep the extra evidence. Both measurements are correct. Only one of them was taken.

What separability would have bought, and how to test for it#

The sequence is not careless, and the assumption underneath it can be written down exactly, which is the first step to checking it. Suppose the score split into a part that depends only on chunk size and a part that depends only on k, the two added together. Then the difference between two chunk sizes at a fixed k is the difference between their two chunk-only parts, because the k-only part is identical on both sides and subtracts away. That difference is therefore the same number at every k. The columns of the table would all agree on the ordering, the best chunk size would be the same in every one of them, and a sweep run at any k whatever would find it.

Under that assumption the sequence is not merely defensible, it is exact, and it costs a twentieth as much. Which is why the belief deserves stating in its strongest form rather than dismissing. Chunk size governs what is inside a passage; k governs how many passages you get. Those really are different questions about different stages of the pipeline. The claim that the two effects add rather than interact is a claim about the system, though, and it is the kind of claim that is cheap to check and expensive to assume.

Checking it takes four evaluations, and four is enough because the assumption predicts an exact zero. Pick two chunk sizes and two values of k and evaluate all four corners of that box. Form the gap between the two chunk sizes at the low k, then the same gap at the high k, then subtract one gap from the other. If the score is a chunk-only part plus a k-only part, every term in that expression cancels: each chunk-only part appears once with a plus sign and once with a minus, and so does each k-only part. The difference of the two gaps is identically zero, for every choice of corners, whenever the assumption holds. Anything other than zero is a measurement of how far from true it is, and no sequence of single-axis sweeps can ever produce that number, because a single sweep holds the other knob still and so only ever yields one of the two gaps. In practice the reading to take is whether the gap changes sign or magnitude as k moves, since a change in either direction is a non-zero contrast.

Measure the corners of the chunk × k box, at (256, k=3) and (256, k=20) and (1024, k=3) and (1024, k=20). In this pipeline the gap at k=3 is +2.8 points in favour of the small chunk and at k=20 is −2.8 in favour of the large one, which is an interaction visible from four numbers.

Those two gaps are equal in size and opposite in sign, and the symmetry forces something worth pausing on. A quantity that is positive at one end of a range and negative at the other passes through zero somewhere in between, and the table puts the crossing between k=10 and k=20. There is therefore a value of k at which these two chunk sizes — the extremes of the range, four times apart — score identically. A chunk-size sweep run at that k would report the axis as flat and conclude that chunk size does not matter here. The failure is not only that a sweep can pick the wrong winner. At one particular setting of the other knob it reports no winner at all, out of data containing an effect as large as any in the table. That is what licenses the flat statement above: the argmax over chunk size is a property of the pair, and reporting it as a property of chunk size discards the half of it that decides the answer.

Why a one-at-a-time search stops where it does#

The same quantity that measures the interaction also decides where the search halts, which is worth showing because it means one check answers both questions.

Take any cell and consider two candidate steps out of it, one along the chunk axis and one along the k axis. Write A for the change in score from the chunk step taken alone, B for the change from the k step taken alone, and C for the change from taking both. Then C equals A plus B plus the interaction contrast for that box — the same difference of gaps built above. That is an identity, not an approximation, and expanding it shows why: the score at the far corner appears once, the score at the starting cell appears once with the opposite sign, and each of the two intermediate cells appears once with a plus and once with a minus, so both cancel completely. Nothing is left over. There is no fifth term.

A search that moves one axis at a time never computes C. It computes A, computes B, and accepts whichever of them is positive. When the interaction contrast is zero that costs nothing at all, because C is then exactly A plus B, and a joint step is worth taking only when at least one single step already was. When the contrast is large and positive, A and B can both be negative while C is positive, and no amount of single-axis searching will reach it. The search halts at a point where every available single move looks like a loss and reports that point as a maximum. What it has actually established is narrower: the improving direction out of that point is not parallel to any axis.

The table contains a worked case of exactly this. Start at 512-token chunks with k=10, which scores 0.566 and is the best cell in its column. Moving to 768 at that k costs about a point and a half. Moving to k=20 at 512 tokens gains a little over half a point. Taking both moves together reaches 0.594, nearly three points up from the start, which is more than the two single moves add up to by a margin larger than either of them individually. A chunk-size sweep run at k=10 rejects 768 on the evidence in front of it, correctly, and having rejected it never returns, so the cell holding the best score in the entire slice becomes unreachable from that point onwards.

This is the specific failure mode of coordinate descent on a surface with interaction: it converges to a point where no single-axis move improves the metric, and that point can sit a long way below the maximum. Every step in the log improved things, which is what coordinate descent guarantees and also all it guarantees. The team's log is weaker still than that guarantee, and the difference matters when you read one. Coordinate descent means cycling the axes until nothing moves; a single pass through the axes in a fixed order stops after one lap, so its endpoint is not even certified as a point where no single move helps. It is certified only as no worse than where it started.

Underneath all of this sits a decision that nobody in the story wrote down as a decision. The requirement was to find a good configuration for these documents and these questions. Searching one axis at a time is not that requirement; it is a design decision taken to keep the number of runs small, and it is a reasonable one often enough that it gets made silently. Two further decisions ride along with it, equally unrecorded. One is the order of the axes, which determines the answer. The other is what the axes not currently being swept are held at, and the team held them at baseline values, so every early sweep was scored against a pipeline nobody intended to ship.

Which of the five axes actually interact#

Five parameters give ten pairs, and treating all ten as equally entangled would be as wrong as treating none of them that way. The pairs differ, and the differences have mechanisms you can reason about before you spend anything.

The mechanism column is doing the real work, and the pattern in it is the one derived above. Every strong pair is strong because two knobs act through a single shared quantity: a token budget, a redundancy fraction, a selectivity ratio. Every weak pair is weak because there is no such quantity to share. That is a test you can apply to a sixth axis before you have measured anything — ask what quantity it moves that another axis also moves, and if there is no honest answer, the pair is probably safe to sequence.

Three strong pairs out of ten is enough to break a sequential search and not enough to make the full grid the only defensible design. What it tells you is where a sequence is allowed to be a sequence. Sweeping the query rewrite last, on its own, is close to harmless as long as you know the number you get is the gain at your final k and not a general property of the rewrite. Sweeping chunk size on its own is not, because the ordering it produces is conditional on two other axes.

The practical middle is to block the interacting axes and sequence the rest: run chunk size × k as a joint grid, since that is 20 cells rather than 9 measurements, then hold that pair and run overlap and rerank depth as a smaller joint grid inside it, then the rewrite.

What the sweep code actually looks like#

The difference between the two designs is smaller in code than it is in outcome, which is part of why the sequential version is so easy to reach for.

from dataclasses import dataclass, replace
from itertools import product

@dataclass(frozen=True)
class Config:
    chunk_tokens: int
    overlap_tokens: int
    k: int
    rerank_depth: int
    rewrite: bool

AXES = {
    "chunk_tokens": [256, 384, 512, 768, 1024],
    "overlap_tokens": [0, 64, 128],
    "k": [3, 5, 10, 20],
    "rerank_depth": [25, 50, 100],
    "rewrite": [False, True],
}

def one_at_a_time(base: Config, evaluate) -> Config:
    current = base
    for axis, values in AXES.items():          # order matters, and that is the problem
        trials = [replace(current, **{axis: v}) for v in values]
        current = max(trials, key=evaluate)
    return current                             # 5 + 3 + 4 + 3 + 2 = 17 evaluations

def full_grid(evaluate) -> Config:
    cells = (Config(*values) for values in product(*AXES.values()))
    return max(cells, key=evaluate)            # 5 * 3 * 4 * 3 * 2 = 360 evaluations

Two things are worth noticing in one_at_a_time. The loop over AXES fixes an order, and the result depends on that order in a way nothing in the function signature admits; starting from k instead of chunk size gives a different answer from the same code and the same data. And replace(current, ...) carries every earlier decision forward, so by the time the loop reaches rerank_depth it is measuring rerank depth on exactly one chunking configuration out of fifteen.

The two comment lines are also a small audit worth reading as one. The additive count and the multiplicative count are built from the same five level-counts, and each reproduces its search size exactly, with nothing unexplained on either side. That closure is what lets you treat the cost comparison as complete: there is no sixth axis hiding in the budget, and the entire difference between seventeen runs and 360 is accounted for by five declared axes at their declared levels.

The blocked version is the same shape with tuples instead of scalars:

BLOCKS = [
    ["chunk_tokens", "k"],          # 20 cells, jointly
    ["overlap_tokens", "rerank_depth"],  # 9 cells, inside the winner
    ["rewrite"],                    # 2 cells
]

def blocked(base: Config, evaluate) -> Config:
    current = base
    for block in BLOCKS:
        grids = [dict(zip(block, combo)) for combo in product(*(AXES[a] for a in block))]
        current = max((replace(current, **g) for g in grids), key=evaluate)
    return current                  # 20 + 9 + 2 = 31 evaluations

Thirty-one evaluations against 360 is the trade being made, and it is a trade against a specific assumption: that the pairs left un-blocked are weak enough that the ordering they produce will not invert. That assumption is checkable, which is the point of the interaction table above, and it is not free.

Screening passes, random search and Bayesian optimisation#

None of this establishes that the full grid is the right search. The grid is exponential in the number of axes, so adding a sixth parameter with four levels makes it 1,440 cells, and the sixth parameter is usually the one somebody wants to add. Random search over the same box frequently finds a comparable cell in a fraction of the trials, precisely because most axes are close to flat and a grid spends its budget resolving flat directions in full detail. Bayesian optimisation, which fits a running model of how the metric responds to the parameters and spends each new trial where that model is most uncertain, and the automated frameworks built on it — AutoRAG is the clearest published example of the genre, declaring a module search space in YAML and benchmarking the combinations — do better still on large spaces. The argument here is narrower: a sequential search cannot find an interaction, and three of these five axes interact through mechanisms you can write down, so the sequence has a known blind spot rather than a probabilistic one.

Nor is one-at-a-time tuning a mistake in general. As a screening pass it is excellent. Seventeen evaluations will tell you which axes move the metric at all, and if overlap turns out to be flat across its whole range at the baseline configuration, dropping it from the grid takes 360 cells down to 120 for the price of three trials. The error is not running the sequence; it is reporting its endpoint as the best configuration rather than as the best configuration reachable from that starting point in that axis order. Those two sentences describe the same seventeen runs and differ only in what is claimed from them, which is why the fix costs nothing and is so rarely applied.

Whether the gaps in the table are real#

Every argument above reads differences off a table of estimates, and estimates carry noise. The gap between 0.560 and 0.566 in the k column may be nothing at all. Reading an argmax off a column of noisy estimates is a separate hazard from reading it off the wrong column — the first is a measurement problem and the second is a design problem, and fixing one does nothing for the other. A team that runs the full grid and reports its best cell has solved the design problem and walked straight into the measurement one, since the maximum of many noisy estimates is biased upward before anybody has done anything wrong. Chapter 4 takes that apart properly.

The two problems do interact in one direction worth flagging. A large interaction contrast is easier to trust than a small single-axis difference, because it is built from four measurements rather than two and because the effect it reports here is several times the size of the gaps that might be noise. The four-corner check is on firmer ground than the argmax it is meant to warn you about, and that is a happy accident of this particular table rather than a general law.

What the whole argument amounts to is a restriction on what a sweep result may claim. A number obtained by holding four knobs still and moving the fifth is a statement about the fifth knob conditional on where the other four were parked, and the conditioning clause is not decoration. Where two knobs act on the metric through a shared quantity, dropping that clause turns a true finding into a false one, and the table above shows it flipping sign across the range of a single other axis. The repair is not the full grid. It is to identify the pairs that share a quantity, block those, sequence everything else, and report each number with the configuration it was measured in.

The immediate objection to the grid is not statistical, though. It is that 360 trials sounds like 360 times the work of one, and on that reading nobody would ever approve it. That reading is wrong by roughly an order of magnitude, and the reason is that the five axes are not merely unequal in how much they interact — they are unequal in what they cost to change. Two of them force you to rebuild the index. The other three do not, and the difference between those two groups is what actually sets the bill.