Systematic pipeline parameter search

Read these first

• RAG-specific metrics
• Held-out sets and eval-set overfitting

What you will know

• Run a parameter sweep across chunking, k, and rerank depth

  Include overlap and query transformations, and preserve interaction effects that one-variable-at-a-time tests will miss.

• Track each configuration as a versioned artifact

  Record code, corpus and index versions, embedding and reranker models, seeds, and evaluation data so a winning run can be reconstructed.

• State the cost of the sweep before running it

  Estimate trial count, indexing and inference spend, storage, wall-clock time, and production latency before committing resources.

• Avoid selecting a configuration that only wins on the visible set

  Use leakage-safe held-out documents or time slices, because repeated comparison against one benchmark turns tuning into benchmark memorization.

• Report a result as a configuration, not a vibe

  Publish the selected values alongside metric deltas, variance or confidence intervals, evaluation-set boundaries, and the baseline it beat.

Chapters

1. 1

   Chunk size and k interact, so one-at-a-time lands on a path optimum

   Sweep chunk size first, keep the winner, then sweep k, then rerank depth: every step improves the metric and the endpoint can still be poor. Halving chunk size halves the evidence per passage, so the k that was right before is now too small, and every later sweep searched one slice of the space. This chapter identifies which of the five axes genuinely interact and what a grid buys you over a sequence.

   Read chapter
2. 2

   Only the chunking axes force a reindex, and they set the real bill

   Reading 5 x 3 x 4 x 3 x 2 as 360 equally expensive trials misprices the sweep. Chunk size and overlap are the only axes that force the corpus to be re-embedded, so the true cost is fifteen index builds, inside each of which the remaining twenty-four combinations run for the price of retrieval and judging. You will learn to cost a proposal by reindex count, judge spend, wall clock and resulting production latency before approving it.

   Read chapter
3. 3

   A run you cannot rebuild recorded a number, not a configuration

   Three months on, a YAML file saying chunk_size 512, overlap 64, k 8, rerank_depth 50 and plus seven nDCG rebuilds to plus two. The swept parameters were recorded; everything else that shaped the result was not. Corpus snapshot, index build, embedding and reranker versions hiding behind stable provider aliases, generator decoding settings, seeds and the exact evaluation split all drifted. Here you define the artifact that makes a winning run reconstructible.

   Read chapter
4. 4

   An identical rerun moves the metric, and the best of 360 moves further

   Rebuild a configuration exactly and the number still changes: judge nondeterminism, sampling, index-build variation. Measure that noise floor first, because it is the scale every delta is read against, and get the interval on a delta from a paired comparison over the same queries with a bootstrap over the query set, never from two independently computed scores. Then the harder fact: the maximum of 360 noisy estimates is biased upward by construction, so the winner of a one-week sweep carries the same optimism as a golden set selected against for six months and has to be re-measured on data that took no part in the selection. This is also what separates a genuinely flat response surface from a 500-query evaluation that was never powered to resolve 1.5 points.

   Read chapter
5. 5

   Selecting forty times against one golden set makes it the objective

   Forty configurations compared on the same 300 queries yields a champion at plus nine on that set and plus one on freshly labelled production traffic. Keeping whatever wins, round after round, is a fitting procedure, and the golden set stops measuring quality and starts defining it. Covers the evidence that distinguishes this from an unrepresentative production sample, and why merging the two sets destroys the only clean measurement the team has.

   Read chapter
6. 6

   Splitting queries but not the corpus leaves the leak in place

   An 80/20 split of 300 golden queries, selecting on 240 and checking the winner once on 60, feels disciplined and often is not. Chunking and index settings were tuned over documents that also answer the held-out questions, so a configuration fitted to those documents carries straight across the split. A leakage-safe design holds out documents or a time slice, and this chapter shows what that costs you in label effort.

   Read chapter
7. 7

   A reportable result carries the values, the baseline and the interval

   A document reading only that a sweep improved quality by 6% is not adoptable: no parameter values, no baseline configuration, no named metric, no confidence interval, no evaluation split. This chapter fixes what a sweep report must publish, then reads the awkward case where the best of 360 trials wins by 1.5 points with overlapping intervals. A flat response surface is a genuine finding that frees you to choose on latency and cost — provided you have first checked the evaluation was powered to resolve a gap that size at all.

   Read chapter