Freezing a retrieval baseline

Every comparison in retrieval evaluation is between two runs, and a run is only comparable to another if the things it was scored against held still. Four of those things decide what the number means rather than merely how it was produced: which documents were searchable, which queries were asked, which documents were judged relevant to them and at what grades, and what depth the metric was cut off at.

The judgments travel as a single file of query-document grades, which the retrieval literature and most tooling call the qrels — a contraction of query relevance judgments — and the name is worth adopting because every evaluation library expects a file in that shape. The other three have no established names and every project invents its own, which is part of why they drift: a thing without a name in the repository is a thing nobody notices changing.

The four that define the measurement#

The judgments row is the one teams resist, because editing a wrong grade feels like an unambiguous improvement. It is an improvement, and it also invalidates every number computed against the old file, which is fine as long as the new file gets a new version and the old runs keep pointing at the old one. What cannot happen is a qrels file that is edited in place while historical results continue to claim comparability with runs scored against its earlier contents.

Beyond those four sits a second group that does not change what is being measured but does decide whether the run reproduces at all: the analyzer and tokenizer configuration, the BM25 parameters $k_1$ and $b$, the search library version, the chunking rules, any random seed, and the commit of the evaluation code itself. An OpenSearch minor upgrade that changes stemmer behaviour will move scores by a point or two with everything else pinned, which is enough to swamp a real improvement. That group deserves a concrete illustration rather than a warning, and the scenario below supplies one: a single minor version bump moved a frozen baseline with the corpus, the queries, the judgments and the cutoff all held still.

Storing it so somebody can re-run it in September#

{
  "baseline_id": "bm25-2024-03-01",
  "created": "2024-03-01T14:22:09Z",
  "corpus": {
    "snapshot_id": "corpus-2024-02-28",
    "documents": 3100,
    "chunks": 48122,
    "manifest_sha256": "9f2a1c...",
    "chunker": "recursive-800-100@v3"
  },
  "queries": { "set_id": "golden-v4", "n": 160, "sha256": "4bb0e7..." },
  "qrels":   { "version": "v4.1", "judgments": 6413, "sha256": "c1d904..." },
  "metrics": { "primary": "ndcg@10", "also": ["recall@10", "recall@50", "mrr"] },
  "retrieval": {
    "engine": "opensearch-2.11.1",
    "analyzer": "english_v2",
    "bm25": { "k1": 1.2, "b": 0.75 },
    "top_k_candidates": 50,
    "hybrid": false,
    "reranker": null
  },
  "code_commit": "e41c8b2",
  "results": { "ndcg@10": 0.610, "recall@10": 0.62, "recall@50": 0.81, "mrr": 0.58 }
}

Three of the numbers in that block appear elsewhere in this topic and agree with what was computed there, which is what a manifest is for. The judgment count is the roughly six and a half thousand the pooling arithmetic predicted from 160 queries at a pool depth of forty. The query count is the 160 the stratified sample produced, with its six strata summing to that total. And the results block reproduces the BM25-only row of the four-run comparison exactly — 0.61 on nDCG@10, 0.62 on recall@10, 0.81 on recall@50. A manifest whose figures disagree with the runs they came from is a manifest describing a different experiment, and checking that they agree costs the time it takes to read three lines.

The manifest is not the whole artefact. Alongside it goes the per-query result file, because a mean is not enough to support the comparisons the next chapter makes — a paired test over 160 queries needs both runs' per-query scores, and reconstructing them later means re-running against a corpus that no longer exists.

baselines/bm25-2024-03-01/
├── manifest.json          # the block above
├── queries.jsonl          # 160 queries with ids and strata
├── qrels.json             # 6,413 judgments, immutable
├── corpus.manifest        # doc_id, chunk_id, sha256 — 48,122 lines
├── run.jsonl              # per query: ranked doc ids with scores, top 50
└── per_query.csv          # query_id, ndcg@10, recall@10, recall@50, rr

$ sha256sum baselines/bm25-2024-03-01/{queries.jsonl,qrels.json,corpus.manifest}
4bb0e7...  queries.jsonl
c1d904...  qrels.json
9f2a1c...  corpus.manifest

Snapshotting a corpus is the part people implement badly, usually by writing down a date and assuming ingestion can be replayed to it. It cannot, because the sources moved: wiki pages were edited, three documents were deleted, and a CMS has no history for the ones that were overwritten. What works is content-addressing at ingest time — every raw document gets hashed on the way in and the hash is stored with the document id, so a snapshot is a list of 3,100 (doc_id, sha256) pairs plus a copy of the raw bytes in object storage. Reconstructing the March index then means fetching those exact bytes and running them through the chunker, and the corpus.manifest file lets you verify afterwards that what you rebuilt is what you had. The raw copies for this corpus came to 1.9 GB and the complete baseline directory, with ranked lists and per-query scores, to about 40 MB, which is small enough that keeping every baseline forever is the correct policy. The ratio is what makes that a decision rather than a slogan: a baseline directory is about a fiftieth of the raw corpus it describes, so a hundred of them would still be only a couple of times the corpus itself, and a team recording one a fortnight would take four years to reach that. The storage argument for deleting old baselines does not exist. The only real cost is the discipline of writing the manifest, and that cost is paid once per baseline whether you keep it or not.

Storing run.jsonl — the actual ranked lists, not just the scores — has a second payoff. Metric definitions change: someone will want nDCG with linear gains, or recall at a different depth, or precision@10 that nobody computed in March. With the ranked lists on disk, all of those are recomputable from the archive in seconds. Without them, a new metric can only be applied going forward, and the history is stuck at whatever was computed at the time.

Why the lexical run has to be the first one recorded#

The temptation with a new retrieval stack is to build the interesting version first, because dense retrieval and reranking are what the work is actually about, and a BM25 index feels like a formality. Recording it first has consequences that are hard to recover later.

The lexical run is cheap to produce, deterministic given a fixed index, and requires no model, no GPU and no API budget, which means it can be re-run against any later corpus snapshot for a few minutes of compute. That property is what makes it useful as a counterfactual: the question "what would the old system have scored on today's corpus" has an answer you can compute on demand, and the answer is what every later claim of improvement is measured against.

There is also a literature reason. In 2009 Armstrong, Moffat, Webber and Zobel published an analysis of ad hoc retrieval results reported since 1998 and found that the reported improvements did not accumulate — successive papers each beat their own baseline while the absolute numbers stayed flat, because the baselines were weak and under-tuned. The paper's title, Improvements that don't add up, is a fair description of what happens to a retrieval roadmap when nobody insists on a strong, tuned lexical baseline recorded before the interesting work starts. A BM25 run with default parameters on a badly configured analyzer is easy to beat by twenty points, and beating it proves very little. Tuning $k_1$ and $b$ on a held-out split, and getting the analyzer right, is part of establishing the baseline rather than an optimisation to do later. Evidently AI's A complete guide to RAG evaluation is a practical walkthrough of the same discipline applied to RAG stacks specifically.

On one docs corpus that tuning was worth more than most people expect. A sweep over $k_1 \in {0.9, 1.2, 1.5, 1.8}$ and $b \in {0.3, 0.5, 0.75, 0.9}$, fitted on 90 of the queries and reported on the 50 held out, moved nDCG@10 from 0.561 at the Lucene defaults to 0.610 at $k_1 = 1.2$, $b = 0.5$ — the lower $b$ reflecting that these chunks are close to uniform in length, so length normalisation has less work to do. Adding a synonym list for the eleven vocabulary pairs the miss analysis surfaced was worth another 0.018. Nearly seven points of headroom sits in a baseline nobody tuned, and leaving it there will make the next change look twice as valuable as it is.

The seven points is the two moves added: 0.561 to 0.610 is a little under five points, the synonym list adds 0.018, and together they come to a bit under seven. The doubling claim is then arithmetic rather than rhetoric. Suppose the next change — a dense retriever, a reranker, whatever the interesting work is — is genuinely worth seven points. Measured against the tuned 0.610 it reads as seven points. Measured against the untuned 0.561 it reads as fourteen, because it is being credited with the tuning nobody did as well as with its own effect. Nothing about the change differs between those two reports, and the difference between them is exactly the headroom left on the table. Which is why tuning the baseline is not an optimisation to schedule after the interesting work; it is the act of deciding what the interesting work will be measured against.

A frozen baseline is not a regression test#

It can be wired into CI as one, and the two are different objects. A regression test asserts a threshold and fails a build; a baseline is an archived measurement that supports comparison. The failure modes differ accordingly, and a CI job that reruns the baseline every night against a moving corpus will produce a red build every time ingestion succeeds.

It is also not evidence about production. A golden set is a sample of last quarter's traffic scored by two labellers against a snapshot of the corpus, and a system that improves on it can still degrade for real users — through latency, through query types that never entered the strata, or through a change in what people search for. Offline comparability is a precondition for making claims, not a substitute for online measurement.

Freezing the corpus is not the same as freezing the index, which catches people out. The snapshot fixes the bytes; the index is those bytes run through an analyzer, a tokenizer, a stemmer and a chunker, all of which live in software that gets upgraded. Reproducing a March number in September may require pinning the search engine version, and if that is impractical, the honest move is to re-run the baseline after the upgrade and record a new one, noting the engine change as the cause of the step.

How often to re-baseline is a judgement rather than a rule, and three events force it: a corpus snapshot old enough that a replay against it no longer resembles production, a qrels version bump, and an engine or library upgrade that moves scores with nothing else changed. Everything else can be handled by replaying the archived configuration against the current snapshot, which is a comparison rather than a new baseline.

Four things pinned, stored together, and versioned rather than edited is the whole of the discipline, and the artefact it produces is about 40 MB per baseline against a corpus of 1.9 GB. What that buys is the ability to ask "what would the old configuration have scored on today's corpus" and get an answer in forty minutes rather than an argument, which is the only form in which a comparison between two dates is worth anything. The forty minutes and the few minutes quoted earlier for a lexical re-run are two different things and it is worth knowing which is which. Searching and scoring 160 queries against an existing index is the few minutes. The forty is that plus everything in front of it: fetching the raw bytes back out of object storage, running them through the chunker, and rebuilding the index. The second number is the one to quote to somebody deciding whether a replay is worth doing, because it is the one that includes the part that cannot be skipped.

The failure the September case exhibits is worth stating as a shape, because the reasoning that produced it is careful reasoning. Same commit, same command, same cluster, therefore same number — and every clause of that is true, and the conclusion does not follow. Pinning the code holds still the one input that was never going to move on its own, and it says nothing at all about the four that move by themselves while nobody is running anything. A control that is identical in the case where the measurement is sound and in the case where it is not cannot distinguish them, and the correction is not a better control over the code. It is a control over the things that drift.

Having both a frozen configuration and a current one still leaves the hard question untouched. When the September number is higher than the March number, several things changed in between — the corpus, the model, possibly the judgments — and the arithmetic that assigns credit among them is not the subtraction it looks like.