A run you cannot rebuild recorded a number, not a configuration

The winner of one team's Q1 retrieval sweep lives in a file called best_config.yaml. It was committed on 12 February, and it is eleven lines long.

# winner of the Q1 retrieval sweep — adopted 2026-02-12
chunk_size: 512
overlap: 64
k: 8
rerank_depth: 50
rewrite: true

# +7 nDCG@10 over previous settings
# owner: retrieval team

Six months later somebody wants to extend the sweep, and the sensible first move is to reproduce the baseline. They check out the pipeline, set those five values, run the 300-query golden set, and measure +2.1 nDCG@10 over the same previous settings the note refers to. Nothing failed, nothing errored, no test went red. The five recorded numbers were reproduced perfectly and the result was not, which tells you that the five numbers were never the configuration. They were the part of the configuration that happened to be under active discussion in February.

The score being compared is the ranking metric the sweep was optimising: it rewards putting the passages a human labelled relevant near the top of the first ten results, and both the recorded gain and the reproduced one are that same measurement taken against a set of labelled questions. So the two figures are commensurable, which is what makes the gap between them a problem rather than a category error. Something in the system moved, the file has no field in which it could have been written down, and nothing about the reproduction attempt was careless.

Everything the sweep held fixed without writing down#

Why the five numbers looked sufficient is worth walking through, because the reasoning is sound right up to the point where it fails. A trial's score is a function of every input to the pipeline, not only the swept ones. A sweep varies a handful of those inputs and holds the rest at whatever they happen to be, and because every trial in one sweep shares the held-fixed inputs, those inputs cancel out of any comparison between two trials. The cancellation is real, and it is what makes a five-line record honest in February: inside the sweep, five values are the only things that differ, so they are the only things that can explain a difference.

That cancellation is a property of the comparison and not of the file. Reproduce the run six months later and the two sides no longer share the held-fixed inputs, so nothing cancels. The difference now measured is the difference in five parameters plus the difference in everything else that moved, and the file has no slot in which the second term could have been noticed, let alone subtracted. Each of the following was constant across all 360 trials in February and is therefore invisible in a file that only lists what varied — and each of them moved between February and August.

The last row is the one most often shrugged off, and it is the one that makes the arithmetic unreadable. Both the recorded gain and the reproduced one are differences, and a difference has two arms. If the settings the February note compared against are not the settings the August rerun compared against, the missing points may sit entirely in the arm nobody was tuning, and every hour spent auditing the winner's configuration is spent on the wrong side of the subtraction. That is the reason the artifact defined below identifies a baseline by a run id rather than by a phrase.

The two inputs a lockfile cannot pin#

Ten rows, and only one of them — the chunking implementation — is the kind of thing a lockfile catches. That is the trap in this failure, because the reflex on hearing "we cannot reproduce a six-month-old result" is to reach for dependency pinning, and dependency pinning is genuinely necessary and would not have saved this run. The two rows most likely to explain a five-point regression are the corpus and the embedding model, and neither lives in the dependency graph. A team that responds by adding a lockfile will hit the identical problem next quarter inside a perfectly reproducible environment.

The reason is structural rather than a matter of diligence, and it is worth having as a chain rather than as a rule. A lockfile pins what your code imports, which it can do because imports are resolved before the program runs, from a declaration the tooling can read. A document store is not imported; it is read at run time, and other people are writing to it while you read. A model behind a provider alias is not imported either; it is reached by name over the network, and the name is resolved on the provider's side after your build has finished. Neither of the two heaviest inputs can appear in a dependency graph, so pinning that graph cannot pin them, and the graph goes on verifying clean while both move underneath it.

That is the shape worth learning to recognise, because it is the general form of a defence that does not defend. The quantity being inspected — the lockfile hash — stays constant while the dangerous property, what embed-v3 resolves to this week, varies freely. A reading that is identical in the world where the defence holds and the world where it fails is not evidence about the defence. The same shape sits in the evaluation metric itself, which is why the model swap is so much harder to catch than the corpus growth: a ranking score depends only on the order of the retrieved passages, so any change that shifts every similarity score by a similar amount leaves the order alone and the metric flat. A dashboard watching nDCG cannot fire on that change, and neither can a test that asserts the metric is above a threshold.

What the artifact should have been#

The unit worth recording is not the parameter set, it is the run: everything that went in, everything that came out, and enough identifiers to rebuild the inputs. In practice that is a manifest, a machine-written record of a single trial, emitted by the evaluation harness — the code that builds the index, answers the queries and scores the results — rather than typed by a person, because anything a person maintains by hand will drift out of agreement with the code within a month.

{
  "run_id": "sweep-2026Q1/trial-0187",
  "created_at": "2026-02-09T14:22:07Z",
  "code": {
    "pipeline_sha": "9f3c1ae4",
    "dirty": false,
    "lockfile_sha256": "b21d…"
  },
  "corpus": {
    "snapshot_id": "docs-2026-02-08T03:00Z",
    "document_count": 210418,
    "content_merkle_root": "sha256:7d41…"
  },
  "index": {
    "index_id": "idx-c512-o64-embedv3-20260209",
    "chunk_size": 512,
    "overlap": 64,
    "tokenizer": "cl100k_base",
    "embedding_model": "embed-v3-2025-11-04",
    "embedding_dim": 1024,
    "metric": "cosine",
    "hnsw": {"M": 32, "ef_construction": 128}
  },
  "query_time": {
    "k": 8,
    "rerank_model": "bge-reranker-v2-m3@sha256:41ab…",
    "rerank_depth": 50,
    "rewrite": {"enabled": true, "model": "haiku-4-5-20260114", "prompt_sha256": "c0e9…"}
  },
  "generation": {
    "model": "sonnet-4-5-20250929",
    "temperature": 0.3,
    "top_p": 1.0,
    "max_tokens": 512,
    "system_prompt_sha256": "1a77…"
  },
  "evaluation": {
    "split_id": "golden-v1-300@sha256:e5b2…",
    "judge_model": "sonnet-4-5-20250929",
    "judge_prompt_sha256": "88fd…",
    "metrics_lib": "ragmetrics==2.4.1",
    "seeds": {"sampling": 11, "generation": 11, "bootstrap": 20260209}
  },
  "result": {
    "ndcg@10": 0.5941,
    "recall@8": 0.812,
    "baseline_run_id": "sweep-2026Q1/trial-0000",
    "delta_ndcg@10": 0.0703
  }
}

Three properties of that document matter more than its exact schema. It is emitted by the code that ran the trial, so it cannot describe a run that did not happen. Every external dependency appears as something immutable — a dated model version rather than an alias, a digest rather than a tag, a snapshot id rather than "prod" — because the whole failure mode being defended against is a stable name pointing at moving content. And the baseline is a run id rather than a phrase, so delta_ndcg@10 is a difference between two objects you can both rebuild rather than a number relative to a memory.

The rule under the second of those is worth stating on its own, since the field list is only an instance of it: a name is safe when the thing it points at cannot change without the name changing too. A content digest has that property by construction, because the name is computed from the content. A dated version string has it by the provider's promise. An alias has it by nobody's promise, which is precisely what makes it convenient. The same test explains two fields that look like clutter. dirty records whether the working tree matched the commit, and without it pipeline_sha identifies nothing, because a commit id plus uncommitted edits is not a state anyone can return to. content_merkle_root exists because a document count does not move when a document is edited and a list of hundreds of thousands of hashes is not a field in a record; hashing the hashes yields one value that changes if any single document changes and holds still if none does. What it will not tell you is which document changed, so the underlying hashes still have to be kept if anyone will ever need to diff two corpora.

The result block also closes against the note in the YAML above, which is the minimum a reader should be able to check for themselves. Subtract the recorded delta from the recorded score and what comes back is the baseline's own score, and the delta is the same seven points the comment claims. Two files, written by different processes, agree about one trial — and it is the same trial chapter 4 takes apart, where this score is the first of five values produced by reruns that changed nothing.

A manifest is also not the same object as the file that configures production, and conflating them is how the eleven-line YAML came to exist in the first place. best_config.yaml is a deployment input: it says what the serving pipeline should do, and it is correct that it contains only the knobs the serving pipeline reads. The manifest is a record of an experiment, which needs fields the serving pipeline has no use for, such as which evaluation split produced the number and which run it was compared against. Teams that keep one file for both end up with the deployment file, because that is the one something breaks without. Keeping them separate and linking them — the deployed config carries the run_id it came from — costs one extra field and preserves the trail.

Storage is unglamorous and the choice matters less than picking one. Writing the manifest as a JSON object into an append-only table keyed by run_id, with the sweep's 360 rows queryable, is enough; so is one file per run in object storage under a sweep prefix. What breaks is storing it only as an attachment on a dashboard the vendor later reorganises, or only in the pull request that adopted the config, since neither survives the two or three years over which these questions get asked.

The idea is borrowed rather than new. Experiment tracking in machine learning has recorded hyperparameters, code version and metrics for a decade, and tools like MLflow and Weights & Biases made it a default. What RAG adds is that two of the heaviest inputs are not in the training script: a live document corpus that changes daily, and third-party models reached by name over HTTP. Classical experiment tracking assumed the dataset was a file you could hash and the model was code you could version, and both assumptions fail here. Data-versioning tools grew up alongside to handle the first; the second is largely a discipline problem, since every major provider offers dated model versions and every convenience example uses the alias.

The residual that survives a perfect rebuild#

A complete manifest does not make a run reproducible in the strict sense, and promising that it does sets the team up for a second round of confusion. Rebuild trial-0187 from every field above and the number will still not come back exactly: the judge is a sampled language model, the generator is running at temperature 0.3, and HNSW graph construction depends on insertion order across parallel workers. What the manifest guarantees is that the differences between two runs are the fields that differ, which is a weaker and far more useful property. Chapter 4 is about the residual variation that survives even an exact rebuild, and about why the winner of a sweep is systematically flattered by it.

That guarantee carries a condition worth attaching to it, since it is a claim about comparison rather than about reconstruction. It holds only while every external name in the manifest still resolves to what it resolved to. Let the provider retire the dated version, or let a retention policy delete the snapshot, and the manifest degrades from an instruction for rebuilding a run into a description of one — still enough to reason about what differed, no longer enough to remeasure it. The step a reviewer should make somebody demonstrate is that the names in the manifest outlive the questions the manifest exists to answer.

Pinning for an experiment, re-baselining after it#

Nor is the manifest an argument for freezing everything permanently. A pipeline pinned to a November embedding checkpoint forever is a pipeline that never gets the next model's gains, and teams that pin hard and then leave it alone find themselves two years behind with no path forward. The discipline is to pin during an experiment so the comparison is clean, then deliberately re-baseline when you upgrade: run the current champion on the new model version, record it as a new run with a new manifest, and let the delta be measured rather than assumed.

The unit that survives all of this is the run and not the winner. A parameter file answers what to set, which is a live question every time the service deploys; it cannot answer what was true when the number was measured, and that is the question somebody asks six months later with a regression in front of them and no idea which of ten inputs moved. Recording the run is a schema plus a discipline about names, and what it buys is that every later difference is a difference in something you can point at.

The cost is worth stating plainly because it is the reason teams skip it. Corpus snapshots are the expensive part — content-hashing 247,000 documents takes about four minutes per build, and retaining a snapshot per sweep is tens of gigabytes that exist only to make an audit possible six months later. Emitting the manifest is an afternoon of harness work and then roughly nothing. Pinning model versions costs vigilance, since the alias is what the SDK example uses and someone will paste it back in. Against that, the thing being bought is the ability to answer "why is it +2 now" with a measurement instead of a fortnight of guessing, which the team above spent.