A score is reproducible only with the run id binding its inputs

The quarterly review takes eleven minutes to run out of evidence. Faithfulness was 0.81 in March and 0.74 in June, and the room produces five hypotheses in the first two minutes: the judge prompt was rewritten in April, someone bumped the retrieval top-k — the number of passages the retriever fetches for each question — from 5 to 8, the model alias rolled to a new snapshot in May, an annotator relabelled a batch, and the chunker changed, meaning the step that cuts source documents into those passages now cuts them differently. All five are plausible, three of them are true, and nobody can say which because the eval database has two columns that matter — run_at and score — plus a notes field that says nightly for all 1,100 rows. The meeting ends with someone volunteering to reconstruct the history from Git blame and Slack, which takes two days and produces a timeline with the word "probably" in it four times.

The alternative is a table you can query. The June row and the March row differ in some fields and agree in others, and reading which is which takes about as long as it takes to run the query. Getting there requires deciding, in advance, what a run is obliged to write down.

Five fields and the identifier that joins them#

A score is a function of the examples, the thing being scored, and the thing doing the scoring. The minimum record makes each of those addressable and gives the whole a name:

• the dataset content hash, in the form the previous chapter established, so membership and preprocessing are pinned rather than located;
• the evaluator version together with its full configuration, because faithfulness@1.4 names the code and says nothing about the threshold, the judge model, the number of samples, or the rubric text it was run with;
• the model artifact identifier, meaning the immutable snapshot that served the requests rather than the alias you asked for;
• the prompt version, covering both the system prompt of the application under test — the standing instructions the model is given ahead of any user input — and the judge's rubric, each pinned by content hash rather than by a name someone can edit;
• the run id, a single opaque identifier that every row, artifact, log line, and per-example score carries, so that all of the above join to exactly one execution.

The run id is the field most often treated as bookkeeping and it is what makes the rest usable. Without it, joining a score to its per-example outputs means matching on timestamp, and timestamps mis-join in precisely the situations that matter: a run that was retried after a rate-limit failure writes two sets of per-example rows minutes apart, a backfill re-scores March data in June so the run's logical date and wall-clock date disagree, and two nightly jobs on different datasets overlap. A run id is also what lets a record be append-only. Nothing about a completed run is ever updated, which means the audit trail cannot be quietly repaired later.

{
  "run_id": "run-2026-06-14-a3f2",
  "dataset_hash": "sha256:71bb3e9c8a4d…",
  "dataset_name": "support-golden",
  "example_count": 2000,
  "evaluator": {
    "version": "faithfulness@1.4.2",
    "code_sha": "sha256:0ac91d55…",
    "config_hash": "sha256:c47e21b0…",
    "config": {
      "judge_model": "claude-sonnet-4-5-20250929",
      "judge_prompt_hash": "sha256:5f10dd82…",
      "samples_per_example": 1,
      "threshold": 0.5
    }
  },
  "system_under_test": {
    "model": "claude-sonnet-4-5-20250929",
    "model_alias_requested": "claude-sonnet-4-5",
    "prompt_hash": "sha256:9de40cc1…",
    "retriever": { "top_k": 8, "index_hash": "sha256:2b7745af…" }
  },
  "harness_sha": "sha256:be0f3311…",
  "result": { "faithfulness": 0.74, "n_scored": 2000, "n_errors": 0 },
  "started_at": "2026-06-14T02:00:11Z"
}

Both model and model_alias_requested are present on purpose. The alias is what the config asked for and is useful for understanding intent; the resolved identifier is what actually answered, and it is the only one of the two that means anything six months later. Aliases move — a provider publishes dated snapshots such as claude-sonnet-4-5-20250929 alongside undated names that route to whichever build is current, and a config that points at the undated name in March and June is a config that pinned nothing. A record showing claude-sonnet-4-5-20250929 in one row and a different resolved model in the other has already answered one of the five hypotheses before anyone opens a terminal.

Recorded at write time, or not recorded at all#

The distinction that decides whether a record is worth having is when each field is resolved. A field resolved at write time — computed during the run and frozen into the row — is durable evidence. A field resolved at read time is a pointer into mutable state, and it degrades from evidence into a suggestion the moment that state moves.

Every row in the right-hand column is the same mistake wearing different clothes, and it is tempting because at the moment of writing the config, the pointer and the value are indistinguishable — configs/faithfulness.yaml does contain the threshold you intend, today. The cost only appears at the point of a challenge, and by then the fix is retroactive and impossible.

The evaluator version deserves a note of its own, because a semantic version is the field most often mistaken for a pin. faithfulness@1.4.2 is a label a human chose, and whether it moves when the code moves depends on whether every contributor remembered to bump it; in practice a judge prompt gets tweaked, the version stays at 1.4.2, and two runs claim the same evaluator while running different code. Recording the commit hash of the evaluator package alongside the semantic version costs one line in the harness and makes the claim checkable, which is why the record above carries both. The semantic version is for humans reading a table, and the hash is for the comparison.

The mechanical defence is to make the recorder refuse anything it cannot pin:

import hashlib, json, re

MUTABLE = re.compile(r"(^|[:/@-])(latest|main|master|head|current|prod)$", re.I)

def pin_str(name: str, value: str) -> str:
    if MUTABLE.search(value):
        raise ValueError(f"{name}={value!r} is a moving reference; "
                         f"resolve it before the run starts")
    return value

def pin_file(path: str) -> dict:
    data = open(path, "rb").read()
    return {"path": path, "sha256": hashlib.sha256(data).hexdigest(),
            "bytes": len(data)}

def pin_config(cfg: dict) -> dict:
    blob = json.dumps(cfg, sort_keys=True, separators=(",", ":")).encode()
    return {"config": cfg,
            "config_hash": "sha256:" + hashlib.sha256(blob).hexdigest()}

pin_config stores the configuration inline as well as hashing it, and the inline copy is the part that earns its place. A hash alone tells you two runs differed and leaves you unable to say how, which turns every investigation into an archaeology exercise against a config store that may not have kept the old value. Evaluator configs are hundreds of bytes; storing them denormalised in every run row costs nothing and removes a whole class of dead end.

The model snapshot has to be captured from the response rather than the request, since that is the only place the truth appears. The Anthropic Messages API returns the resolved snapshot in the model field of the response body even when the request used an alias, and the pattern generalises to any provider that publishes dated snapshots:

resp = client.messages.create(
    model="claude-sonnet-4-5",           # alias in the request
    max_tokens=1024,
    messages=[{"role": "user", "content": prompt}],
)
run["system_under_test"]["model"] = resp.model   # dated snapshot in the reply
run["usage"] = {"in": resp.usage.input_tokens, "out": resp.usage.output_tokens}

Capture it on the first call of the run and assert that every subsequent call returns the same value, because an alias can roll mid-run for a job that takes forty minutes, and a run that used two different snapshots is a run whose single recorded model identifier is a lie about half its rows.

This is a lockfile for an experiment#

The shape of the problem is the one that build systems worked through two decades ago. A build that says "depends on requests" is not reproducible, a build that says "depends on requests==2.31.0" is closer, and a build that records the resolved hash of every transitive artefact is reproducible in the sense that matters — you can tell whether two builds had the same inputs without trusting anyone's memory. Package lockfiles, Nix derivations, container image digests, and more recently the SLSA provenance attestations that CI systems emit alongside artefacts — signed machine-readable records of which sources, tools and build steps produced a given binary — are all the same instinct applied at different scopes: describe the inputs of a computation precisely enough that the output can be defended.

An eval run is a build whose output happens to be a number rather than a binary, and it has one property that makes the discipline more important rather than less. A binary that was built from the wrong inputs usually misbehaves, so the error surfaces on its own. A score computed from the wrong inputs is a perfectly well-formed float that looks exactly like a correct one, gets pasted into a dashboard, and is used to make a decision. There is no runtime symptom to catch it.

The cost of recording is not a consideration at any realistic scale. A run record of the kind above, with the evaluator config inline, is roughly 4 kB; a team running 40 evaluation runs a day writes 160 kB a day, or about 58 MB a year, which is smaller than the per-example outputs of a single run. Per-example rows — the run id, example id, score, and judge rationale — are the bulk, at perhaps 600 bytes each, so 2,000 examples per run comes to 1.2 MB and the same 40-runs-a-day team writes about 17 GB a year. That is worth a retention policy, and it is not worth truncating the run record to save.

A complete record is not an explanation#

Having all five fields does not tell you why the number moved; it tells you which candidate causes are still standing. Comparing the March and June rows in the example that opened this chapter narrows five hypotheses to three — the judge prompt hash differs, the model snapshot differs, and the dataset hash differs, while the harness and the retriever config are identical. That is a real gain and it is not an attribution, because three factors moved together and their effects can be of either sign. Sorting that out needs deliberate reruns rather than a query, which is the subject of chapter 5.

The record is also silent about whether the difference deserves an investigation. Seven points looks decisive; on a 200-example subset scored by an LLM judge it might not be, and on 2,000 examples it probably is, but "probably" is doing load-bearing work in both of those sentences. Every comparison in the chapters that follow — the crossed reruns, the restore test, the diff of two dataset versions — asks whether two numbers differ, and none of them means anything without a tolerance. The next chapter establishes where that tolerance comes from and why it is not zero even when every field in the run record is pinned.