A path records where the data was, not what it was

The June regression review had one action item: confirm that the March number was real. The run record for run-2f81c9 was complete by the standards anyone had asked for — faithfulness: 0.86, dataset: s3://evals/golden/v3/, model: claude-sonnet-4-5, evaluator: faithfulness@1.4, started_at: 2026-03-11T09:14:22Z. Re-running that configuration four months later produced 0.79. No step failed, no object was missing, the harness exited zero, and the log line that should have raised the alarm went past unread: scored 2,140 examples in 18m41s. The March run had scored 2,000.

Between the two runs, an annotator's export job had written 148 new examples under the same prefix in April, a second batch of 31 landed in May, and a lifecycle rule — a storage-side policy that deletes or archives objects once they reach a set age — with a 90-day expiry on a subdirectory of transcripts had quietly removed 39 older ones. The set had grown by seven percent and shifted in composition, and the run configuration recorded none of that because it had never been in a position to. What it recorded was a string that told the harness where to go looking. What came back depended entirely on when it looked.

Both of those numbers came out of the same machinery, and the parts of it are worth naming before the argument about them starts. An evaluation set, or eval set, is a fixed collection of examples that a system is run over in order to produce a quality number; each example holds an input, whatever documents or context belong with it, and the label or reference answer that says what a good response would have been. An evaluator is the program that turns the system's output on one example into a score, and for a quality such as faithfulness — whether an answer is genuinely supported by the documents retrieved for it — the evaluator is usually a second language model handed a rubric and asked to judge, which is what faithfulness@1.4 names. The harness is the code that joins the two: it resolves the dataset reference, runs the system under test over every example, calls the evaluator, and writes the run record. Topic 3.4 covered how a curated set of this kind is assembled, and this topic starts from one that already exists.

Versioning an eval set means giving each state of it an identifier that cannot quietly come to mean a different collection of examples later on. Lineage is the recorded chain from a number back to the inputs that produced it — these examples, this evaluator and its configuration, this model, this prompt — so that any score in the history can be traced to what it was computed on. Both exist to make one question answerable: when a number moves, did the system get worse, or did the thing measuring it change underneath. The week that question arrives is usually a quarterly regression review, a sign-off on a model upgrade, or a customer escalation asking whether a claim the team made was ever true, and at that point the answer has to be evidence rather than recollection. Without versioning and lineage, an evaluation history is a column of numbers with dates beside them and no way to explain any of the gaps.

A prefix is a query, not a container#

Object stores encourage the confusion. s3://evals/golden/v3/ looks like a directory containing a fixed set of files, and the console renders it as a folder tree, but there is no directory object in S3 or GCS — there are keys, and a prefix is a filter applied to the key space at the moment you list it. Every eval harness that resolves a dataset by listing a prefix is therefore executing a query against mutable state and treating the result as a constant. The v3 in the path is a human convention with no enforcement behind it. Nothing in the storage layer knows that v3 is supposed to be frozen, so nothing rejects a write into it, and nothing flags that a listing returned a different set of keys than it did last quarter.

The same defect explains a disagreement that costs teams days. Two teams report results on support-golden-v2. Team A reports 0.88, team B reports 0.79. Same model, same evaluator version, same shared harness, no custom preprocessing on either side. The instinct is to go looking for a difference in code, because the numbers are far enough apart that something substantial must differ, and the code is where substantial differences usually live. The actual cause is that team A pulled the set in the morning and team B pulled it after an annotator merged a batch of forty hard escalation cases that afternoon. The name support-golden-v2 was bound to a location, both teams resolved it faithfully, and neither run recorded which state of that location it received. The disagreement is not reconcilable from the run records, only from the storage access logs, if those are still retained.

What is missing is a way for a run to say which examples it scored, in a form that a later run can check rather than assume. The rest of this chapter is about how a result computed on a curated set stays defensible after the set itself moves on.

Hash the contents, resolve through a manifest#

A dataset commit has to capture the contents, and the preprocessing that turned those contents into scoreable inputs, in a single identifier that changes whenever either of them does. Content addressing gives you that directly: hash each example canonically, hash the sorted list of those hashes together with the preprocessing configuration, and use the resulting digest as the name of the version. This is what Git does with trees and what container registries do with image digests, and the property that matters is the same in all three cases — the identifier is derived from the bytes, so it is impossible for the bytes to change without the identifier changing.

The unit that gets hashed is a manifest: a small document listing every example in the set by its own content hash, plus the configuration that governs how those examples are prepared.

{
  "manifest_version": 1,
  "dataset": "support-golden",
  "created_at": "2026-03-10T16:02:41Z",
  "preprocessing": {
    "normaliser": "nfkc-lower-collapse-ws",
    "chunker": "recursive@700/80",
    "retriever_corpus": "sha256:9c1f0ab4…",
    "label_schema": "faithfulness-v3"
  },
  "examples": [
    { "id": "sup-000001", "sha256": "4d7a19c0…" },
    { "id": "sup-000002", "sha256": "b02fe511…" }
  ],
  "example_count": 2000,
  "content_hash": "sha256:71bb3e9c8a4d…"
}

content_hash is computed over the canonicalised manifest with the content_hash field itself removed, so it covers both the membership list and the preprocessing block. The code that produces it is short enough that there is no excuse for hand-waving it:

import hashlib, json

def example_hash(ex: dict) -> str:
    # Only the fields that affect scoring. Canonical JSON: sorted keys,
    # no insignificant whitespace, UTF-8, ensure_ascii off.
    payload = {
        "input": ex["input"],
        "expected": ex["expected"],
        "labels": ex["labels"],
        "context_ids": sorted(ex["context_ids"]),
    }
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":"),
                      ensure_ascii=False).encode("utf-8")
    return hashlib.sha256(blob).hexdigest()

def manifest_hash(manifest: dict) -> str:
    body = {k: v for k, v in manifest.items() if k != "content_hash"}
    body["examples"] = sorted(body["examples"], key=lambda e: e["id"])
    blob = json.dumps(body, sort_keys=True, separators=(",", ":"),
                      ensure_ascii=False).encode("utf-8")
    return "sha256:" + hashlib.sha256(blob).hexdigest()

Two choices in example_hash decide how useful the whole scheme is. The payload deliberately excludes fields that do not affect scoring — created_by, annotation_notes, imported_at — because a hash that changes when an annotator's username changes produces version churn that trains people to ignore version changes. It just as deliberately includes the labels, since a relabelled example is a different example for every purpose an eval has, even though its input text is byte-identical. Sorting context_ids normalises an ordering that carries no meaning; the input string is not normalised at all, because a trailing space in a prompt is a real difference that has moved real scores.

The run then records dataset_hash: sha256:71bb3e9c… rather than a path, and resolving it is a lookup: the manifest store maps the hash to the manifest, and the manifest maps each example id to a content-addressed blob, meaning the stored bytes of that one example filed under their own digest rather than under a name someone chose. Where the bytes physically live becomes an implementation detail that can change — a bucket migration, a region move, a switch from S3 to a managed dataset service — without invalidating a single historical run record.

The dotted edge is where mutability is allowed to live. A prefix can still be the convenient handle humans type, and an export job can still write into it; what changes is that publishing a version means computing a manifest from whatever the prefix contains at that moment and freezing the manifest, so the mutable listing is upstream of the version rather than being the version.

Per-object version ids pin objects, not membership#

The tempting shortcut is bucket versioning. S3 and GCS will both keep every version of every object and hand you a version id per object, and recording those ids in the run config feels like content addressing without any manifest machinery. It is genuinely useful and it does not solve this problem, because a per-object id is a statement about an object that exists and says nothing about which objects belong to the set. Add a file to the prefix and the dataset has changed while every recorded version id still resolves perfectly. Delete one and the same is true — you can fetch all 2,000 objects you recorded, and the set that a re-run built by listing the prefix has 2,140. The ids certify the parts and never the whole.

The failure gets worse in the case where you most need it. Lifecycle configurations that expire noncurrent versions are the standard way to stop versioned buckets growing without bound, and a rule of NoncurrentVersionExpiration: 90 days means that the version ids from a run older than ninety days become unresolvable exactly when someone challenges an old result. You are then in the position of having a precise reference to something that no longer exists, which is materially worse than having no reference, because the precision invites the belief that the number was reproducible.

For an eval set of a few thousand text examples, the manifest is the cheap option and the operational cost is close to zero: 2,000 entries of an id and a 64-character digest is roughly 180 kB of JSON, and publishing a version is a hash of a few megabytes of text, which finishes in well under a second. The dedicated data-versioning systems earn their keep when the artefacts are large binaries — audio, video, image corpora — where deduplicated content-addressed storage is the actual problem being solved rather than a side effect. LangSmith and Braintrust both build dataset versions into their eval products along these lines, which is worth knowing if you would rather not run the manifest store yourself.

"We have never overwritten a versioned directory"#

The defence offered for the path-based scheme is usually operational discipline: in three years, nobody has ever written into a directory that was already published. Taken at face value it is a claim about intent, and the failures in this chapter were not intentional. The April export job did not overwrite anything, it appended, and appending into a prefix requires no rule to be broken because nobody wrote the rule down anywhere the storage layer could read it. The lifecycle expiry was configured by a platform team for cost reasons and applied to a key pattern, not to a dataset. A restore from backup after an incident rewrites objects with new timestamps and, depending on the tool, a different key ordering. None of that is somebody overwriting a versioned directory.

The deeper problem with the defence is that it cannot be checked. A team with no content hashes has no mechanism that would have told them if a directory had been modified, so the observation "it has never happened" is indistinguishable from "we would not have noticed." The value of hashing is not that it prevents mutation, since it prevents nothing; it is that mutation stops being silent. A re-run that resolves sha256:71bb3e9c… and finds that the assembled examples hash to something else fails loudly at the start rather than producing a plausible number twenty minutes later.

Content addressing settles which examples a number was computed on, and it settles nothing else. It does not say which evaluator configuration scored them, which model artefact produced the outputs, or which prompt was in force — and a March-to-June comparison in which the dataset hash is identical can still move by seven points for any of those reasons. It also says nothing about why the set changed, only that it did. Pinning the data is one field of a record that needs several more before a past score is genuinely reproducible, and the next chapter sets out the minimum that record has to contain.