Versioning and lineage for evaluation sets

Read these first

• Building a golden dataset

What you will know

• Version an eval set so a past result can be reproduced

  A dataset commit must capture contents and relevant preprocessing, not merely the name of a storage location that may change underneath it.

• Track which dataset version produced which number

  Each recorded score needs a durable link to the exact set, evaluator configuration, model artifact, and run that generated it.

• Reproduce a score change with the eval data held constant, then with code held constant

  These controlled reruns distinguish evaluator or model changes from changes in the examples, labels, or sampling policy.

• Restore a previous version and re-run

  A restore test exposes missing blobs, mutable dependencies, nondeterministic sampling, and other gaps that make a version label meaningless.

• Connect a metric change to a data change rather than a code change

  A paired diff of examples, labels, and inclusion rules should provide evidence for the attribution instead of relying on timing or intuition.

Chapters

1. 1

   A path records where the data was, not what it was

   Every run in your history records dataset: s3://evals/golden/v3/, and a result from four months ago will not reproduce. A prefix is a location — objects behind it can be overwritten, appended to, or expired by a lifecycle rule, and nothing notices. The same defect explains two teams reporting 0.88 and 0.79 on the same named set with identical models and evaluators. Content hashing resolved through a manifest pins membership, which per-object version ids never do.

   Read chapter
2. 2

   A score is reproducible only with the run id binding its inputs

   Asked why faithfulness fell from 0.81 in March to 0.74 in June, you either guess or read it off. The minimum recorded per run is the dataset content hash, the evaluator version with its full configuration, the model artifact identifier rather than a moving alias, the prompt version, and a single run id tying them together. Anything resolved at read time — a latest tag, a config file edited in place — is a field you only appear to have.

   Read chapter
3. 3

   An eval score is a draw from a distribution, not a reading

   Every comparison in the chapters that follow asks whether two numbers differ, and none of them means anything without a tolerance. Get one by repeating a single identical run several times with the dataset hash, evaluator, model artefact and prompt all pinned, then characterising the spread — which for an LLM judge is not zero even at temperature 0, and which widens as the set gets smaller. What comes out is a band: inside it a difference is a wobble, outside it a difference is evidence. Without that band, 'well inside normal noise' is an assertion rather than a finding.

   Read chapter
4. 4

   A version label means nothing until something restores it

   Your sets are versioned, every run records which version it used, and nobody has ever restored one — so the first thing a restore finds is usually a blob a retention policy deleted, failing before a single example is scored. Mutable dependencies and nondeterministic sampling follow. Build the restore as an automatic job with explicit assertions, and decide in advance what a recorded 0.85 coming back as 0.83 means, using the measured run-to-run variance rather than an appeal to noise.

   Read chapter
5. 5

   Cross the runs to separate a data change from an evaluator change

   When a score moves between releases, run the old evaluator against the new dataset and the new evaluator against the old one. A single crossed cell is decisive only when exactly one factor moved; with a judge prompt change, a judge model upgrade and 200 relabelled examples all inside the window, the effects can point in opposite directions and partially cancel. The chapter also covers what to do when the provider has deprecated the old judge and one variable cannot be restored at all.

   Read chapter
6. 6

   Only the unchanged intersection compares across two dataset versions

   A faithfulness score jumps five points overnight with nothing deployed, and an annotator spent yesterday relabelling. Diff the two dataset versions example by example — added, removed, relabelled — then re-score only the unchanged intersection with the same evaluator and see whether the jump survives. The identical rule governs the week your set grew from 1,400 to 1,600 examples and read 0.86: that figure came from a different population and is not comparable in either direction, footnote or not. Once it has been on a dashboard it has to be retracted and restated on the comparable population, and the dashboard itself has to break or re-baseline its series when the dataset version changes.

   Read chapter