Only the unchanged intersection compares across two dataset versions

The nightly run finished at 02:41 and the faithfulness score read 0.863, against 0.812 the night before. Nothing had been deployed in eighteen days. The model snapshot, prompt hash, evaluator version, evaluator configuration hash, and harness commit were identical across the two runs; the only field that differed was dataset_hash. An annotator had spent the previous day working through the faithfulness labels on a batch of ambiguous support cases.

Five points is well outside the ±1.8-point band this pipeline was measured at, so the movement is real. What it is not yet is a quality improvement, because the two numbers were computed over two different label sets, and a score is only meaningful relative to the labels it was scored against. The instinct at this point is to look at the annotator's session notes and form a theory. The cheaper move is to diff the two versions first, because the diff usually makes the theory unnecessary.

Diff by content hash, not by count#

Both versions resolve to manifests, and both manifests list every example by id and content hash — which, because chapter 1 folded the labels into example_hash, means a relabelled example has a different hash under the same id. That gives four partitions from a set operation:

def diff_manifests(old: dict, new: dict) -> dict:
    o = {e["id"]: e["sha256"] for e in old["examples"]}
    n = {e["id"]: e["sha256"] for e in new["examples"]}
    both = set(o) & set(n)
    return {
        "added":     sorted(set(n) - set(o)),
        "removed":   sorted(set(o) - set(n)),
        "modified":  sorted(i for i in both if o[i] != n[i]),
        "unchanged": sorted(i for i in both if o[i] == n[i]),
    }

d = diff_manifests(v7, v8)
print({k: len(v) for k, v in d.items()})
# {'added': 0, 'removed': 0, 'modified': 168, 'unchanged': 1832}

Comparing counts alone would have said the two versions were identical — 2,000 examples each — which is why a diff that reports only sizes is worse than no diff. The 168 modified ids are the annotator's session, and modified is deliberately coarse: it says the scoreable content changed without saying whether that was the label, the expected answer, or the retrieved context ids. Narrowing that is a second pass over the two blobs, field by field, and it matters when the labels are not the only thing an annotation tool can touch.

Re-score the intersection, then account for the rest#

With the partitions in hand, the comparison that means something is the one restricted to unchanged: 1,832 examples whose content is byte-identical across both versions, scored by the same evaluator. Running the March-night and the following-night evaluations over that subset alone gives 0.860 both times, a difference of zero within the band. The pipeline did not change, and now that has been demonstrated rather than assumed.

The rest is arithmetic, and the arithmetic should close:

The 168 relabelled examples went from being failed 71 percent of the time to being passed 90 percent of the time, and 168 divided by 2,000 multiplied by 0.607 is 0.051, which is the entire movement. When the partition contributions sum to the aggregate delta within the band, the account is complete and there is nothing left to attribute. When they do not sum, something outside the dataset moved as well, and the crossed design from the previous chapter is the tool for that.

What the accounting has not settled is whether the relabelling was right. A jump concentrated in a relabelled subset is equally consistent with an annotator correcting 168 genuinely wrong labels and with an annotator applying a looser reading of "faithful" than the guideline intends. Those have opposite implications and the same arithmetic signature, so the next step is adjudication rather than analysis: sample 30 of the 168, have a second annotator label them blind against the written guideline, and compare. In this case 8 of 30 disagreed with the new label and all 8 disagreements were in the same direction, which points at a drift in interpretation rather than a correction, and sends the batch back rather than into the baseline.

The week the set grew#

The same rule governs a case that looks entirely different. A weekly quality metric reported to leadership has sat between 0.78 and 0.80 for two months. This week it reads 0.862. The model has not changed. The eval set grew from 1,400 to 1,600 examples when a teammate added a batch of new cases.

The temptation is to attribute the whole jump to the new examples, and a moment of arithmetic shows that cannot be right. Adding 200 examples to 1,400 means the new material is 12.5 percent of the set, so even if every added example passed, the ceiling on the aggregate is (1,400 × 0.790 + 200 × 1.000) / 1,600 = 0.816. A batch of new examples cannot lift a 1,600-example aggregate to 0.862 no matter how easy it is. That bound is worth computing before any theory, because it disqualifies the obvious explanation in about ten seconds.

The diff and the intersection re-score locate what actually happened:

The comparable population moved 5.3 points on its own, and the composition effect — the aggregate moving because the mix of examples changed rather than because the system did — accounts for the remaining 1.6. The 5.3 is the finding, and it is a finding on a fixed population with nothing deployed, which is the signature of an unpinned dependency, the second failure family from chapter 4. In this case the retrieval index, the searchable store of document passages the system answers from, had been rebuilt on Tuesday against a corpus with two months of newer documentation, which is a genuine improvement to the product and a change the run record described only as prod-index.

So the honest answer to what gets reported is the 0.846 on the comparable population, with the composition effect broken out and the index rebuild named as the cause. The 0.862 is not reported as this week's value of the series, because it was measured on a population the previous eight points were not measured on. The failure of "report it with a footnote" is not that the footnote is dishonest; it is that a footnote annotates a number without changing what the number does. By next week 0.862 is the anchor everyone remembers, a return to 0.85 on the stable set reads as a regression, and somebody spends a sprint bisecting releases for a change that never shipped.

The result that would make the improvement believable is specific: the unchanged intersection moving outside the measured band, with every dependency pinned well enough that chapter 4's restore reproduces both runs, and a crossed run confirming the evaluator configuration is identical across the two. That is exactly what happened here, and it is why the 5.3 survives while the 6.9 does not.

Retract on the same channel, and make the chart break#

Leadership has already seen 0.862 on a dashboard, so there is a correction to issue and the only question is how. Send it the same week, through the same channel, to the same audience: state the number that was shown, state that it was computed on a set that grew by 200 examples and is therefore not comparable to the preceding weeks in either direction, give the comparable figure of 0.846, and name what caused the real part of the movement. A correction that arrives a month later inside a longer document is not a correction, because by then the wrong number has been used to make decisions.

The dashboard itself is the durable fix, and the design rule is that a series must not span a change in the inputs that define it. Every point on the chart is keyed by the pair of hashes that produced it — the dataset hash and the evaluator configuration hash — and when either changes, the line breaks rather than continuing:

{
  "series": "weekly_faithfulness",
  "segment_key": ["dataset_hash", "evaluator.config_hash"],
  "on_segment_change": "break",
  "annotate": ["dataset_hash", "example_count", "run_id"],
  "points": [
    { "week": "2026-05-24", "value": 0.791, "dataset_hash": "sha256:71bb3e9c…", "n": 1400 },
    { "week": "2026-05-31", "value": 0.793, "dataset_hash": "sha256:71bb3e9c…", "n": 1400 },
    { "week": "2026-06-07", "value": 0.862, "dataset_hash": "sha256:e0417c6b…", "n": 1600 }
  ]
}

Three policies are available at that boundary, and they suit different situations:

Backfilling is the option that chapters 1 through 4 make possible: re-scoring the eight previous weeks against the new dataset version requires that each historical run's model snapshot, prompt, and evaluator configuration still resolve, which is precisely what the restore drill has been verifying every week. Roughly $240 buys a continuous, honest eight-week series on one population. A team without that machinery has only the first and third options, and the third is not really an option.

Where this leaves you#

The pieces now compose into one loop. A dataset version is a content hash resolved through a manifest, so membership is pinned rather than located. A run record binds that hash to an evaluator configuration, a model artefact, a prompt, and a run id, so a past number has evidence behind it. A measured band turns "that looks different" into a decision with a stated basis. A scheduled restore keeps the past actually reachable rather than nominally referenced. Crossed runs attribute a movement to a factor. A version diff and an intersection re-score attribute it to a population.

If you are building this from nothing, the order that pays back fastest is not the order this topic presented it in. Start by storing per-example scores alongside every aggregate, because it costs nothing and every later technique depends on it. Then add the run record with the five pinned fields, since it is the artefact everything else reads. Then measure the band once, which takes an afternoon and immediately changes how the team argues. Content-addressed manifests and the restore drill come next and are the larger engineering task. The crossed-run and diff techniques need no new infrastructure at all once the first three exist.

The test of whether it is working is not that scores stop moving. Scores should move; the product changes and the data changes with it. The test is that when someone asks why a number is different from the one they remember, the answer takes twenty minutes and comes with a run id attached, instead of taking two days and ending in a word like probably.