A rerun that reverses itself measured a different system

The pipeline shipped in February on the strength of the paired result, and in May somebody re-ran the harness — the code that loads both arms, runs them over the fixture set, scores every run and totals the results — before a planning meeting, expecting to paste a confirming number into the deck. The command was the same one in the README, the fixture directory was the same 150 contracts, five trials per task, jobs interleaved. The loop came back at 88.6% and the pipeline at 85.1%, a gap of three and a half points in the opposite direction from the one that had decided the architecture. Both agent implementations were checked with git log --since=2025-02-01 on their source directories and neither had a single commit. The first reaction in the room was that one of the two runs must be broken.

Neither run is broken. They are two measurements of two different systems, scored by two different rubrics, and the harness recorded nothing that would let anyone say which parts differed. That is the failure worth naming, because it is not a mistake anybody made on a particular day — it is what happens by default when an evaluation reads its inputs from wherever they happen to live.

What the harness resolved at run time#

The command in the README named an arm, a fixture directory and a trial count. Everything else it needed, it looked up.

Each of those five arrows moved between February and May, and each moved for a reason that was correct in isolation. The model alias repointed twice as the provider shipped new builds, which is what an alias is for. The system prompts live in the application repository because they are application code and belong under review with the code that uses them; fourteen commits touched them in the quarter, mostly tightening the redline formatting instructions after customer feedback. The document-search tool gained an optional jurisdiction filter and its default result count went from five snippets to eight, a change made by the platform team to help a different consumer. Six fixtures were re-labelled in March when a customer disputed the ground truth on a set of assignment clauses. And in month two the rubric gained a criterion for whether the redline preserved defined-term capitalisation, worth one of eight points.

The only two rows that held still are the two everybody checked. That is not a coincidence, because those are the rows that live in the directory the comparison is nominally about, and the instinct when a result reverses is to look there first. Everything else in the table is infrastructure that nobody thinks of as part of the experiment until it changes an answer.

The direction of the reversal even has a plausible mechanism, which is worth stating so that pinning does not sound like superstition. Eight retrieval snippets instead of five is roughly 60% more retrieved context per tool call; the pipeline makes three such calls across its stages against the loop's typical two, so the pipeline's context grew more, and its clause-normalisation stage began exceeding the window on long amendments and truncating the policy list. The new capitalisation criterion is one the loop's single prompt handles incidentally because it drafts the redline in the same context that read the original, while the pipeline's drafting stage receives normalised clauses in which the defined terms have already been lower-cased. Either of those alone could account for three points. The problem is that this is a story constructed after the fact from two aggregate numbers, and there is no artefact from February that could confirm or refute it.

A manifest is the unit that makes a number attributable#

Pinning means the harness writes down, and where possible fixes, every input that the numbers depend on, and stores that alongside the results. The file it writes them into is the run's manifest, and the one below is what the eval emits before it dispatches a single job; the run identifier is a hash of it, so two runs with the same identifier are the same experiment by construction.

{
  "run_id": "9f2c1a7d",
  "created_at": "2025-05-14T09:12:44Z",
  "harness_commit": "9c02ab1",
  "arms": {
    "loop": {
      "impl_commit": "b7710e4",
      "model": "claude-contract-2025-01-07",
      "prompt_hashes": { "system": "sha256:4b1f…8ac2" },
      "tool_schemas": { "doc_search": "sha256:19de…03f7" }
    },
    "pipeline": {
      "impl_commit": "b7710e4",
      "model": "claude-contract-2025-01-07",
      "prompt_hashes": {
        "extract": "sha256:c00a…7712",
        "normalise": "sha256:88ef…41b9",
        "draft": "sha256:2d5c…9e01"
      },
      "tool_schemas": { "doc_search": "sha256:19de…03f7" }
    }
  },
  "fixtures": { "set": "contracts-150", "labels_rev": "v2", "sha256": "e3b0…b855" },
  "rubric": { "version": "3", "criteria": 8, "scorer_commit": "5aa9013" },
  "pricing_table": "2025-05-01",
  "trials_per_task": 5,
  "concurrency": 20
}

Two details in there carry more weight than their size suggests. The model field holds a dated snapshot rather than an alias, which is the difference between "whatever is current" and "the build that produced these numbers"; providers publish snapshot identifiers precisely so that evaluations can name one. The pricing table is dated separately from everything else, because cost per task is a product of token counts and a price list, and a provider price cut will change your cost column without changing your system at all — a comparison that reports dollars without a price date is reporting two things fused together.

Prompts are hashed rather than copied so the manifest stays small, with the full text written into the run's artefact directory. The hash is what makes a diff possible; the text is what makes the diff readable. Loading them through the harness rather than through the application's own import path is a small piece of discipline that pays off later, because it means the eval can run a February prompt against a May model without checking out the application at an old commit.

def load_arm(spec: ArmSpec) -> Agent:
    prompts = {name: read_pinned(spec.prompt_refs[name]) for name in spec.prompt_refs}
    tools = [load_tool_schema(ref) for ref in spec.tool_refs]
    return Agent(
        model=spec.model,            # dated snapshot, never an alias
        prompts=prompts,
        tools=tools,
        client=client_for(spec.model, pricing=PRICING[spec.pricing_table]),
    )

Capture per run, not per experiment#

Chapter 1's Run record already carried tokens, tool calls, latency and a trace identifier for every execution, and the manifest gives those records a context. What the May meeting needed and did not have was the ability to open a February failure and read it. Traces are the artefact that survives: the full message list, every tool call with its arguments and its response, the stop reason, and the scored output with the rubric's per-criterion breakdown rather than a single boolean. Storing them costs about 40 KB per run compressed, so the 1,500 runs of a five-trial comparison come to roughly 60 MB, which is a rounding error against the $200 of model spend that produced them.

With manifests stored, the reversal stops being a mystery and becomes a diff. The harness can print one, and the printed form is what goes into the meeting instead of two aggregate percentages:

$ evals diff 3c8e04b1 9f2c1a7d
model            claude-contract-2025-01-07  →  claude-contract-2025-04-22
prompts.loop.system      4b1f…8ac2  →  0d92…5b31   (7 commits)
prompts.pipeline.draft   2d5c…9e01  →  a417…cc60   (5 commits)
tool_schemas.doc_search  19de…03f7  →  7f60…22ab   (+jurisdiction, top_k 5→8)
fixtures.labels_rev      v1  →  v2                 (6 documents)
rubric.version           2  →  3                   (+defined_term_case)
pricing_table            2025-01-01  →  2025-05-01
arms.*.impl_commit       b7710e4  →  b7710e4       (unchanged)

Every line there is a hypothesis you can test one at a time, and the cost of testing one is a re-run of the two arms with a single field changed. Holding the May manifest and reverting only rubric.version to 2 took forty minutes and closed out the capitalisation theory: the loop still led, by 2.1 points instead of 3.5. Reverting the tool schema as well closed the remaining gap to 0.4 points, which is the answer to what actually happened, and it is an answer nobody could have reached from the two numbers alone.

Repeatability is not determinism, and not currency#

Pinning gets misread in two directions, and both misreadings cause real damage.

The first is expecting identical numbers. Chapter 1 established that variance survives temperature 0 because tool responses, retrieval ordering and timeouts all move; pinning does not touch any of that. A repeatable comparison is one where a re-run under the same manifest produces a gap whose interval overlaps the original's, not one that reproduces 86.3% to the decimal. If someone re-runs a pinned manifest and gets 85.9% against a recorded 86.3%, the harness is working. Treating that difference as a regression sends a team hunting for a bug in the only part of the system that was behaving correctly.

The second misreading is treating a pinned result as a current one. A manifest freezes the experiment so it can be repeated and attributed, and it does exactly nothing to keep the measured system aligned with the one serving traffic. A comparison pinned to a January model snapshot and January prompts tells you what those artefacts did; production is running May prompts against a May model, and the ranking there is an open question. Pinning and freshness pull in opposite directions, which is why the practice that works is a pinned manifest plus a scheduled re-baseline, meaning the same comparison run again from a manifest cut against whatever the system is made of that month: keep the February manifest so the February decision stays auditable, and cut a new manifest with today's model and prompts on a fixed cadence, so there is always a recent run whose inputs are also written down. Dated model snapshots are usually retired six to twelve months after release, so a manifest that is never re-cut eventually becomes unreproducible regardless of how carefully it was written.

There is also a cost worth stating plainly. Pinning tool schemas means the eval stops seeing platform changes until someone re-pins, so a tool that starts returning malformed responses in production will not show up in the harness. That is the correct trade for an architecture comparison, where the question is which topology is better under fixed conditions, and it is the wrong trade for a production regression suite, where the question is whether today's system still works. The two suites want opposite defaults and should be separate jobs.

What to do about the decision that was already made#

The February choice was made on numbers that cannot now be attributed, and the May numbers have the same defect, so neither supports flipping production. Reverting to the loop today on the strength of the May run repeats the original error with the sign changed, and the next re-run will reverse it again. The move that ends the cycle is to cut one manifest with today's model, today's prompts, today's tool schemas, the v2 labels and rubric v3, run both arms under it, and record the result with its run identifier in the design doc next to the February number, marked as measured under different conditions. If the loop wins under that manifest with an interval clear of zero, there is a case for changing the architecture, and it is a case somebody can check in November.

A pinned harness makes a result attributable and repeatable, which is a smaller claim than it sounds like, because nothing in the manifest records how each arm came to be in the state it was pinned in. Two systems can be measured under identical conditions and still be unfairly compared, if one of them has spent more of the preceding weeks being adjusted against the very fixtures the comparison uses.