Six weeks of tuning fits the arm with more places to patch

Six weeks in, the sprint review slide read 91.2% for the pipeline and 86.4% for the loop over 120 tasks, five trials each, run under a single pinned manifest. Both arms had started the six weeks near 79%. Both had been worked on by the same two engineers, alternating, in the same room, with the same rule: pick the worst-scoring fixture, read its trace, fix what you find, re-run. Nobody had touched one arm on the sly. The interval on the paired gap was +4.8 points with bounds at +2.6 and +7.0, which is the first time in this comparison a gap has cleared zero by a comfortable margin, and the request in the room was to ship the pipeline in the current sprint and close the investigation.

The result is not fabricated and the process was not unfair in any sense the team would recognise. It is still not evidence that the pipeline is the better topology, because the two arms were not equally exposed to the thing six weeks of trace-driven iteration does to a system, which is overfitting: absorbing the particular examples you tuned against, so that the score on those examples overstates how the system will do on anything else.

Effort was equal; surface area was not#

Count what each arm gives an engineer to adjust when a fixture fails. The loop has one system prompt, one tool schema set and a stopping rule. The pipeline has three system prompts, two boundary contracts describing what one stage passes to the next, a routing condition that decides whether the normalisation stage runs at all, and a retry policy on the extraction stage. Every one of those is a place where a sentence can be added that happens to resolve the fixture in front of you.

The commit log over the six weeks makes the asymmetry concrete. Both arms received continuous attention; the pipeline had more to receive it with.

The last three rows are the ones that matter. A clause such as "when the agreement is an amendment referencing a prior master agreement, resolve defined terms against the master before normalising" is a legitimate instruction that describes a real category of contract, and it also encodes something the engineer learned from fixtures 037, 041 and 092. Thirteen such clauses, each derived from between one and four documents in a set of 120, is a system that has absorbed a substantial fraction of that set into its own text. The loop absorbed two.

The statistical name for this is degrees of freedom, and the mechanism is the one that made train/test splits standard practice in statistics long before machine learning existed. A model with more free parameters can fit more of the noise in the data it was fitted on, so its performance on that data is a biased estimate of its performance elsewhere, and the bias grows with the number of parameters. Kaggle made the effect legible to a wide audience with its two-leaderboard design: competitors tune against a public scoreboard and are ranked on a private split they never saw, and the reshuffling between the two is routinely large enough to move a top-ten team out of the top hundred. Prompt clauses are not parameters in a gradient-descent sense, and they behave like parameters for this purpose, because the fitting procedure is the same — an optimiser (an engineer) adjusting a system in response to measured error on a fixed set.

Two consequences follow that people find counterintuitive. Adding fixtures to the tuning set does not fix it, because the additional tasks get tuned against too; a 400-task tuning set produces a better-fitted system and the same biased estimate. And the bias is not visible in the tuning-set numbers by inspection, since a genuinely better architecture and a more thoroughly fitted one both show up as a rising line on the same chart.

Buying a held-out answer with what is already on hand#

The answer is a held-out set: tasks deliberately kept outside the tuning loop, never read, never scored during the six weeks, so that a run against them estimates how each arm behaves on documents it was not shaped by. All 120 fixtures are contaminated in that sense, having influenced both systems through every trace anyone opened, so the question is where a clean set comes from inside a week without a new labelling project. Three sources were available, and they are not equally good.

The first option is tempting because it costs nothing, and it is the weakest, since a prompt clause written after reading fixture 037 helps every amendment in the set including the ones nobody opened. Holding out tasks after the fact only works if the held-out tasks were invisible during tuning, and these were visible in every aggregate score the team watched.

The team took the third option, with the second as a check. Sixty contracts that arrived in the two weeks after the tuning window opened were pulled from the archive, sampled to match the tuning set's mix of document types, and labelled by the same two engineers using the rubric at its current version. Establishing which fixtures were even eligible took a script rather than a memory, because six weeks of commits and trace reviews are not something anyone recalls accurately:

def contaminated(fixture_ids: set[str], repo: Path, since: str) -> set[str]:
    """Any fixture whose id appears in a commit, a prompt, or a review note."""
    log = subprocess.run(
        ["git", "log", f"--since={since}", "-p", "--", "prompts/", "agents/"],
        cwd=repo, capture_output=True, text=True).stdout
    notes = (repo / "eval/trace-reviews").read_text()
    haystack = log + notes
    return {fid for fid in fixture_ids if fid in haystack}

The run itself reused the February design without modification: both arms, five trials per task, jobs interleaved, one pinned manifest covering model snapshot, prompts as of the end of the six weeks, tool schemas, the new fixture set and rubric v3. Sixty tasks at five trials is 600 runs, about $80 of model spend and twenty-five minutes of wall clock.

The drop, and which arm took it#

Both arms are expected to drop a little on unseen data, and the informative quantity is the difference between the drops. The loop lost eight tenths of a point, which is inside the noise of the measurement. The pipeline lost four and a half, and the shape of that loss is exactly what the commit table predicted: the arm carrying thirteen document-pattern clauses lost most on documents those clauses were not written for. Reading the held-out failures confirms the mechanism — nine of the pipeline's held-out failures involve document types with no matching clause, where the normalisation stage applies a general rule that a clause elsewhere in the prompt had been overriding for the tuning-set cases.

The remaining +1.1 point gap is a real point estimate and its interval spans zero, so the honest reading is that on unseen documents the two topologies score about the same, with the pipeline possibly a point ahead and possibly a point behind. Against that, the pipeline costs 2.05× per task and 1.7× at p95 latency, both of which were measured on the same runs and neither of which is inside anyone's noise. Six weeks of work did produce a genuinely better system than the one the team started with, since both arms rose about seven points from their 79% baseline on the tuning set and the loop carried nearly all of its gain onto documents it had never seen.

Presenting this to people who spent six weeks on it#

The finding is about the measurement, and the failure mode when delivering it is to let it sound like a finding about the people. Three things are true at once and all three belong in the write-up: the six weeks improved both systems substantially, the tuning-set gap was an artefact of unequal fitting surface rather than of anyone's carelessness, and the held-out numbers make the architectural question a cost question rather than a quality one. The specific fixes that generalised are worth naming individually, because several of the pipeline's prompt clauses describe real contract categories and can be lifted into the loop's single prompt, where they cost nothing extra to maintain. That reframes the six weeks as having produced portable domain knowledge under an experimental design that could not tell portable from fitted, which is a description of the situation rather than a consolation.

The recommendation at a 1.1-point gap is the loop, on the grounds that the quality difference is not measurable and the cost and latency differences are, with the pipeline's generalisable clauses ported across and a re-run to confirm the port did not cost anything. Presenting this alongside the tuning-set slide rather than instead of it matters, because the team will otherwise remember one number and a reversal, which is the pattern the previous chapter's manifests exist to break.

What holding out does not do#

A held-out set is a consumable. The moment somebody reads a held-out failure trace and edits a prompt in response, those sixty documents join the tuning set and the next held-out result will be biased in the same way. Teams that use one for a few cycles end up with numbers that drift back toward the tuning-set optimism, usually without noticing the moment it happened. The usable discipline is to hold the set closed except at decision points, budget how many times it will be opened before it is retired, and label a replacement on a schedule rather than in a panic.

Holding out also does not detect the problem this chapter is about; it measures it after the fact. If both arms had shown a 4.5-point drop, the conclusion would have been that the fixture set is unrepresentative of arriving traffic, which is a different problem with a different fix. And it does nothing about the case where an architecture is genuinely worse for a reason unrelated to fitting, which is a distinct question that no amount of splitting the data will answer.

The held-out numbers say the loop is at least as good as the pipeline and cheaper, and they say nothing about why. A gap that comes from one specific place in the pipeline is a bug with a fix; a gap spread across the whole design is a property of the topology. The traces distinguish those two, and mistaking the first for the second is how a team ends up with an architectural conviction it can never revisit.