Last week's transcripts versus this week's proves nothing

The slide had two columns of transcript excerpts, forty on each side. On the left, agent sessions from the week before the payments team's CLAUDE.md landed; on the right, sessions from the week after. Two engineers had read all eighty and rated each one on a four-point scale, and the right-hand column scored 3.1 against the left-hand column's 2.4. The conclusion on the final slide was that the file had made the agent's work noticeably better, and it was received well, because everyone in the room wanted it to be true and the sample looked respectable.

What else moved between those two Mondays is the whole problem. The repository took 147 commits, including a change to the root turbo.json that made turbo run test cache correctly for the first time. The sprint boundary fell in between, so the left column is mostly bug triage on the ledger service and the right column is mostly feature work on a new invoice endpoint, which are different kinds of task with different failure modes. Three of the five people writing prompts in the second week had not been writing prompts in the first, and one of them writes unusually detailed ones. The agent's underlying model version was bumped on the Wednesday. Every one of those is a plausible cause of a 0.7-point swing on a subjective scale, and the study design cannot separate any of them from the file.

The comparison is not weak because the sample is small. Eighty sessions is a decent number. It is uninformative because the treatment was never isolated: two populations differing in a dozen ways were compared, and the difference was attributed to the one that the presenter was interested in. Increasing to eight hundred sessions would produce a tighter estimate of a quantity nobody wants.

Vary one thing and hold the rest still#

The honest version of the same question fixes everything except the entry under test. You choose a commit and stay on it. You choose a task and write the prompt once. You run that task twice against that commit, once with the entry in the file and once with it removed — the two conditions are the experiment's arms, and they are what the rest of this chapter calls Arm A and Arm B — and you compare what the agent did rather than how good the result looked. What the agent did is recorded in the session's trace, the machine-readable log of every tool call the agent issued and every result it got back, which is a different artifact from the readable transcript the slide was grading.

The mechanics are ordinary version control. A worktree gives you two checkouts of the identical tree, which matters because it removes the temptation to run the second arm "later, after that other PR lands":

SHA=9f2c1a4
git worktree add /tmp/ab-with  "$SHA"
git worktree add /tmp/ab-without "$SHA"

# Arm B: replace only the entry under test, keeping the file's length
python3 - <<'PY'
import pathlib
p = pathlib.Path("/tmp/ab-without/CLAUDE.md")
src = p.read_text()
p.write_text(src.replace(
    "Run one test file: `pnpm vitest run packages/billing/src/invoice.test.ts`",
    "The billing package's tests live alongside the source files they cover.",
))
PY

for arm in with without; do
  for i in 1 2 3 4 5; do
    claude -p "$(cat prompts/proration-bug.txt)" \
      --cwd "/tmp/ab-$arm" \
      --output-format stream-json > "traces/$arm-$i.jsonl"
    git -C "/tmp/ab-$arm" reset --hard "$SHA" && git -C "/tmp/ab-$arm" clean -fd
  done
done

Two details in that snippet are doing more work than they look like they are. The reset between runs matters because an agent that has already fixed the bug will behave differently on the second attempt, and it is easy to leave a modified worktree in place and quietly measure the agent's reaction to its own previous patch. Replacing the entry with a sentence of similar length, rather than deleting it, holds the file's size and the positions of everything after it roughly constant, so that a difference in behaviour is attributable to the content of the line rather than to the fact that the rest of the file shifted 80 tokens earlier in the prompt.

Read the trace, not the patch#

Grading the final diff is what the transcript study did, and it inherits the same weakness at smaller scale: patch quality is a noisy, partly subjective aggregate of everything that happened in the session. The paired run gives you something better, because you predicted in advance which decision the entry was supposed to change, and the trace records whether that decision went the way you predicted.

For the test-command entry from the previous chapter, the predicted decision is the first command the agent runs to get test feedback on the proration fix. Across five runs per arm on commit 9f2c1a4, the traces separated completely:

The bottom row is the one you would have led with if you were grading patches, and it is the least informative row in the table. A 5-versus-4 split on five runs each is consistent with the entry doing nothing at all. The rows above it are not close: the entry changes which command the agent reaches for, on every run, and everything else in the table follows from that one change. Reporting the decision rather than the outcome is what makes five runs enough, because the decision is nearly deterministic given the prompt while the outcome depends on a dozen other things.

Deciding what counts as the predicted decision before the runs happen is what keeps the method honest, because a trace of an agent session contains dozens of things that differ between two runs and it is very easy to find one that flatters the entry after the fact. Write the predicate down first, as something a script can evaluate against the trace, and let the script report the split:

import json, pathlib

def first_test_command(trace_path):
    for line in pathlib.Path(trace_path).read_text().splitlines():
        ev = json.loads(line)
        if ev.get("type") != "tool_use" or ev["name"] != "Bash":
            continue
        cmd = ev["input"]["command"]
        if "vitest" in cmd or "turbo run test" in cmd or "jest" in cmd:
            return cmd
    return None

SCOPED = "pnpm vitest run packages/billing/src/invoice.test.ts"
for arm in ("with", "without"):
    hits = sum(first_test_command(f"traces/{arm}-{i}.jsonl") == SCOPED
               for i in range(1, 6))
    print(f"{arm}: {hits}/5 ran the scoped command")

Writing the predicate as code has a second payoff that shows up two chapters from now: a predicate over a trace is the same object as a regression check, so an experiment you ran once to justify an entry can be kept and re-run whenever the entry or the repository changes.

Five runs per arm is not a large number and it is not chosen by a power calculation. It is chosen because a clean 5–0 split is already strong evidence and a 3–2 split is already the answer you needed: an entry whose presence changes the decision on three runs out of five is not reliably changing anything, and no amount of additional sampling turns it into an entry worth its permanent place in the prompt. The interesting failures of this method are not statistical, they are about which task you chose.

Choosing the task the entry claims to fire on#

The import ban from the previous chapter's ranking — billing must never import from internal/experiments — will show no effect at all on the proration task, because nothing in that task goes anywhere near internal/experiments. Running the paired comparison on a randomly chosen ticket and reporting a null result would be a straightforward mistake, and it is the mistake that makes ablation studies of config files look inconclusive, an ablation being a run with one component taken out to see what its absence changes.

An entry that fires rarely needs the task built for it. In this repository, the honest fixture — a fixture being a task chosen or written so that it puts the agent at the fork the entry exists to control — is a ticket that requires exponential backoff on the invoice webhook sender, where the only existing backoff helper in the monorepo lives at internal/experiments/retry/backoff.ts and is genuinely the most convenient thing to import. That task puts the agent at the exact fork the entry exists to control. On five runs per arm, Arm B imported the experimental helper in 4/5 and wrote a local one in 1/5; Arm A wrote a local helper in 5/5 and, in 3 of those, said in its summary that the experiments package was off-limits for billing. That is a smaller and rarer effect than the test command, and it is a real one.

Building the fixture is also a filter on the entry itself. If you cannot describe a task on which the entry would change a decision, you have discovered something about the entry rather than something about your imagination, and the previous chapter's ranking already told you what to do with it. The forty-line files that nobody can defend are exactly the files whose entries have no constructible fixture.

What the paired run does not settle#

It does not generalise past the task you ran. Five runs on the proration bug establish that the test-command entry changes the first command on that task; the claim that it does so across the team's work is an inference from how common that task shape is, and it is a reasonable inference, but it is not what you measured. When someone challenges an entry, the useful move is to add a second fixture from a different corner of the repository rather than to add more runs of the first.

It says nothing about cost. A paired run tells you an entry changes behaviour, and a file of forty entries that each change behaviour on their own fixture can still be too expensive to load on every session, because the cost is paid jointly and the benefits are paid one entry at a time. Nothing in this method surfaces that trade-off.

The wall-clock numbers in the table above also need reading with some suspicion, for a reason that has nothing to do with the agent. The two arms have different prompt prefixes, so they have different prefix-cache behaviour — providers keep the already-processed front of a prompt for a short window, so a later call whose opening tokens are identical is read back from that cache instead of recomputed — and a run whose prefix is already cached starts producing tokens sooner than one whose prefix is not. Those differences are on the order of a few hundred milliseconds while the effect being measured here is on the order of minutes, so the table survives, but the same design used to compare two phrasings of the same entry would be measuring cache state as much as anything else. Either run both arms cold, or treat latency from these runs as a rough indicator and measure it separately.

There is also a class of entry the method handles badly on principle. An entry whose purpose is to prevent a rare catastrophe — the one that stops a destructive command from being run against production — has, on any task you can safely construct, exactly the same trace signature as an entry that does nothing, because the dangerous branch is one you are not going to take in a test harness. You can build a fixture that gets close to the branch and observe what the agent says it is about to do, and that is genuinely useful, though it is weaker evidence than the clean split on the test command, and the sixth chapter argues that entries of this kind should not be relying on standing text in the first place.

It is also blind to effects that only appear on long sessions. A fixture task that resolves in six turns will not reveal a file that becomes a problem at turn forty, when the session's context is under pressure and the resident text is competing with everything the agent has read. Symptoms of that kind — the agent forgetting a constraint it was told about twenty turns ago, or re-reading a file it already read — do not appear in a short paired run at all.

Finally, the method measures the file as it is written, not as it will be. The comparison you ran this morning was against the tree at 9f2c1a4; the entry can stop being true tomorrow without any of these traces changing, because the traces are a record and not a check. Anthropic's short explainer The CLAUDE.md file is clear that the file loads on every session; nothing in a repository loads a verification of that file on any session, which is the gap the fourth chapter of this topic is about.

The immediate next question is the one the cost objection raises. Having established that a handful of entries genuinely move decisions, you still owe an account of what carrying the whole file costs on every session — and the obvious way to price it produces a number so small that it ends the conversation in the wrong place.