A command nobody executes stays wrong for four months

The trace is from a Tuesday in July, on a ticket to fix rounding in applyProration:

[turn 14] Bash: yarn test
  → /bin/sh: yarn: command not found        (exit 127)
[turn 15] Bash: yarn --version
  → /bin/sh: yarn: command not found        (exit 127)
[turn 16] assistant:
  The repository's documented test command (`yarn test`) isn't available here —
  yarn doesn't appear to be installed in this environment. I've made the change
  to applyProration and reviewed the surrounding code carefully to confirm the
  arithmetic is right. You'll want to run the suite locally before merging.

The repository migrated to pnpm on 14 March. The line in CLAUDE.md that says the test command is yarn test was written the previous November and had not been touched since. Between the migration and the day someone finally read a trace closely, 410 sessions ran against that file; grepping the trace archive for exit code 127 on a yarn invocation found it in 63 of them, and in 41 of those the session ended without any test having been run. Three of those 41 shipped changes that broke tests in CI, which is where the cost eventually landed, four months and several hundred dollars of engineer-hours downstream of a one-word edit.

Why the wrong line survived#

The interesting question is not how the line became wrong, which is obvious, but why nothing surfaced it for four months while every other kind of staleness in that repository surfaced within days. A broken import fails the type check. A renamed script fails the build. A stale lockfile fails pnpm install --frozen-lockfile. A wrong path in a test fails the test. The repository is full of machinery whose entire job is to fail when a claim about the repository stops being true.

The config file is the one checked-in artifact that nothing in the repository depends on. Nothing imports it, nothing parses it, no job reads it, and its only consumer is a process that runs outside CI and never reports back. It has the same relationship to the codebase that a comment has, with the difference that comments are read by people who are looking at the code beside them and will notice the mismatch.

The consumer is also the wrong kind of reader for keeping documentation honest. A new engineer who follows a stale README hits the error, swears, and either fixes the README or complains in a channel, and either way the wrongness becomes visible within an hour. An agent that hits the same error writes a fluent paragraph explaining that the environment appears to be misconfigured and proceeds, because proceeding usefully in a degraded environment is generally the behaviour you want from it. That paragraph in turn reads as a plausible environmental note to whoever skims the session summary, so the failure is not merely silent, it is actively camouflaged by a reasonable-sounding explanation. Silent compensation is the characteristic failure mode of stale standing instructions, and it is why the count of affected sessions reached 63 without anybody filing anything.

Compensation does not always look like giving up, either, which is what made the pattern hard to spot even for people who did read summaries. In 22 of the 63 sessions the agent recovered on its own: it saw the failure, went looking for the real package manager, found pnpm-lock.yaml, and ran pnpm test, which worked. Those sessions produced correct work and a clean-looking summary, having spent three or four turns and roughly 9,000 tokens rediscovering something the file claimed to be telling them. A stale entry with a competent reader behind it degrades into an expensive no-op rather than an obvious break, and an expensive no-op generates no signal at all.

Reproducing it on purpose before touching the line#

The temptation on finding a trace like this is to fix the word and move on, which loses the two things that make the incident worth anything: a dated account of how long the entry was wrong, and a reproducible failure you can point a check at.

Dating it is version control. The migration commit is whatever last changed the package manager declaration, and the entry's age is whatever last changed that line of the config:

$ git log -1 --format='%h %cs' -S'"packageManager"' -- package.json
a41e07c 2025-03-14

$ git log -1 --format='%h %cs' -L'/yarn test/,+1:CLAUDE.md'
c8d3915 2024-11-02

The gap between those two dates is the window in which the file was making a false claim on every session, and it is the number that turns "the file was a bit out of date" into a specific 122-day exposure that a team will act on. Counting the affected sessions needs a trace archive, and if the team keeps one, a single pass over it gives the 63 and the 41 quoted above; if the team keeps no traces, the honest thing to report is that the exposure cannot be measured, which is itself a finding worth raising.

Reproducing it takes the harness from the second chapter, pointed at the commit before the fix. Run the proration fixture on 9f2c1a4 five times with the file as it stood, and the exit-127 path appears in 5/5; run it with the single word corrected and it disappears in 5/5. That pair of results is the regression check in embryo, and it is worth keeping as a trace predicate alongside the CI job, because the CI job checks that the command resolves while the predicate checks that the agent actually reaches for it. The two fail on different days.

Making the repository execute its own file#

Correcting yarn to pnpm fixes one line and leaves the mechanism that produced it entirely intact, so the next stale command will also survive four months. The fix that changes the mechanism is to give the file a consumer inside CI.

The convention that makes this cheap is to require that every command the file names appears inside a fenced bash block, rather than inline in prose, so an extractor can find them without parsing English:

# scripts/verify_config_commands.py
import re, shutil, subprocess, sys, pathlib

FENCE = re.compile(r"^`{3}bash\n(.*?)^`{3}", re.S | re.M)

def commands(path: pathlib.Path) -> list[str]:
    text = path.read_text()
    return [line.strip()
            for block in FENCE.findall(text)
            for line in block.splitlines()
            if line.strip() and not line.strip().startswith("#")]

def resolvable(cmd: str) -> tuple[bool, str]:
    binary = cmd.split()[0]
    if shutil.which(binary) is None:
        return False, f"binary not found: {binary}"
    probe = subprocess.run(["pnpm", "run", "--silent", "--if-present", "--", "true"],
                           capture_output=True)
    if probe.returncode not in (0, 1):
        return False, "package manager not usable in this checkout"
    return True, ""

failures = []
for cmd in commands(pathlib.Path("CLAUDE.md")):
    ok, why = resolvable(cmd)
    if not ok:
        failures.append(f"{cmd!r}: {why}")

for f in failures:
    print(f"CONFIG COMMAND FAILS IN A CLEAN CHECKOUT: {f}", file=sys.stderr)
sys.exit(1 if failures else 0)

The phrase that carries the weight is clean checkout. Running the check on a developer machine where yarn happens to be installed globally proves nothing, because the environment the agent gets is a fresh container with whatever the repository declares and nothing else. The job therefore has to run in CI, with no caching of global binaries:

name: config-commands
on:
  pull_request:
  schedule:
    - cron: "0 6 * * *"
jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with: { node-version-file: .nvmrc }
      - run: pnpm install --frozen-lockfile
      - run: python3 scripts/verify_config_commands.py
      - name: Execute the commands for real
        if: github.event_name == 'schedule'
        run: python3 scripts/verify_config_commands.py --execute

Two tiers are doing different jobs. Resolution checking is fast enough — about 4 seconds on this repository — to run on every pull request, and it catches the whole yarn-to-pnpm class. Actually executing the commands takes 6 minutes 20 seconds because it runs the test commands and the codegen, meaning the generators that write source files from a schema rather than from a keyboard, and it catches the subtler case where the binary exists and the script it invokes has been renamed. Running the expensive tier nightly rather than per-PR means a genuinely broken command is wrong for at most a day instead of four months, which is the improvement that matters; per-PR execution buys hours off that and costs six minutes on every PR.

Not every command the file names can be executed, which is the first objection anyone raises and it has a clean answer. pnpm gen:api rewrites files under generated/, so running it in CI dirties the checkout; a deploy command has side effects nobody wants on a cron; a backfill command must never run anywhere near the production database. For each of those, the executable tier degrades to something weaker and still worthwhile: assert that the script name exists in package.json, run the command with its own dry-run flag where it has one, and run the codegen in a scratch worktree and diff rather than commit. The rule that keeps this from becoming a set of exceptions is that a command with no safe way to be checked is a command the file should describe by pointing at the runner rather than by quoting an invocation, which is a thread the sixth chapter picks up in a more serious form.

It is also worth running this job in the same workflow as the token budget from the previous chapter, under one name like config-health, because the two failures have the same owner and the same remedy shape. A single required check that says the standing instructions are within budget and currently true is a thing a team will keep green; two separate advisory jobs are two things that go orange and get ignored.

The entries a command check cannot reach#

Executable entries are the easy half. The other kinds of entry in the file need their own checks, and the checks are different enough that treating them as one problem produces a job that verifies the cheap thing and declares victory.

The last row is the honest one. That entry names no command to resolve and no path to assert; every noun in it still exists; it parses, it reads as sound engineering practice, and it can be wholly obsolete without a single character of it becoming false in a way a script could detect. Nothing in this chapter's machinery touches it.

Where the check stops helping#

Executing a command proves it runs, and proves nothing about whether it is the command the agent should be running. pnpm test resolves cleanly, executes cleanly, and runs all fourteen packages for four minutes; it would pass every check in this chapter while quietly reintroducing the slow feedback loop the first chapter's entry existed to avoid. Correctness of that kind is what the paired run from the second chapter measures, and the two checks are complements: CI tells you the file is not lying about the repository, the paired run tells you the file is changing the decision you wanted changed.

There is also a way to satisfy the check by making the file emptier. A team under pressure from a red config-commands job can delete the commands section rather than fix it, and both this check and the budget check from the previous chapter will go green on a file that has stopped saying anything. Neither script can distinguish a file that is correct from a file that has been hollowed out, which is a general property of checks that verify claims rather than requiring them.

The deeper limitation is the one in the last table row. An instruction can be stale in a sense no extractor reaches: written for a workflow that the tooling has since replaced, so that following it literally is either impossible or a no-op, while every word in it remains individually true. That failure needs a different kind of investigation and, as it turns out, a different kind of remedy, and this topic gets to it two chapters from here.

What comes first is a failure that is not about truth at all. Two entries can both be current, both be precise enough to execute, both pass every check in this chapter, and still produce behaviour that varies from run to run — because they both apply to the same action and the file never says which one wins.