Two rules for one action, and the model arbitrates differently each run

The ticket is small: add an idempotency_key field to the invoice create request in openapi/invoices.yaml, which is the OpenAPI specification of the service — a machine-readable description of every endpoint and its request and response shapes, from which the team's TypeScript client is generated rather than written — and make that generated client carry the new field. Running it eight times on the same frozen commit with the same prompt produced five distinct behaviours.

Three correct out of eight, on a task with no genuine difficulty in it, against a repository where the correct sequence is two commands long. The cause is two lines of the config file that are forty lines apart and were written seven weeks apart by different people:

## Conventions

- Follow the existing structure of each package.
- Never edit files under `generated/`.
- Keep migrations reversible.

## API changes

- The OpenAPI specs in `openapi/` are the source of truth.
- After changing the OpenAPI schema, regenerate and commit the client in `generated/api/`.

Both entries are precise. Both would pass the executability test from the first chapter, both are currently true statements about how the team works, and both would survive every check the previous chapter built. They collide on exactly one action, which is the action this task requires.

Why the behaviour scatters#

The model is not choosing to disobey. Every one of the eight runs followed at least one of the two entries faithfully; run 2 obeyed the conventions section, run 5 obeyed neither by trying to satisfy the API section's outcome without its mechanism, run 3 obeyed both in sequence. When two instructions apply to one action and the text states no relationship between them, resolving the conflict becomes part of the generation, and generation is sampled: the model draws each token from a probability distribution rather than picking the single likeliest one, so a prompt that leaves two readings genuinely open gets a fresh draw on every run.

Small, irrelevant things tilt the arbitration, which is what makes the failure so hard to attribute in the field. A prompt that says "update the schema" leans toward the conventions section; a prompt that says "ship the idempotency key end to end" leans toward the API section. The order of the two entries in the file matters, and so does which section heading the model reads last. None of that is stable across sessions, so the same team sees the same task go three different ways in a week and concludes that the agent is unreliable.

That conclusion is expensive in a specific way, because it points remediation at the wrong layer. A team that believes the variance is intrinsic responds by adding review steps, by asking engineers to spell out the codegen step in every prompt, or by abandoning agent assistance for schema work altogether — three responses that each cost real time and none of which touch the two lines producing the behaviour. The distinguishing test is cheap: run the fixture eight times and look at whether the failures are varied-but-each-internally-coherent, which is the signature of arbitration, or whether they are simply sloppy in different ways, which is not. Runs 2, 3 and 5 above are each a defensible reading of the file. Nothing in them looks like a model failing to follow instructions.

The obvious remedy is emphasis, and it is worth doing once to see what it buys. Rewriting line 18 as **NEVER** edit files under \generated/` — this is a hard rule, no exceptionsand re-running the eight moved the distribution to 1 correct, 6 refusals and 1 hand-edit. The emphasis worked, in the sense that it made one side win reliably; it made the outcome worse because the side that won was the wrong one for this task. Emphasising both entries returned the split to roughly where it started, at 4 correct out of 8. The escalation also leaked: in three subsequent sessions on unrelated tickets, the agent declined to runpnpm gen:sqlbecause its output lands undergenerated/db/`, which is a regression caused entirely by the strengthened wording.

Resolving it in the text#

The fix is to leave the model nothing to arbitrate. There is one section about generated/, it states the general rule, and the general rule carries its own exception rather than delegating the exception to a section forty lines away:

## Generated code

Everything under `generated/` is written by a generator and never edited by hand.
The generators are `pnpm gen:api` (writes `generated/api/` from `openapi/*.yaml`)
and `pnpm gen:sql` (writes `generated/db/` from `db/schema.sql`). When you change
one of those inputs, run its generator and commit the output in the same commit
as the input. `pnpm check:generated` fails CI when an input and its output disagree.

Four properties of that rewrite carry the improvement, and each of them is worth stating as a rule for the next conflict.

One action gets one location. If two sections need to talk about generated/, that is the signal that the file has one topic split across two headings, and the split is what allowed the entries to be written independently in the first place. Consolidation is not a tidiness exercise here, it is what removes the possibility of independent drift.

The general rule states the exception, rather than the exception stating itself elsewhere. This asymmetry matters because of how the two are read: an agent that reads only the general rule must not come away with a wrong picture. Putting "regenerate and commit" only in the API section means the conventions section is, on its own, misleading, and there is no guarantee about which parts of a file dominate on any given turn.

The rule names a mechanism rather than a prohibition. "Never edit" tells the agent what not to do and leaves regenerate-and-commit unclassified, so the agent has to decide whether running a generator counts as editing. "Written by a generator and never edited by hand" answers that question in the same breath as asking it.

The rule ends at something checkable. pnpm check:generated gives the boundary a consequence outside the text, which both tells the agent how the claim is verified and gives the team a failure mode that does not depend on any instruction being read.

Re-running the eight after the rewrite gave 8/8 correct, and the gen:sql regression disappeared with the emphasis that caused it.

Turning the reproduction into a check#

The eight runs are already most of a regression test. The predicate from the second chapter is the missing piece: a script that reads a trace and answers one question about it, in this case whether the agent ran the generator and committed its output.

# tests/agent/test_generated_code_rule.py
import json, pathlib, pytest

FIXTURE = "prompts/add-idempotency-key.txt"

def actions(trace: pathlib.Path):
    return [json.loads(l) for l in trace.read_text().splitlines()]

@pytest.mark.parametrize("run", range(5))
def test_schema_change_regenerates_and_commits(agent_run, run):
    trace = agent_run(FIXTURE, commit="9f2c1a4")
    calls = [e["input"]["command"] for e in actions(trace)
             if e.get("type") == "tool_use" and e["name"] == "Bash"]
    assert any("gen:api" in c for c in calls), "generator never ran"
    committed = agent_run.changed_paths(trace)
    assert any(p.startswith("generated/api/") for p in committed), \
        "generated client not committed alongside the schema"
    assert not agent_run.hand_edited(trace, under="generated/"), \
        "generated file written directly"

Five runs at roughly 90 seconds each is seven and a half minutes and a few dollars per execution, which is too slow for a pre-merge check and reasonable as a nightly job alongside the command verification. The value of keeping it is that the next person to add a line about generated/ finds out within a day that they have reopened the conflict, which is the only mechanism that stops the file from re-acquiring the shape it just lost.

Finding the next one before it fires#

Waiting for a scattered task to expose each conflict is slow, and there is a cheap sweep that catches most of them. Group the entries by the action they govern rather than by the heading they live under. Every entry that mentions a path, a command, or a package is making a claim about some operation, and two entries that name the same noun are candidates whether or not they sit in the same section:

# Which nouns does the file talk about more than once?
grep -oE '`[^`]+`' CLAUDE.md | sort | uniq -c | sort -rn | head -20

On the payments team's file that produced four repeated nouns: generated/, pnpm test, packages/billing, and migrations/. Three of the four turned out to be redundancy — two entries agreeing about migrations in slightly different words, which is a budget problem rather than a correctness one. The fourth was the conflict this chapter opens with, and it took about ninety seconds to find once the question was asked in that form. The sweep misses conflicts phrased without a shared token, such as a rule about "the database schema" colliding with one about db/schema.sql, which is an argument for naming things the same way throughout the file for reasons that have nothing to do with style.

The other cheap sweep is over the file's history. An entry added by someone who did not read the whole file is the normal way conflicts arrive, so a review rule that any addition mentioning a path already present in the file must edit the existing section rather than open a new one prevents most of them at close to zero cost.

Conflicts that are invisible in any one file#

Not every repetition is a conflict. Two entries that say compatible things about the same topic are redundancy, which costs tokens and is the budget's problem rather than this chapter's. A conflict requires two directives over the same action with no stated relationship, and the diagnostic is whether you can construct a task where following one means not following the other.

The harder conflicts are the ones spread across layers, because no single file looks wrong.

The personal layer produces the worst incidents, because a conflict that lives half in a version-controlled file and half in someone's home directory reproduces on exactly one machine. When behaviour differs between two engineers on the same commit and the same prompt, that asymmetry is the first thing to check, and the check is to run the fixture in a clean container with no personal config and see which behaviour it matches. The cross-tool row is a growing version of the same problem, and the open AGENTS.md format exists so that one file can serve several agent tools rather than each accumulating its own; where a repository does keep both, the useful arrangement is one real file and a symlink, so that they cannot disagree.

Resolving a conflict in the text raises reliability on this task from 3/8 to 8/8, and 8/8 is not a guarantee. The rewritten entry is still standing text competing with everything else in the context, and a longer session or an unusual prompt can still produce the ninth run that hand-edits a generated file. For a mistake whose worst outcome is a confusing pull request that CI rejects, that residual is acceptable. The next chapter takes an action where it is not.