A layer can lose one instruction and survive on another

Someone at Acme ran the obvious experiment. They put "format Python with 4-space indentation" in their personal instruction file, "format Python with 2-space indentation" in the repository's, opened a session, asked for a small function, and got two spaces. They wrote a line in the team wiki — project layer overrides user layer — and everyone believed it for four months, because it was true about that instruction and it was stated as a fact about layers.

The wiki line broke on a Thursday when an engineer noticed the agent was writing commit messages in a style nobody on the project had asked for, in a repository whose instruction file specifies a completely different style. The style came from her user layer. The project layer had a commit-message rule too, and it was present, and it lost. So the project layer had won on indentation and lost on commit messages, in the same session, against the same user layer, which is not a thing a sentence with the word "overrides" in it can describe.

The question a precedence test actually has to answer#

Two schemes produce very different worlds, and the four-space experiment cannot tell them apart because it only ever tested one instruction.

Under wholesale replacement, the presence of a project-layer file suppresses the user layer entirely. Everything in the user file is gone for sessions in that repository — its indentation preference, its commit style, its rule about never using git push --force, all of it — and the observed two spaces is a side effect of the user layer being absent rather than of it being outranked. This scheme is easy to reason about and brutal in practice, since a single project file silently disables forty personal preferences that had nothing to do with the project.

Under field-by-field merging, both layers contribute, and the resolution happens per instruction. The project's indentation rule and the user's indentation rule are both about indentation, and something decides between them; the user's commit-message rule may face a project rule, or may face nothing at all, and it survives in either case. A layer can therefore lose one instruction and survive on another, which is exactly what Acme observed and exactly what "overrides" cannot express.

There is a third possibility that looks like merging and behaves worse, which is that both texts are simply concatenated and nothing decides anything until generation time. The two indentation rules coexist in context and the model picks; it picked two spaces four times in a row on a formatting question where the nearer, more specific instruction is a reasonable read, and it would pick differently on a prompt phrased differently. The test result — two spaces — is identical in this world and in the merging world, which is why a single trial of a single instruction is not a measurement.

Distinguishing the first from the other two takes one observation. Distinguishing the second from the third takes repetition, because their difference is variance rather than value.

Run it with values you can recognise#

The design principle is that every instruction in the test should produce an output token you can find with your eyes, and no two instructions should produce the same one. Numbers work well; so do invented names. Vague qualities — "write clean code", "be concise" — produce results you have to interpret, and interpretation is where a test stops being a test.

Three layers, four subjects, and a deliberate asymmetry: two subjects appear in more than one layer, and two appear in exactly one.

<!-- ~/.claude/CLAUDE.md — user layer -->
- Wrap lines at 73 characters.
- Name test files `*.spec.py`.
- Start every commit message with `chore(personal):`.

<!-- platform/CLAUDE.md — project layer -->
- Wrap lines at 91 characters.
- Name test files `test_*.py`.
- Every module gets a docstring ending with the marker `PROJECT-DOC`.

<!-- org policy — organisation layer -->
- Wrap lines at 108 characters.
- Every file gets the header comment `# (c) Acme, internal use only`.

Line width is contested by all three layers with three distinct values, so the winning number names the winning layer with no inference required. Test-file naming is contested by two. The commit prefix exists only in the user layer and the docstring marker only in the project layer, and those two are the ones that answer the question the four-space experiment could not: if chore(personal): appears in the output while the line width came from the project file, the user layer lost one instruction and survived on another, and the scheme is a merge rather than a replacement.

Then the run itself. Ask for something that exercises every subject at once, from a clean session, and repeat it.

$ cd ~/work/platform
$ for i in 1 2 3 4 5; do
    claude -p "Create src/billing/retry.py with a function retry_with_backoff,
               a test file for it, and commit the result." \
      --output-format json | jq -r '.result' > /tmp/run-$i.txt
  done
$ grep -ho 'test_[a-z]*\.py\|[a-z_]*\.spec\.py' /tmp/run-*.txt | sort | uniq -c
   4 test_retry.py
   1 retry.spec.py
$ grep -c 'PROJECT-DOC' /tmp/run-*.txt | tr '\n' ' '
/tmp/run-1.txt:1 /tmp/run-2.txt:1 /tmp/run-3.txt:1 /tmp/run-4.txt:1 /tmp/run-5.txt:1
$ grep -ho 'chore(personal)' /tmp/run-*.txt | wc -l
   5

The user layer is present, since chore(personal): survived in all five runs on a subject nothing else in context mentions. Wholesale replacement is ruled out. The test-file naming went to the project layer in four runs out of five, which is a majority rather than a rule, and the one deviation is the entire finding — a loader that resolves per field produces five out of five, because the losing instruction was never in the text. Four out of five means both strings reached the model and the model arbitrated, so the scheme here is concatenation with the resolution left to inference.

The line width tells the third part of the story. Measuring it needs a slightly careful eye, because a model asked to wrap at 91 characters produces a longest line of 84 or 88 rather than exactly 91, so read the distribution rather than the value.

The last row is the one worth staring at. The organisation's copyright header, uncontested by any other layer, appeared in three runs out of five. Nothing outranked it and nothing contradicted it; it was 1 line inside 1,180 tokens of policy text that the model had no particular reason to prioritise while writing a retry helper. Precedence had nothing to do with that outcome, and a team that only ever tests contested subjects will never see it.

Record the table, not the conclusion. The wiki line that started this chapter was a conclusion — five words with no date, no host version and no indication of what had been tested — and it survived four months precisely because it was too short to argue with. A committed file at docs/agent-precedence.md holding the five layer files verbatim, the exact prompt, the host version, the date and the observed counts is maybe forty lines, costs one afternoon, and lets the next person re-run it rather than re-derive it. It also makes the failure mode legible: when someone later observes the user layer winning on test names, the recorded 4-of-5 tells them the behaviour was never deterministic, instead of sending them looking for what changed.

Two practical notes on running it. Do the whole thing on a scratch branch, since the test deliberately puts contradictory rules into files that other people's sessions read, and a colleague who starts work during your twenty-minute experiment will get genuinely strange results. And use invented markers such as PROJECT-DOC rather than plausible real conventions, because a model asked for a docstring will often write one anyway, and a subject whose absence you cannot distinguish from the model's default behaviour contributes nothing to the measurement.

Prose does not overwrite prose#

Acme's security team ships an org-layer instruction: any change touching authentication code must be flagged for security review before merge. A product team, tired of a security ticket on every session-timeout tweak, adds a line to their project file saying that authentication changes in this repository are reviewed by the platform team and do not need a security flag. The org rule stops applying. The team reports, accurately, that a project-layer sentence silently disabled an organisation-level control.

What produced it is almost certainly the third scheme rather than the first. Both sentences are in the assembled context. The project sentence is nearer, more specific, mentions this repository by name, and reads like an exception written by someone who knew about the general rule, which is precisely the shape of text a model treats as governing. No component decided that the project layer outranks the organisation layer. The instructions were placed next to each other and the more specific one usually wins an argument it was never formally entered into.

Whether this is a bug depends on what the host documents. If it documents deep-merge — the field-by-field scheme above, where each layer contributes what the layers below it do not already say — with local specificity preferred, the behaviour is the product working as designed and the defect is in the assumption that org-layer text is a control. If it documents that organisation policy is authoritative, this is a real bug and the assembled view from chapter 2 is your evidence, because it shows both strings present with no enforcement between them.

The distinction that matters for the security team is between a rule and a control. Their org line is a rule: it changes what the agent tends to do, at a cost of zero and a coverage of every repository at the company, and it will be followed on most runs by most people. It is not a control, because a control is something that holds when someone is actively working against it, and every mechanism in this chapter is one sentence away from being outvoted by another sentence. Telling them "we'll move it to the end of the context so it wins" trades an honest weak guarantee for a dishonest strong-sounding one.

Where the analogy to config merging breaks#

Engineers reach for tsconfig.json extends chains, Kubernetes kubeconfig merging, or Helm values files, and those are the right intuition for the shape of the problem and the wrong intuition for the reliability of the answer. In those systems the merge is a pure function of the inputs: you can compute the effective value offline, the tooling can print it, and running it twice gives the same result twice. That is what makes kubectl config view --merge a proof.

Here, part of the merge is performed at generation time by a model, which makes a precedence test a measurement with a confidence interval rather than a proof. Five runs distinguish "always" from "usually" and do not distinguish "usually" from "95% of the time"; getting that second distinction takes something closer to fifty runs, which for a five-subject test is maybe twenty minutes of wall-clock and a few dollars of tokens. Whether that is worth it depends on whether the instruction is a preference or something you are about to describe to an auditor.

The result is also specific to the host version, the model, and the size of the assembled text. A team that measured "project usually beats user" against a 2,800-token assembly can see it shift after adding a large architecture section to the root file, because the relative position and dilution of both instructions changed. Re-running a five-subject test after a host upgrade takes ten minutes and is the cheapest configuration test anyone on the team will ever write.

Knowing that the project layer usually wins on test naming still does not tell you where an instruction stops applying. The commit prefix from the user layer travelled into a repository that had its own opinion, and the same mechanism operates inside a single repository: text loaded for a session does not know which directory it was written for. The next chapter follows a convention out of the package it was scoped to and into three that never asked for it.