Check what was loaded before you rewrite the rule

The pull request added proration to the subscription upgrade path, and the diff contained this:

def prorate(plan_price: float, days_remaining: int, days_in_period: int) -> float:
    """Amount owed for the remainder of the current billing period."""
    return round(plan_price * (days_remaining / days_in_period), 2)

The repository's root AGENTS.md has said the same thing for fourteen months, in bold, in the second paragraph: all monetary values use money.Money; never a float or a bare Decimal. The reviewer pasted the rule into the agent's channel. Someone proposed the fix that gets proposed every time, which is to put the rule in capitals, move it to the very top of the file, and append it to the task prompt as a reminder. Someone else suggested dropping temperature to 0.

Both suggestions describe the same underlying belief, which is that the instruction reached the model intact and the model chose to disregard it, so the remedy is emphasis. That belief may turn out to be true. It is the last item on a list of six, and every item above it is cheaper to check and more likely to be the actual cause.

The order to check things in#

Formatting changes have a specific defect as engineering: nobody can tell whether they worked. If the agent violated the rule in six of ten runs and after the rewrite violates it in four of ten, that is entirely consistent with pure noise — the 95% confidence interval on six-of-ten runs, meaning the range of true violation rates that could plausibly have produced six failures out of ten, stretches from roughly 26% to 88%, so a two-run improvement is indistinguishable from nothing. Ten runs cannot resolve the effect size a formatting change plausibly has, and nobody is going to fund a hundred. So the change ships, the number moves by an unmeasurable amount, and the team learns that instruction plumbing is a matter of tone. Meanwhile, if the rule was never in scope for services/billing in the first place, capitalisation reaches nothing at all.

Run the checks in this order, because each one is decisive when it fires and the early ones cost minutes.

Checks one through three are mechanical and take about ten minutes with the session logs. Most harnesses write one JSONL file per session — JSON Lines, a plain text file carrying one JSON object per line, one line per turn — so the assembled request is recoverable after the fact:

# Which config files did this session actually load, and in what order?
jq -r 'select(.type=="user") | .message.content
       | if type=="array" then (.[]?|.text?) else . end
       | select(. != null)
       | capture("(?<p>[A-Za-z0-9_/.-]*AGENTS\\.md)").p' session.jsonl \
  | cat -n

On the failing sessions that produced prorate, the output was two lines:

     1  AGENTS.md
     2  services/billing/AGENTS.md

The root file was loaded, so check one clears. Check two is where it stopped being a wording problem. services/billing/AGENTS.md had been written eight months earlier by the team that owns subscriptions, and its fourth bullet reads: Prices coming from the plan catalogue are plain floats for historical reasons; convert at the boundary. Two configs, both in scope, both loaded, saying incompatible things about the same values, and the nearer one is the one that talks about the exact objects the task is about. The agent did not ignore an instruction; it picked the more specific of two, which is what a competent engineer would also do.

Check three, had it been needed, is the one to hold loosely. The Money rule sat at roughly token 2,100 of a request that had reached 58,000 tokens by the time the agent wrote prorate, which sounds damning and is hard to convert into a fix. Position effects in long contexts are real, modest, and extremely difficult to measure on ten runs; a team that re-orders its config file and re-runs is measuring noise. The useful output of check three is not "move it up" but "this rule is competing against fifty-six thousand tokens of evidence, so consider a mechanism that arrives closer to the work" — a hook, a skill body, or a failing check, all of which land mid-transcript.

If the harness does not persist transcripts, or persists them in a form you cannot parse, checks one and two are still available through the canary approach from the previous chapter. Add a distinctive token to each config file, run the failing task five times, and ask the agent at the end of the run to list every instruction file it was given verbatim. The self-report is not authoritative, because a model will happily produce a plausible list, but a canary appearing in that list is strong evidence and a canary absent from every one of five runs is strong evidence the other way.

The counter-evidence nobody counts#

Suppose check two had cleared as well, which it does in about half the cases worth investigating. Check four is the one that then explains most of the remainder, and it is the one teams skip because it does not feel like a configuration question.

Before writing prorate, the agent read seven files: the subscription model, the plan catalogue, two service modules, the existing proration test, and two files it pulled from a traceback. Here is what those files contain, counted across the package:

$ rg -t py --stats 'float|\* *1\.|/ *100\b' services/billing | tail -4
41 files contained matches
213 matched lines
0.42s

Forty-one files of float arithmetic in the package the agent was asked to change, of which it read seven, is a demonstration with forty-one witnesses against a sentence with one. Retrieved code is not neutral background; it is the largest instruction source in the run, as the previous chapter's inventory made concrete, and it teaches by example whether anyone intended it to. The model is doing pattern completion over a codebase where the pattern is floats. Any account of the failure that does not mention those 213 lines is an account of a different failure.

This also predicts something useful, which is that the violation rate should track the retrieval. Sessions where the agent happened to open services/billing/money_adapter.py — the one module that does use Money consistently — produced compliant patches noticeably more often. That is a testable prediction rather than a story, and it points at a fix that has nothing to do with the config file: convert the forty-one files, or add a check that makes the violation fail, both of which change what the agent sees rather than what it is told.

Why lowering the temperature makes this worse#

Temperature comes up because the behaviour is inconsistent, and inconsistency reads as randomness. Temperature controls how much the sampler explores away from the model's preferred continuation, so lowering it makes the output more consistently whatever the model already prefers. If the model's modal behaviour on this task is float arithmetic — which is exactly what 213 lines of local precedent would produce — then temperature 0 does not move the violation rate from 6/10 towards 0/10. It moves it towards 10/10, because the four compliant runs were the exploration.

Teams do sometimes see the rate improve after a temperature drop, and the reason is worth naming: at temperature 0 the runs stop being independent samples, so ten runs of the same task give you one observation repeated ten times. Whatever that single observation is gets reported as a rate with a spurious denominator. Turning down the randomness is a legitimate move when you want reproducibility for debugging, and it is not a compliance intervention.

When the rule cannot be followed#

Check five looks like a formality and catches a real class of failure. On this repository, money.Money was implemented with __add__, __sub__ and __mul__ against int, and nothing else:

class Money:
    def __mul__(self, other: int) -> "Money": ...
    # no __truediv__, no __mul__(Fraction), no ratio helper

Proration needs days_remaining / days_in_period, which is a ratio, and there was no supported way to multiply a Money by one. An agent that tried to comply would have written code that fails at runtime with a TypeError, seen the failure in its own test output, and backed off to floats — which is precisely what the transcripts of two of the ten runs show it doing, complete with a comment reading # Money does not support ratio multiplication. The instruction was received, attempted, and defeated by the library. No amount of prompt engineering fixes a missing method.

This is the general shape of the check: write the compliant patch yourself, by hand, before you accuse the model of not writing it. If you cannot, the rule is aspirational and belongs on the backlog rather than in the config.

Check six then gets its turn, and on this rule it does have something to say, though not what the capitalisation proposal assumed. All monetary values use money.Money is unambiguous about what to use and silent about the two things an engineer writing proration actually needs to know: whether a tax rate or a proration ratio counts as a monetary value, and what to do at the boundary where the plan catalogue hands over a float. A rule that answers those is longer and more useful:

Monetary amounts use money.Money. Rates and ratios are Fraction, never float. Convert at the edge with Money.from_minor_units(...); catalogue prices arrive as floats and must be converted in the adapter, not carried inward.

That version is not louder. It removes the two decisions the agent was left to make on its own, and it names the function to call, which matters more than any amount of emphasis because it tells the model what to write rather than what to avoid.

What this is not#

Tracing is not a claim that wording never matters. Check six exists on the list and does fire, most often on rules phrased as values rather than as behaviours — be careful with money, prefer explicitness — where there is no specific action the agent could take differently. The point of the ordering is that wording is the hypothesis with the highest cost to test and the lowest prior, so it goes last.

The fix that shipped for prorate was three things, none of them a rewording: Money.__mul__ gained a Fraction overload so the compliant patch became writable, the contradictory bullet in services/billing/AGENTS.md was replaced by the boundary-conversion sentence above, and a check was added that fails on float-typed parameters named like money in services/billing. The fix that was refused was the original proposal — capitals, reordering, and a reminder appended to every task prompt — on the grounds that it is unmeasurable at any sample size the team would actually run, it would have been credited with whatever the other three changes achieved, and it leaves the contradicting package rule in place to defeat the next person.

It is also not a debugging technique that scales to every observed oddity. The trace answers which instructions were in force and what competed with them, and it answers that well. It does not tell you why a model produced a particular reasoning step, and attempting to read intent out of a transcript is how people end up with elaborate theories about attention that they cannot test. Keep the questions to ones the logs can settle.

Finally, the trace is diagnosis rather than architecture. The prorate investigation ended with a specific, honest finding: two loaded configs disagreed, and the nearer one won by an accident of how the harness happens to concatenate files. That accident is doing the work of a policy decision. Nobody wrote down that the package file beats the root file, or that the root file's Money rule is one nobody may override, and until somebody does, the next conflict resolves by whatever the assembler does next month. The following chapter makes that resolution an explicit, written rule with a log behind it.