Thirty-one points of compliance still costs tokens on every request

The result slide had four lines on it:

convention compliance      46% -> 77%   (+31 points, McNemar p=0.0009)
first-attempt CI pass      61% -> 61%   (no change, n=200/arm)
tokens per task            62.4k -> 73.6k input, 9.0k -> 9.4k output  (+18%)
wall-clock per task        4.2 min -> 4.6 min  (+9%)

The author read the first line and proposed shipping. A director read the third and fourth and asked what the first line was for. Both readings are incomplete in the same way: compliance is a proxy for something nobody in the room had measured, and the token and latency figures are a real cost that will be paid on every request from now until someone deletes the file.

Price the recurring side first, because it is the easy half#

The cost is arithmetic, and doing it takes two minutes, which is why there is no excuse for a proposal that omits it. At this team's contracted rate of $3 per million input tokens and $15 per million output tokens:

An extra $160 a month, or about $1,900 a year, against two weeks of an engineer's time already spent. On that basis alone the decision is easy and slightly boring, which is why the token figure should never be the whole cost argument.

Two details underneath it are more interesting than the total. The conventions file is roughly 3,000 tokens and the measured increase is 11,200 input tokens per task, so the file itself accounts for about a quarter of the growth. The rest comes from behaviour: skills the model invoked, whose bodies loaded mid-run, and the files those skills pointed it at. A conventions change that tells the agent to consult the repository layer causes it to go read the repository layer, and that is the intended effect with a bill attached.

The other detail is caching. The always-loaded prefix — system prompt, root config, package configs, skill descriptions — is byte-identical across sessions for hours at a time, which makes it exactly the shape that prompt caching is built for; cached reads typically bill at a fraction of fresh input. The retrieved repository code, which is the bulk of the 62,400, varies per task and mostly will not cache. So the marginal cost of the conventions file specifically is closer to a tenth of its face value on a harness that caches its prefix, and closer to full price on one that rebuilds the prompt every turn. Which of those you are on is worth knowing before arguing about the file's length.

Latency is 24 seconds per task. For a nightly batch that is nothing. For an engineer sitting in front of the agent forty times a day it is sixteen minutes, which is a real number that competes directly with the review time the conventions are supposed to save.

There is a third recurring cost that does not appear on the slide because no meter records it. The conventions occupy 11,200 tokens of a context window that also has to hold the retrieved code, the accumulated tool output and the patch being written, which is the arithmetic topic 13.3 worked through. On a task that needs 90,000 tokens of repository context to be done well, spending 11,200 on instructions is a choice about what gets left out, and the thing left out is usually the least obviously relevant file, which is disproportionately often the one that would have shown the agent the right pattern. That trade is invisible in the token count and shows up, if it shows up anywhere, in the failure categories below.

Compliance is a proxy; name the thing it stands for#

Nobody wants convention compliance. What the team wants is agent patches that a human can merge without rewriting them, and code that is still coherent in a year. Compliance is a cheap, fast, automatable stand-in for both, and its value depends entirely on the correlation holding, which was never checked.

Three measurements make the downstream claim, and all three come from data the team already has.

Acceptance rate is the share of agent-authored pull requests merged with no review comment that required a substantive change. Define "substantive" mechanically — a comment that resulted in a code change before merge — so the number does not drift with reviewer mood. Rework is the volume of human editing on top of the agent's patch before it merges, measured as lines changed in commits authored by a human on the agent's branch, plus the revert rate within fourteen days. Failure categories are a taxonomy applied to every run that did not produce a mergeable patch, which turns "it failed" into something you can act on.

-- Six weeks of agent PRs, split by whether the run's instruction-set hash
-- was the pre- or post-conventions one (see chapter 3's run record).
select
  case when r.instruction_set_hash = :old_hash then 'before' else 'after' end as arm,
  count(*)                                                       as prs,
  avg(case when p.change_requesting_comments = 0 then 1.0 else 0 end) as accept_rate,
  percentile_cont(0.5) within group (order by p.human_lines_added) as median_rework,
  avg(case when p.reverted_within_14d then 1.0 else 0 end)       as revert_rate
from agent_runs r
join pull_requests p on p.run_id = r.run_id
where p.created_at >= now() - interval '6 weeks'
group by 1;

On this repository the query returned 180 pull requests, 94 before and 86 after:

That is the evidence the compliance number was standing in for, and it moved. The revert rate did not move meaningfully, which is consistent with the CI pass rate being flat: conventions govern the shape of correct code and have little to do with whether the logic is right. Six weeks of pull requests is observational rather than controlled — the mix of tasks changed, two engineers went on leave — so it is corroboration for the experiment rather than a substitute for it.

Two confounds in that table are worth naming rather than glossing. Reviewers comment less on agent pull requests as they get used to them, so an acceptance rate measured over six weeks partly measures reviewer fatigue; and the task mix shifted towards new endpoints, which are more convention-sensitive than bug fixes and therefore flatter the after arm. Both are partly correctable by measuring human-authored pull requests over the same window as a control. On this repository the human acceptance rate moved from 0.63 to 0.66 across the two periods, which suggests a small general drift and leaves most of the 0.41-to-0.58 movement attributable to the change. That difference-of-differences — the 17-point movement in the agent arm minus the 3-point movement in the human arm over the same weeks, on the assumption that whatever drifted affected both — is a weaker instrument than the controlled experiment and it is the strongest one available for a metric that only exists downstream of a human.

The failure taxonomy is the part teams skip and the part that tells you where to spend the next two weeks. Categorised across the 200 runs per arm:

The second row is a gain nobody predicted and is worth more than the compliance headline: naming the repository layer and the Problem envelope in the conventions gave the agent the correct symbols, so it invented fewer. The fourth row is a loss caused directly by the intervention — 18% more tokens per task pushes some runs into the attempt budget, and six extra runs died there. That row is what "check whether anything unrelated got worse" produces when it is done properly, and it argues for raising the attempt budget or trimming the file rather than for abandoning either.

The call#

Ship it, and move the load-bearing rules out of the file. The compliance gain is concentrated in rules that have or could have a machine check, which the previous chapter established, and a check costs nothing per request and reports when it slips. Converting money-001 and db-002 into lint rules should hold most of the 31 points while giving back a share of the 11,200 tokens, and that is a measurable prediction rather than a hope — re-run the same 40 tasks after the conversion and see.

Keep the prose that carries reasons the agent can act on, drop the rules with no check and no measurable effect, raise the attempt budget by one to recover the six lost runs, and put the acceptance-rate query on a dashboard so the claim stays live rather than becoming a slide from March.

The flat CI pass rate is not a mark against the work, and reporting it as one is a mistake in the other direction. Convention compliance and first-attempt correctness are close to independent; a patch that uses floats for money and a patch that uses Money both pass a test suite that does not assert on the type. Expecting the conventions file to move correctness is expecting the wrong instrument to move.

What would mean the two weeks were wasted#

The result that condemns the work is compliance up and nothing downstream moving. If acceptance rate, change-requesting comments and rework had all come back flat within noise across a hundred-odd pull requests, then the proxy is not tracking the thing it stands for on this repository, and the team has bought a 31-point improvement in a metric it invented, at 18% more tokens and 9% more latency, forever. Ship nothing and find out which conventions reviewers actually complain about.

Three narrower results would say the same thing in different words. If the entire compliance gain came from rules that already had lint rules before the two weeks started, the file added nothing and CI was doing the work. If the incomplete-run count had gone from 8 to 40 rather than to 14, the token overhead is crowding out retrieval badly enough to break tasks, and the file is a net loss regardless of compliance. And if the revert rate had risen — patches that look conventional, pass review more easily, and fail in production — that is the worst outcome available here, because the intervention would be making bad patches more persuasive.

Write those thresholds down before the numbers arrive. A falsification criterion decided after seeing the result is not a criterion, because there is always a reading of a flat downstream metric that preserves the conclusion — the window was too short, the reviewers were distracted, the task mix was wrong. Committing in advance to acceptance rate must move by at least 5 points across at least 80 pull requests costs nothing when the result is good and is the only thing that makes the exercise capable of saying no.

None of those happened, which is why the call is to ship. Being able to say what would have changed the call is the part that makes it a decision rather than a preference, and it is the part a director is actually asking for when they push back on a slide with one number on it.

What none of these measurements capture is time. Every figure here describes a repository at one commit with a file written last month, and both of those move. A rule that measured well in March describes code that may not exist in September, and a rule that describes code that no longer exists does more damage than one that was never written.