Compliance is an impression until identical tasks run both ways

The pull request was 640 lines: a rewritten root AGENTS.md, four package configs with scoped and flagged rules, and three skills. Its description read, in full, Agent now follows our conventions much more reliably. The author had spent two weeks on it and had watched the agent's output improve over those two weeks, which is a real observation about a period during which the model version changed once, the repository gained a hundred commits, and the tasks he happened to try shifted from bug fixes to new endpoints. Nothing in the PR distinguishes the effect of the file from the effect of everything else that moved.

Turning that into evidence takes three decisions, and they are the same three decisions for any comparison: what is held constant, what varies, and what is scored. Each of them has a wrong answer that is easy to reach by accident.

The comparison runs the same tasks twice, in two arms, which is the word for the groups an experiment contrasts. The control arm runs against a checkout with the conventions removed, the treatment arm runs against one with them in place, and everything else about the two is meant to be identical.

Hold the repository still#

Everything that could plausibly change the output has to be pinned, and the list is longer than it first appears because a coding agent's behaviour depends on the state of the machine as much as on the model.

The seed row is where most in-house comparisons fail. A seed is the number that fixes the sampler's random draws, so two runs given the same seed and the same inputs are meant to make the same choices; running five seeds per task and reusing those same five in both arms means the two arms sample the model's behaviour at the same five points rather than at ten unrelated ones. Agent runs on the same task are not deterministic even at temperature 0, because tool results come back in varying order and timing, so a single run per arm gives a per-task result that is a coin flip dressed as a measurement. Five seeds per task is the smallest number that lets you talk about a per-task rate rather than a per-task outcome, and it multiplies the bill by five, which is the honest cost of the claim.

The one thing that varies is the presence of the conventions. That sounds trivial and has a subtlety: removing the file from the loader is not the same as removing it from the repository. An agent with a search tool will find AGENTS.md on disk, read it as an ordinary file, and comply with it, which produces a control arm that is quietly a treatment arm. Delete the files in the control checkout, and delete the skills directory, and confirm it by grepping the control transcripts for a distinctive phrase from the file. On the first serious run of this at one team, four of forty control tasks had the agent open AGENTS.md unprompted because a README linked to it.

The task set has to be able to fail#

A task set assembled from "recent tickets" will mostly contain work where the convention never comes up, and those tasks contribute nothing but cost and noise. Every task needs two properties: the change it requires must be one where the convention can be violated, and the violation must be decidable by a program.

The cheapest source of such tasks is the review history. Search the last six months of pull requests for review comments citing a convention — use the repository layer, this should return Problem, Money not float — and each hit is a task where a human already established that the convention was in reach and was missed. Forty tasks across six conventions is a workable size: enough that the aggregate has a usable interval, small enough that a person can read every failing diff, which you will need to do.

Here is one of them in full, because the level of specification matters more than the count.

id: T-017
base_commit: 8c41f0d
prompt: |
  Add POST /v2/refunds to services/billing. It takes an invoice id and an
  amount, creates a Refund row, and calls the payment provider's refund API.
  Return the created refund. Handle the case where the invoice does not exist.
conventions_in_reach: [err-001, money-001, db-002]
verifier: verifiers/t017.py
budget: {attempts: 4, context_tokens: 120000}

The budget field caps what a single run may spend. Four attempts means the agent gets four passes at making the repository's tests go green before the harness stops it and the run is recorded as incomplete, and 120,000 tokens is the ceiling on the context it may fill while doing so. Both numbers have to be the same in both arms, because an agent given a larger budget will find its way to a compliant patch more often for reasons that have nothing to do with the conventions.

Mining the tasks is mostly mechanical. Pull every review comment on the last six months of pull requests, filter to comments whose body matches a small vocabulary drawn from the conventions themselves — Money, repository layer, Problem, structured logger, import — and read the resulting few hundred comments to find the ones where a convention was genuinely missed rather than discussed. Each survivor gives you a base commit, a change description that can be rewritten as a task prompt without leaking the answer, and a known-possible violation. Rewriting the prompts is the part that needs care, because the original PR title often contains the convention's vocabulary and handing the agent the word Money in the task text is a hint the control arm should not get either.

conventions_in_reach is the field that keeps the task set honest. A task listed against money-001 has to involve an amount; a task listed against db-002 has to touch persistence. If a task's list is empty, it does not belong in this set, and if every task lists every convention, somebody has stopped thinking.

The verifier decides, not the reviewer#

Scoring by reading diffs works for the first ten and then degrades, because the person reading knows which arm they are looking at. A verifier per convention removes that, and it has a second benefit that matters more in the long run: a verifier is a check you can also run in CI, which is the subject of the next chapter.

import ast, pathlib, sys

FORBIDDEN_RETURN = "dict"

def handler_returns_problem(tree: ast.AST) -> list[str]:
    """Every function decorated with @router.* must return Problem or a model."""
    bad = []
    for node in ast.walk(tree):
        if not isinstance(node, ast.FunctionDef):
            continue
        if not any(isinstance(d, ast.Call) and "router" in ast.unparse(d.func)
                   for d in node.decorator_list):
            continue
        for ret in (n for n in ast.walk(node) if isinstance(n, ast.Return)):
            if isinstance(ret.value, ast.Dict):
                bad.append(f"{node.name}:{ret.lineno} returns a bare dict")
    return bad

if __name__ == "__main__":
    failures = []
    for path in pathlib.Path("services/billing/api").rglob("*.py"):
        failures += handler_returns_problem(ast.parse(path.read_text()))
    print("\n".join(failures))
    sys.exit(1 if failures else 0)

A verifier of thirty lines catches the thing a review comment would have caught, and it catches it identically in both arms. Write it against the convention rather than against the expected patch, because a verifier that pattern-matches one correct solution will fail valid alternatives and inflate the difference between arms in whichever direction the author expected.

Alongside the convention verifier, score three more things on every run, all of them cheap because the harness already emits them: whether the repository's own test suite passes on the patch, total input and output tokens, and wall-clock duration. Those are the columns that stop a compliance win from being reported in isolation, and the next-but-one chapter is entirely about what they turn out to say.

The shape of the result#

Four hundred runs — forty tasks, five seeds, two arms — at roughly 62,000 input and 9,000 output tokens per run comes to about 28 million tokens, which lands near $170 at current mid-tier coding-model prices and takes about three and a half hours with eight workers in parallel. That is the entire cost of replacing "much more reliably" with a number, and it is small enough that the objection to running it is never really the money.

Analyse it paired, task by task, rather than as two pooled percentages. Pooling throws away the fact that the same forty tasks appear in both arms, and the pooled interval is wider than it needs to be. Per task, compute the compliance rate across its five seeds in each arm, then count the tasks that moved:

tasks improved (control < treatment): 19
tasks unchanged:                      18
tasks regressed (control > treatment): 3
pooled compliance: control 0.46 (92/200), treatment 0.77 (154/200)
McNemar exact, discordant 19 up / 3 down: p = 0.0009

McNemar's test is the one that suits this shape of data. It ignores the tasks that behaved the same way in both arms and looks only at the discordant ones, asking how often chance alone would split them nineteen-to-three rather than something closer to eleven-to-eleven; p = 0.0009 says about once in a thousand repetitions of an experiment where the conventions did nothing. Nineteen up against three down is the result that carries the claim, and the three regressions are the most informative rows in the table. On this run they were all the same shape: tasks in services/reporting, where a newly added package rule about Result types pushed the agent away from a handler convention it had previously satisfied. That is the resolver from the previous chapter behaving exactly as specified and the specification being wrong for those files, and it would have been invisible in a pooled 46-to-77 headline.

The cheap version, and what it cannot tell you#

Four hundred runs is the comfortable size. Teams often start with what they can run over a lunch break, and it is worth being precise about what a smaller design buys, because the temptation is to run twenty tasks with one seed each and report the difference as though it were the same kind of number.

The right-hand column is the one that decides whether the exercise is worth doing at all. A conventions file that improves compliance by 10 points is plausibly worth shipping and is invisible to the first two designs, so a null result from forty runs means nothing and should not be reported as evidence against the file. The forty-task design also survives being sliced by convention, which the next chapter depends on: 200 runs per arm split six ways gives roughly 30 runs per rule per arm, which is thin but interpretable, while 40 runs split six ways gives nothing.

A worthwhile middle path is to run the cheap design first as a smoke test of the harness itself — checking that the control arm is genuinely clean, that verifiers do not crash, that tasks are solvable at all — and only then spend the $170. On the first attempt at this, three of the forty verifiers had bugs and one task turned out to be impossible at the pinned commit because the fixture it needed had been deleted. Finding that with 40 runs rather than 400 is worth the lunch break.

What this measures and what it does not#

The number is a compliance rate on a fixed task set at a fixed commit, and it generalises to the tasks in that set. It is not a claim about the model, which is why comparing it against a published benchmark is meaningless: the same conventions file on a different repository with different counter-evidence in the code will produce a different delta.

It is also not a quality measurement. Compliance says the patch followed the house rule; it says nothing about whether the patch was correct, whether a human accepted it, or whether the rule was worth following. The control arm's first-attempt test pass rate and the treatment arm's may be identical while compliance moves thirty points, and that combination is common enough that it is worth expecting.

Two smaller cautions. Do not let the person who wrote the conventions also write the task prompts, because prompts drift towards the vocabulary of the rules and the treatment arm gets a hint the control does not. And do not run the arms a week apart; run them interleaved, alternating arm per task, so that a provider-side model change during the window damages both arms equally rather than one.

The aggregate hides something specific, which the next chapter pulls apart. The 31-point improvement is not spread evenly across the six conventions; two of them account for nearly all of it, and the property those two share is not that they were better written.