Apply rate is not a failure rate

The slide has two bars on it. Across 200 tasks, unified diff applied cleanly 91% of the time and whole-file rewrite applied cleanly 99.5% of the time, and the recommendation under the chart is to move the agent to whole-file rewrite. The measurement is real, the runs happened, and the arithmetic is right. The conclusion does not follow, because the metric on the y-axis cannot see the failure the rewrite format is worst at.

Here is one of the 199 rewrites that counted as a success. The task was to add a retry to SettlementClient.post. The model returned a complete, syntactically valid, 812-line file, the harness wrote it, and the applier had nothing to do because there is nothing to apply. The diff against the original contains the intended six-line change, and also this:

-    def _migrate_legacy_ref(self, row):
-        """Pre-2021 rows store the ref in the memo field."""
-        if row.external_ref:
-            return row.external_ref
-        parsed = LEGACY_MEMO.match(row.memo or "")
-        return parsed.group(1) if parsed else None
-
-import hashlib
 import json
+import hashlib
 import logging
-    timeout = settings.SETTLEMENT_TIMEOUT_S
+    timeout = 30

A 43-line method the model did not consider important is gone, the import block has been reordered to the model's preferred sort, and a line that a previous edit in the same session had changed from 30 to a settings lookup has been reverted to its old value. That file applied at 100% success by the metric on the slide. It is also three defects, one of which will surface as a production incident when a 2019 settlement row arrives and external_ref is null.

Apply rate measures whether the edit format's mechanics worked. Reliability is about whether the repository ended up in the state the task intended, and the gap between those two is precisely the space where whole-file rewrite does its worst work.

The categories worth counting#

A failure rate needs a taxonomy before it needs a number — a taxonomy here being nothing more than an agreed list of the distinct ways an attempt can go wrong, each defined precisely enough that two people classifying the same run agree — and the taxonomy should have a row for every way an attempt can end badly, along with what detects it and what it costs when it escapes.

Apply rate covers rows two and three, which are the two cheapest rows in the table. Row one is invisible to it because a truncated rewrite never reaches an applier at all, so under most harnesses it is either not counted or counted as something else entirely. Rows four and five are invisible to it by construction, since both describe attempts that applied perfectly. Row six is mostly a property of the model rather than the format, and it is worth carrying in the taxonomy anyway so that it does not get silently attributed to whichever format happened to be under test.

Detecting row five is the part that takes work, and it is the part that makes the exercise worth doing. It requires each fixture — one frozen starting state of a repository plus one task to perform against it, kept in version control so that every format under test begins from identical bytes — to carry an intended change set, meaning the ground-truth diff, or at minimum the set of symbols the task is allowed to touch, and a comparison that flags anything outside it:

def unintended_changes(before: str, after: str, allowed: set[str]) -> list[str]:
    """Symbols added, removed or modified outside the allowed set."""
    b, a = symbols(before), symbols(after)
    touched = {name for name in b | a if b.get(name) != a.get(name)}
    return sorted(touched - allowed)

Run over the settlement example with allowed = {"SettlementClient.post"}, this returns ["_migrate_legacy_ref", "<module imports>", "SettlementClient._configure"], which is the defect the apply rate scored as a success. It is perhaps eighty lines of tree-sitter work per language, and without it the whole comparison is blind in the direction that decides the answer.

The denominator, and why the obvious one is wrong#

The natural denominator is the number of model outputs that reached the applier, because that is the population the applier logs. It is wrong in two directions at once. It excludes the truncated rewrites, which never got that far, so the format most prone to truncation is rewarded by having its worst failures removed from its own denominator. And it counts retries as independent trials, so a format that fails twice and succeeds on the third attempt contributes three data points and a 67% success rate to a metric that is supposed to be about outcomes.

Count tasks. One fixture, one budget of attempts, one verdict: the task ends resolved with no unintended change, or it does not. That denominator answers the question a format decision actually turns on, which is what fraction of the work the agent can finish correctly, and it makes retries visible as cost rather than as extra chances. Keep the per-attempt rates alongside it, because they are what you multiply by the cost of an attempt to get the money and latency figures, and a format with a 10% per-attempt failure rate and a thirteen-second attempt is a different proposition from one with a 3% rate and a four-minute attempt.

The record for a single attempt has four fields that matter, and none of them is "did it apply":

{
  "fixture": "billing-py/add-currency-guard",
  "format": "search_replace",
  "attempt": 1,
  "valid": true,
  "unintended_changes": [],
  "output_tokens": 812,
  "latency_ms": 14210,
  "failure_category": null
}

Validity and unintended change decide correctness; tokens and latency decide whether the format is affordable at the volume you run it. Latency deserves its own column rather than being inferred from token counts, because output is generated serially and a rewrite of a 3,000-line file spends minutes per attempt where a search-and-replace block spends seconds. A format that is two points more reliable and twenty times slower is not obviously the better choice, and a comparison that records only correctness cannot even pose the question.

Why last month of production traffic will not answer this#

The objection to building fixtures is always the same and always reasonable: there is a month of production runs sitting in the database, it cost nothing, and a fresh fixture set is a week of somebody's time. The month of data is confounded in exactly the direction that decides the outcome, confounded meaning that something other than the variable under test differs systematically between the groups being compared, so the difference you measure cannot be attributed to the thing you changed.

The tasks that reached the agent were chosen — by a router, by a heuristic, by engineers learning what it is good at — because they suited the format the agent already runs. If the agent emits unified diffs, small localised changes get sent to it and whole-module refactors do not, so the sample is enriched with the tasks that format handles and depleted of the ones it fails. Measuring a candidate format against that sample measures the fit between a format and the tasks its predecessor already handled well.

There are three further contaminants in the same data. The model version changed at least once during the month, and edit formats interact strongly with model capability. The prompt changed, because somebody was tuning it. And the retry logic means the log contains correlated attempts on the same task, so the effective sample size is well below the row count.

A fixture set fixes all four by construction. A hundred and twenty tasks across four repositories, each with a ground-truth change set and a test command, with file sizes deliberately spanning the output cap — some at 200 lines where a rewrite is comfortable, some at 3,000 where it is impossible — and each format run over identical prompts against identical starting trees. Everything varies except the thing under test.

Where the fixtures come from

The fixture set should be built from history rather than invented, because invented tasks drift toward the tidy cases. Take the last two hundred merged pull requests that touched the services the agent will work on, keep the ones whose diff is a self-contained change with a test that exercises it, and revert each one to produce a starting tree plus a ground-truth change set plus a command that passes only when the change is present. That gives tasks with the file sizes, naming conventions, indentation habits and code duplication the agent will actually meet.

Then stratify deliberately, because a uniform sample of real changes will under-represent the cases that separate the formats. Roughly a third of the fixtures should sit in files comfortably inside the output cap, a third near it, and a third well past it; roughly a quarter should be changes touching more than half of a file. Without the large-file stratum the truncation category never fires and rewrites look excellent; without the large-change stratum, search-and-replace never has to produce thirty anchors and looks better than it is.

Sample size deserves one moment of arithmetic. At 120 fixtures, a measured rate of 10% carries a 95% interval of roughly 5% to 17% — the range that would contain the true rate in 95 out of 100 repeats of the same measurement — which is wide enough that two formats differing by three points are not distinguishable. That is fine for the decisions this measurement supports, which are of the form "rewrite loses 14% of its attempts to truncation and search-and-replace loses none" — a gap that no plausible sample of this size would invert. It is not fine for tuning a threshold on a two-point difference, and a team that finds itself doing that should either grow the fixture set or accept that the two options are equivalent.

The week is defensible on arithmetic rather than on principle. The number decides which format runs on every edit the agent makes for the next year; at 400 edits a month that is roughly 5,000 edits the decision applies to. A one-point difference in the silent-defect rate is 50 defects that apply cleanly and reach review looking correct, and one of those reaching production costs more than the week on its own. The free data is not cheaper than the fixtures, because it cannot answer the question at any price. Buying a confounded number is worse than buying nothing, since the team will act on it.

What the fixtures say when you run them#

Over 120 fixtures, three formats, identical prompts, an 8k output cap, and up to three attempts per task:

The apply-rate column that the original slide reported is the "failed to apply" row, where rewrite still wins decisively at 0.5% against 9.1%. It wins that row and loses the comparison, because 14% of its attempts never produced a file and 11% of the ones that did carried a change nobody asked for. Reporting a single number from this table means choosing which failures to count, and the honest version of the slide has all seven rows on it.

None of this picks a format on its own. The 88% figure was produced under one cap, one model, one distribution of file sizes and one repository mix, and every one of those is a variable that moves the answer. The next chapter takes the numbers as inputs and works through the decision itself, starting with a legacy Java service whose files run past 2,000 lines and whose test suite is unusually good.