An agent that needs heavy review has not saved anybody time

The harness change added a generated summary to every agent patch: a short description of the intent, a list of files touched with one line each, and a note on which tests were run. Median time from patch opened to first approval fell from 22 minutes to 17 over the following month, and the programme's monthly update reported a 23% reduction in review cost. In the same month, two of the four reviewers who handle most agent patches said in retrospectives that the work had got heavier, and one of them had started batching agent patches to Friday afternoons because they had stopped fitting between other things.

Both observations are accurate. Time-to-first-approval is measuring a real interval that really did shrink, and it is measuring it the way a latency metric that stops at first byte measures a response: the number is genuine and the thing the user experiences continues after the number stops. Everything expensive about reviewing an agent patch happens after somebody first clicks approve.

The intervals nobody is counting#

Four of the transitions in that diagram consume reviewer time and none of them appears in a time-to-first-approval median. Revision rounds are the largest by volume: an agent patch on this programme averaged 1.9 review rounds before merge, and the second and third rounds are not cheaper than the first, because the reviewer has to reload context they had already dropped. Reviewer-authored rework is the most invisible: when the fix is small and the round trip through the agent takes twenty minutes, reviewers push a commit to the branch themselves, and 31% of merged agent patches had at least one reviewer commit on them. Abandoned patches carry review time with no merged change to show for it, and 18% of agent patches were closed without merging after somebody had already read them. Escaped defects arrive weeks later as incidents and get attributed to the code rather than to the review that passed it.

The generated summary plausibly improved all of the first-approval numbers and made at least one of the others worse. A confident, well-organised description in a familiar house style is exactly the thing that shortens the interval between opening a patch and approving it, because it lowers the effort of forming a judgement without changing how correct the patch is. Over the same month, the fraction of agent-authored merged changes that were reverted or hot-fixed within 14 days went from 3.1% to 5.8%. That combination — the tracked metric improving while an untracked balancing metric degrades — is what makes time-to-first-approval a bad primary metric rather than merely an incomplete one. A balancing metric is one you commit to watching alongside the headline figure precisely because the headline figure can be improved at its expense, and for anything measuring review speed the balancing metric is the rate at which defects reach production.

There is a further reason first approval is a poor stopping point for agent patches specifically. A human author is present after their patch is approved; they answer the follow-up question, they notice that the migration needs a backfill, and they carry the change through to production. An agent patch's author is not available in that sense, so the reviewer inherits the authorship role at approval, and everything an author would normally do afterwards lands on them. Measuring up to first approval therefore excludes not just some of the reviewer's work but the specific portion of it that the agent created by not being a person.

The rest of the gaming surface follows from the same logic. Splitting one change into four small patches improves the median while multiplying the fixed cost of reviewing anything. Routing patches to whoever approves fastest improves it. Having the agent avoid the parts of the codebase where review is slow improves it, and that is the payments ledger, which is where the value would have been.

A metric whose denominator is the unit of value#

The quantity worth tracking is reviewer-minutes spent per merged agent change, with everything in the numerator and merged changes in the denominator.

def review_load(period) -> dict:
    merged = [p for p in period.agent_patches if p.merged]
    abandoned = [p for p in period.agent_patches if p.closed_unmerged]

    minutes = 0.0
    for p in period.agent_patches:            # merged AND abandoned
        minutes += sum(r.active_minutes for r in p.review_sessions)
        minutes += sum(c.authoring_minutes for c in p.reviewer_commits)
        minutes += sum(t.minutes for t in p.linked_followup_fixes)

    return {
        "minutes_per_merged_change": minutes / max(len(merged), 1),
        "rounds_per_merged_change": sum(len(p.review_rounds) for p in merged) / max(len(merged), 1),
        "abandoned_share": len(abandoned) / max(len(period.agent_patches), 1),
        "reviewer_commit_share": sum(bool(p.reviewer_commits) for p in merged) / max(len(merged), 1),
        "escape_rate_14d": len([p for p in merged if p.reverted_or_hotfixed_14d]) / max(len(merged), 1),
    }

Choosing merged changes as the denominator is the decision that gives the metric its meaning, and the alternatives are worth seeing side by side:

Merged changes is the unit the business gets, so putting abandoned patches in the numerator and merged changes in the denominator makes abandonment expensive in the metric exactly as it is expensive in reality. On the programme above, the full calculation for the month after the summary change came to 61 reviewer-minutes per merged agent change, against 34 for human-authored changes of comparable size in the same period. The tracked 17-minute median was measuring roughly a quarter of the cost.

Recomputing the previous month the same way is what turns the reviewers' complaint into evidence:

Every row except the first got worse, and the first is the one that was on the dashboard. The mechanism is visible in the middle three rows: the summary made patches easy to approve on a first pass, so more of them got through the first pass, and the work moved downstream into revisions and reviewer commits. The programme was reporting a 23% cost reduction during a month when the cost per unit of shipped work rose by 27%.

Having a human baseline in the same units is what turns the number into a decision. Without it, 61 minutes is a number that can only be compared with itself, and the question the programme actually has to answer is whether routing a ticket to the agent costs less total engineering time than assigning it to a person. The sharpest published result on that question comes from METR, an independent research group that runs controlled measurements of what AI systems do to real work: in its study of experienced open-source developers, the developers using AI assistance on their own repositories were about 19% slower on the tasks measured, while the same developers estimated afterwards that they had been roughly 20% faster. The size of the gap between perception and measurement is the argument for instrumenting this at all, and the direction of the gap is the argument for not letting the programme grade itself on retrospectives.

Active-minutes measurement needs a definition that survives contact with how people work. The version used here counts a review session as the span between opening the diff and the last interaction with it, with gaps over five minutes excluded, and it undercounts thinking that happens away from the screen and overcounts tabs left open. It is a proxy, and it should be validated once against a small diary study rather than trusted indefinitely.

Reviewers can read the dashboard too#

Any metric computed from reviewer behaviour changes reviewer behaviour, and pretending otherwise produces a corrupted number quickly. The design choices that keep it usable are mostly about what the metric is allowed to be attached to.

Report it at programme level only, never per reviewer, and make that a stated policy rather than a current convention. The moment minutes-per-change appears next to a name, the cheapest way to improve it is to approve faster, which is precisely the behaviour that made the original metric useless. This also means the metric cannot be used to compare teams, because teams will start comparing themselves.

Set no target on it. A target invites optimisation, and this metric's job is to be an observation that informs decisions about the agent — whether to expand it to another service, whether a harness change was worth it, whether to keep paying for a vendor. Pair it permanently with the 14-day escape rate and the abandonment share, published in the same table every month, so that an improvement in minutes accompanied by a rise in escapes is visible in one glance instead of requiring somebody to go looking.

Collect reviewer experience separately and by sampling, because the thing reviewers noticed in the retrospective was true and arrived before the numbers did. A two-question prompt on one in five reviews — how much of this did you have to verify yourself, and would you rather have written it — costs a few seconds and produces a signal that leads the quantitative metrics by weeks. Treat that as an instrument in its own right rather than as colour commentary.

The honest limits are worth stating with the metric. Reviewer-minutes does not capture the cost of context switching into an unfamiliar area of the codebase, does not capture the slow erosion of trust that makes people read agent patches more suspiciously than they need to, and does not capture the cases where reviewing a bad patch taught somebody something. It also cannot distinguish a patch that took 40 minutes because it was subtle and good from one that took 40 minutes because it was subtle and wrong; only the escape rate can, and it arrives late.

With review load instrumented alongside resolution, the programme has two numbers that move for different reasons, which makes a specific and awkward pattern legible: a suite score that climbs steadily while review load stays flat and reviewers keep saying nothing has changed.