Inner-loop value is defects found per second of iteration time

The trial ran for two weeks on a quarter of incoming tasks: same agent, same prompts, same tickets, with the full integration build added to every iteration alongside the checks already there. The results are not ambiguous. End-to-end defect detection — the fraction of seeded defects rejected somewhere before merge — went from 62% to 68%. Median task time went from 4.1 minutes to 21 minutes. Tasks that previously completed inside a coffee break started spanning meetings, the queue of pending tickets stopped draining, and two engineers went back to writing the small changes by hand because waiting was worse than typing.

Six points is a real improvement and nobody disputes the measurement. The argument in the room is that catching more defects is obviously worth more wall clock, which sounds like seriousness about correctness and is an accounting error. The integration build catches 61% of seeded defects standalone; running behind the three cheaper checks, the marginal column from the previous chapter says it catches 6 points nothing else got. Five times the task duration bought the 6, and the 61 was never on the table.

The unit that makes checks comparable#

A check's cost inside the loop is its duration multiplied by the number of attempts a task takes, because it runs on every attempt. Attempts per task is a number the harness already has; on this repository the median is 3.2 and the 90th percentile is 6. Its value is the marginal detection column, measured over one corpus in ladder order. Dividing gives points of detection per second of loop time, which is the unit that makes a two-second linter and an eleven-minute build comparable at all.

The ratio between the second row and the fourth is roughly 480 to 1. That is not a close call being decided by a debatable weighting; it is two things that do not belong in the same phase of the loop. The unit-test row is the interesting one, because it is where the argument gets genuine: 0.076 points per loop-second is twenty times worse than the type check and twenty-seven times better than the build, and the 22 marginal points it contributes are the only points in the table that come from executing the code. Chapter 2's whole second half — the off-by-one, the inverted condition, the wrong field — lives in those 22 points and in nothing else on this list. A ranking by rate alone would drop it; the reason it stays is that the class of defect it uniquely catches is the class an agent produces most.

The other reason to keep both latency columns rather than collapsing to a mean is that the p95 is what the loop's timeout has to accommodate, and the two columns diagnose different things. A check whose p95 is 40 seconds against a 6-second p50 can still sit in the inner loop, since it is fast on the overwhelming majority of attempts and the tail is a cache problem with known fixes. A check whose p50 is 90 seconds is slow every single time, and no cache work will change that, because the cost is the execution of the code rather than the analysis of it.

Whether a failure is something the agent can act on#

Speed is only half of why the integration build does not belong in the loop. The other half is what it says when it fails:

$ make integration
[compose] starting 9 services...
[compose] billing-api        exited with code 1 after 8.2s
[compose] worker             healthy
[compose] postgres           healthy
[smoke] GET /v1/invoices/inv_88123 -> 502 Bad Gateway (expected 200)
[smoke] 1 of 34 checks failed
make: *** [Makefile:88: integration] Error 1

There is no path in that output, no line, no column, and no code. It names a service that did not start, and a URL that returned the wrong status when one of the smoke checks — a small set of requests fired at the assembled system to see whether it is alive at all — went looking for it, which is a description of a symptom four abstraction layers away from the edit. Handed that, an agent will do one of three things: guess at a file and patch it speculatively, start reading container logs and burn its budget on investigation, or edit the smoke test. Every one of those is a worse use of an attempt than the six-second type check that would have named the line.

That property is what the actionability axis from chapter 4 measures, and it is close to binary in practice. A finding either arrives as a location the agent can open and edit, or it arrives as evidence that something is wrong somewhere. The second kind is genuinely valuable — it is exactly what you want gating a merge, and it is what catches the version skew and the renamed config key that nothing else sees — and it is nearly worthless as inner-loop feedback, because the loop's premise is that each attempt ends with a diagnostic the next attempt can act on directly.

The same axis explains why a check can be fast and still not belong. A two-second smoke test that reports "health endpoint returned 503" is affordable and unactionable; a forty-second type check that names three lines is expensive and actionable. Ranking by points per loop-second and then filtering by actionability, in that order, is what produces a defensible inner-loop set rather than a list of everything that finished quickly.

The purchase is earlier discovery, not detection#

There is a version of the correctness argument that survives the arithmetic above, and it is worth stating properly because it is the strongest form of the case for expensive checks in the loop. Catching a defect costs different amounts depending on where it is caught. In the loop, the cost is one more attempt: roughly 20 to 60 seconds of agent time and a few thousand tokens. In review, the cost is a human reading a wrong patch, writing a comment, and waiting for a round trip — call it 25 minutes of engineer attention and a day of latency. In production, it is an incident.

If those were the alternatives, paying eleven minutes to avoid a 25-minute review round trip would be arguable. They are not the alternatives, because the integration build runs before merge regardless of whether it also runs in the loop. Nothing the build would have caught in the loop escapes to production; it is caught twenty minutes later by the same command in CI. What the in-loop placement buys is the difference between the agent fixing it autonomously and a human seeing a red pipeline and sending it back, and that difference is worth something, weighed against a fivefold increase in the duration of every task including the ones that never had a defect in the first place.

The arithmetic of that trade is unfavourable in a specific way. The build's 6 marginal points mean it changes the outcome on about 1 task in 17. The cost is paid on all 17. At 3.2 attempts and 11 minutes, the loop spends roughly 10 hours of wall clock across those 17 tasks to save a handful of review round trips on one of them, and it does that while making the other 16 tasks slow enough that people stop routing work to the agent. The escape-cost argument is right that not all detections are worth the same; it does not rescue a check whose entire contribution is moving a catch from CI into the loop.

What this metric leaves out#

Points per loop-second is severity-blind, and severity is not uniformly distributed. A check that catches 2 marginal points consisting entirely of missing tenant filters in SQL is worth more than one catching 10 points of unused imports, because the escape cost of the first class is a cross-tenant data leak and the second is nothing. The repair is to weight the corpus rather than to abandon the metric: tag each seed with an escape-cost class when you build it, and read the marginal column weighted by that tag. On most repositories this reorders the middle of the table and leaves the ends where they were, because the expensive-defect classes tend to need execution, which strengthens the case for keeping the impacted unit tests and does nothing for the integration build.

The metric also ranks placement in the inner loop and says nothing about whether a check should exist. Every row in that table earns its place somewhere in the pipeline; the integration build is the last line of defence against a class of failure nothing else sees, and removing it from CI on the strength of 0.0028 points per loop-second would be a serious misreading. The question the number answers is narrow: given that the loop runs this on every attempt of every task, is the detection worth the iteration time.

Two further things the number does not capture. A check that fails spuriously costs an attempt and misleads, so its effective cost is higher than its duration — a suite with a 3% flake rate consumes about one extra attempt per ten tasks and occasionally sends the agent to patch code that was never broken, which is why 13.6 put quarantine in the harness rather than in the agent's judgement. And attempts per task is not independent of the checks: adding a check that fails often raises the attempt count, so a check's true cost is its duration times the attempt count it produces, which can only be measured by running the configuration rather than predicted from the table.

Ranking gives an order. Turning an order into a loop means deciding what runs on every attempt, what runs only when the cheap rungs are clean, and what never runs in the loop at all — and doing it in a way that does not drift from the pipeline that ultimately gates the merge, which is the last chapter.