Cheap checks go inside the loop, slow ones at its edge

The design doc for the payments-web agent had two proposals for the feedback signal, written by two engineers who each thought the other had missed something. Proposal A: run tsc --noEmit on every attempt, which takes 8 seconds on this repo with an incremental build info file, 34 seconds cold. Proposal B: run the Jest suite, which takes 11 minutes across 2,400 tests and is the only thing in the repository that has ever caught a rounding bug. The doc framed them as alternatives and asked for a decision, and the decision it was asking for does not exist, because the two proposals are not competing for the same slot.

Start with what each one alone would miss on this specific service. The refund handler dispatches on a discriminated union of six settlement outcomes — six object shapes distinguished by a literal kind field, which lets the type checker work out which branch of the switch is handling which shape — and the tests construct four of them; a patch that returns undefined for partial_reversal type-errors instantly and passes the suite, for the structural reason established two chapters ago. In the other direction, formatMinorUnits rounds half-to-even where the payment processor rounds half-away-from-zero, and every type in that function is correct — (cents: number, locale: string) => string is satisfied by any string at all, including the wrong one. Only a test that asserts "1.25" catches it, and the suite has that test because someone was burned by it in 2023.

The arithmetic of placement#

The reason 8 seconds and 11 minutes cannot swap places is not that one number is bigger. It is that the loop multiplies its inner cost by the attempt count while the edge pays once, so the same check has two completely different prices depending on where you put it.

The 147-hour row is the one that decides it. That is not merely expensive compute; it is a task latency that changes what the system is. A loop that answers in 5 minutes can be pointed at a ticket and checked on after coffee. A loop that answers in 30 lands its results into a queue nobody watches, and the reviewer who eventually reads the patch has lost the context they had when they filed the ticket. Meanwhile the marginal value of the suite's verdict on attempt 2, when the code does not yet type-check, is close to zero, because most of those runs are reporting failures that the 8-second check already reported more precisely.

That gives the asymmetric arrangement, and the asymmetry is the part worth stating carefully. The cheap check runs inside the loop and at the edge, because running it again at the end costs 8 seconds and guards against the loop's last edit having gone in after the last check. The expensive check runs only at the edge. "Both in both places" is not a compromise between the proposals; it is Proposal B with extra steps, since putting Jest inside the loop is what makes it Proposal B, and the 8 seconds of tsc at the end were never the thing anyone objected to.

Between the two there is a middle tier that most teams find within a month, and it is worth reaching for deliberately rather than by accident:

# agent-checks.yaml — what runs, and when
tiers:
  - name: shape
    when: every_attempt
    budget_s: 15
    commands:
      - eslint --max-warnings=0 $CHANGED_FILES   # 0.9 s
      - tsc --noEmit --incremental               # 8 s
  - name: related
    when: every_attempt_after_shape_passes
    budget_s: 90
    commands:
      - jest --findRelatedTests $CHANGED_FILES --silent   # 38 s median
  - name: full
    when: before_accept
    budget_s: 1200
    commands:
      - jest --ci --coverage=false               # 11 min
      - tsc --noEmit                             # 8 s, non-incremental

jest --findRelatedTests on a typical two-file patch runs 60 to 90 tests in about 38 seconds, which buys back a real slice of the behavioural class at a price the loop can pay two or three times. It is not a substitute for the full run, because a patch to a shared utility breaks tests in modules the dependency graph does not connect to the changed file, and it does not touch the integration suite at all. Tiering is a bet that most of a patch's behavioural risk is local, and that bet is right often enough to be worth 38 seconds and wrong often enough that tier 3 still has to exist.

What happens when the edge check fails#

Putting the suite at the edge raises a question the two-proposal framing hides: the agent has stopped, and now 11 minutes later something is red. Handing that straight to a human throws away the property that made the arrangement worth building, since the diagnostic is a test name and an assertion diff — exactly the kind of feedback the loop consumes well.

The arrangement that works is a second, much smaller loop at the edge, with its own budget of two attempts and the full suite as its check. It costs at most 33 minutes of wall clock in the worst case and it converts a meaningful share of edge failures into accepted patches: on payments-web, of the 29% of tasks that reached the edge red, roughly half were resolved within those two attempts, mostly snapshot updates — a snapshot test compares rendered output against a stored copy of the last accepted version, so an intended change to that output makes it fail until the stored copy is refreshed — and assertions that needed to move with an intentional behaviour change. The other half went to a human with the failing test named, which is a far better handoff than a patch and a hope. What the edge loop must not do is grow a third and fourth attempt, because at that point you have quietly rebuilt Proposal B and the arithmetic above reasserts itself.

The shape tier has a symmetric hazard worth pricing. A lint rule that fires spuriously — a project-local ESLint rule with a bad heuristic, a no-floating-promises complaint about a pattern the codebase uses deliberately — sits in the innermost position and taxes every attempt of every task. Two spurious rules costing one extra attempt on a fifth of tasks add about 68 wasted attempts a week, and worse, they teach the loop to write code that placates the rule rather than code that is right. The checks in the fast tier are the ones that need to be trusted absolutely, which argues for a conservative rule set there and for parking anything opinionated at the edge where a human sees it in context.

Deciding placement by measurement rather than argument#

Everything above is reasoning from cost, which is enough to settle tsc versus Jest and is not enough to settle the argument you will actually have, which is a teammate proposing to delete the type check because the suite already rejects things. That argument is empirical and you can run it in an afternoon.

Hold the task set fixed: 60 tickets from the last two quarters, each with a known-good human resolution, replayed from the commit that preceded them. Run the loop twice, changing exactly one thing — the presence of tsc in the shape tier — and record three numbers per arm. Useful patch rate is the share of tasks where a reviewer merged the agent's patch with no changes beyond formatting, which is a different and much lower number than the share that passed the checks. Attempts per task tells you what the missing check costs in iteration. Reviewer minutes per merged patch is the one the programme is ultimately spending, and it is measured with a timer, not a survey.

Removing the type check made the loop 30 seconds faster per task and cost nine reviewer-minutes per merged patch, which on 43 merges a week is about six and a half hours of senior engineering time bought with 21 minutes of compute. The eleven patches that reached review with type errors are the concrete form of the answer: they were not rejected by the suite, because the suite never walked those branches, so they became a human's job to notice. The numbers will differ in your repository — a codebase with 95% branch coverage and pervasive any would produce a much weaker result — which is the point of running it on your task set rather than quoting mine.

The same experiment answers the reverse proposal, and the discipline is to actually run it that way. Ablate the suite from the edge — remove that one component from the arrangement and re-run everything else unchanged — for a week on a low-risk service, and watch the escaped-defect rate, meaning the defects that clear every check the loop has and are found later by a user or an alert; if it does not move, the suite was not doing what you thought, and that is worth knowing before you spend another quarter maintaining 2,400 tests.

What the experiment does not settle#

A green run of both proposals, with the tiering in place and the numbers above in hand, still does not establish that the change does what the ticket asked, which chapter 3 covered and which no amount of placement work will fix. It also does not establish that the patch is safe to deploy: nothing in the check set knows that the migration it added locks a 400-million-row table, and tsc is content with a correctly typed DROP TABLE.

There is a subtler limitation in the measurement itself. The replay uses tickets whose resolutions you already know, which biases toward tasks that were resolvable, and it cannot measure the tasks nobody filed because the agent was known to be bad at them. Useful patch rate on a fixed historical set is a good instrument for comparing two configurations of your loop and a bad instrument for estimating what the programme is worth, because the denominator was chosen by looking backwards.

The instrument also invites its own corruption. Check-pass rate is easy to collect, moves in the right direction when you weaken the checks, and correlates with nothing anybody wants; useful patch rate and reviewer minutes are annoying to collect and are the ones that mean something. Which number a team ends up reporting to its leadership turns out to matter more than any of the placement decisions in this chapter, and it is the subject of the last one.

Everything here has assumed the tiers exist at all — that there is some mechanical verdict, at some price, to place. The next chapter runs the same harness on a task where there is not.