A detection rate is a measurement you made, not a property a check has

The design document says the scoped type check catches roughly a third of defects, and the footnote points at a conference talk. The talk is about a Scala monorepo with a stricter effect system than anything in your repository, the measurement was over a corpus of student submissions, and the number was reported to two significant figures. Somebody has already used it to argue that the type check is enough and the impacted unit tests can leave the inner loop, which would remove the only check in the staircase that observes behaviour, on the strength of a figure that was never about this codebase.

There is no fact of the matter about how many defects a type checker catches. There is only a fraction: the numerator is the seeded defects that a given check rejected, the denominator is the seeded defects in the corpus, and the number tells you about the corpus at least as much as about the check. If the corpus is full of null-dereference mutations, the type checker looks superb. If it is full of off-by-ones, the type checker looks useless. Both runs are honest and neither transfers. Producing the number yourself is not a nicety; it is the only version of the number that means anything for your loop.

Building a corpus that resembles the defects you get#

Two sources, used together, because each covers what the other misses.

The first is mutation-style seeding, borrowed wholesale from mutation testing, where a program transforms the source in a small, semantically meaningful way and the question is whether anything notices. The operators that matter here are the ones whose products look like agent output rather than like corruption:

# libcst parses Python into a syntax tree that can be edited and printed back
import libcst as cst

class OperatorSwap(cst.CSTTransformer):
    """Relational operator replacement: <= becomes <, == becomes !=, ..."""
    SWAPS = {
        cst.LessThanEqual: cst.LessThan,
        cst.GreaterThanEqual: cst.GreaterThan,
        cst.Equal: cst.NotEqual,
        cst.Is: cst.IsNot,
    }

    def __init__(self, target_index: int) -> None:
        self.seen = -1
        self.target = target_index

    def leave_Comparison(self, node, updated):
        op = updated.comparisons[0].operator
        replacement = self.SWAPS.get(type(op))
        if replacement is None:
            return updated
        self.seen += 1
        if self.seen != self.target:
            return updated
        target = updated.comparisons[0].with_changes(operator=replacement())
        return updated.with_changes(comparisons=[target])

The full operator set is small and each member corresponds to a mistake that shows up in real patches: relational operator replacement, boundary shift (n to n + 1 in a slice or a range), condition negation, removed await or removed .result(), swapped arguments of the same type, an early return None inserted on a guard path, a field reference changed to a sibling of the same type, and an exception handler widened to bare except. Each seeded mutant is one operator applied at one site, saved as a patch, and stored with the file, line and operator that produced it.

The second source is your own history, which is where the defects that actually happen here are recorded. Take merge commits whose message matches a fix convention, isolate the hunk that changed non-test source, and reverse-apply it to HEAD. The result is the historical defect reinjected into the current code, and unlike a mutant it carries the shape of mistakes this team makes against this domain — the tenant filter that got dropped, the timezone that was assumed to be UTC, the retry that was not idempotent. These are harder to generate, because the reverse-apply frequently does not compile against a codebase that has moved on, and roughly one in three survive on a two-year-old repository. Those that do are worth several mutants each.

There is a third source once the loop is running, and it eventually becomes the best one: every patch a human rejected during review, stored with the reason. Those are, by definition, defects your agent produces on your codebase, which is the exact distribution the corpus is trying to approximate.

Knowing when to stop seeding#

A rate computed over 40 seeds is not a rate. With 12 of 40 caught, the point estimate is 30% and a Wilson interval at 95% runs from about 18% to 45%, which is wide enough to contain both "this check is essential" and "this check is barely worth running". A confidence interval is the range of true rates consistent with what you observed, at a stated level of confidence, and the Wilson form is the one to use for a proportion measured over small counts, where the textbook formula misbehaves near zero and one. The interval narrows with the square root of the count, so the practical schedule is to seed in batches and watch both the estimate and the interval:

The estimate stopped moving in any decision-relevant way after 80 seeds; the interval kept tightening, and at 400 it is still nearly five percentage points wide on either side, which is a useful reminder about how much precision a corpus of this size can buy. The last column is written in pp, percentage points, because the distance between two rates of 31% and 26% is five points and calling it five percent would mean something else entirely. Stop when further batches move the estimate by less than the difference you would act on. If you are deciding between two checks whose rates differ by 20 points, 160 seeds is plenty. If you are deciding between two that differ by 4, no realistic corpus will settle it and you should decide on latency instead.

Run the checks in sequence, over one corpus#

The most consequential mistake in this measurement is running each check against the corpus independently and then reading the four resulting percentages as if they were a menu. Independent rates overlap, heavily, because the checks are not sampling defects at random — they are all aimed at the same code, and the easy defects are easy for everything.

Run them instead in the order they will actually run in the loop, and for each seed record the first check that rejected it:

LADDER = ["lint", "typecheck", "unit", "integration"]

def first_catcher(seed_patch: Path, workspace: Path) -> str | None:
    apply_patch(workspace, seed_patch)
    try:
        for check in LADDER:
            result = run_check(check, workspace)   # records duration + records
            if result.failed:
                return check
        return None                                 # escaped every check
    finally:
        revert(workspace)

The two columns that come out of this differ enough to change decisions:

The integration build's headline of 61% and its marginal contribution of 6 points are both true, and only the second one describes what adding it to the loop would purchase. It catches nearly everything the type checker catches, because a program that fails to type-check fails to build; it catches most of what the unit tests catch, because it runs them. What is unique to it is the small set of defects that only appear when packages are wired together — a version skew between two internal libraries, a generated client that no longer matches its schema, a service that fails to start because a config key was renamed. Six points is a real and unusual six points, and it is six, not sixty-one.

Latency belongs in the same measurement#

The same runs that produce the detection column produce the timing column, at no extra cost, provided the harness writes a record per invocation rather than a running average:

{"ts":"2026-03-04T11:22:41Z","check":"typecheck","scope":"packages/billing",
 "duration_ms":2870,"cache":"hit","files_changed":2,"diagnostics":3,"exit":1}
{"ts":"2026-03-04T11:31:08Z","check":"typecheck","scope":"packages/billing",
 "duration_ms":39610,"cache":"miss","files_changed":2,"diagnostics":0,"exit":0}

Averaging those two gives 21 seconds, a duration that never occurred and describes nothing. The distribution is the thing, and it is read at two points: the p50, or median, which half of runs come in under, and the p95, which only one run in twenty exceeds. A scoped type check at 3 seconds p50 and 40 seconds p95 is not a 21-second check, it is a 3-second check that occasionally is not. A p95 thirteen times the p50 is a bimodal distribution wearing a percentile's clothing, and bimodal almost always means a cache with two states.

Reading the p95 as "sometimes it is slow, average it out" hides all five of those. The p95 is also the number the loop's timeout has to respect, because a 30-second timeout on a check whose p95 is 40 seconds manufactures the harness-error case from chapter 1 on one attempt in twenty, and a loop that mistakes a timeout for a failing check will hand the agent an empty diagnostic list and ask it to fix something.

What the corpus is not#

A seeded corpus measures the checks against defects you generated, and the generator is a model of agent behaviour rather than agent behaviour itself. Mutation operators skew mechanical: they produce single-site, single-token changes, while a real agent patch is frequently twenty lines implementing a slightly wrong idea, and no operator in the set above produces "the agent reimplemented pagination without the cursor". That skew flatters checks that are good at local defects, which is most static analysis. Blending in the replayed history and the rejected patches is what keeps the corpus honest, and the blend ratio is worth stating whenever the number is quoted.

Two mechanical hazards are worth naming. Equivalent mutants — a seeded change that alters the source without altering behaviour, such as flipping a comparison whose operands are always equal — are uncatchable by definition, and they sit in the denominator lowering every rate uniformly. Seeds in code that nothing reaches do the same thing. Neither ruins comparisons between checks, since both affect all checks equally, and both make the absolute rates read lower than they should, so a 38% escape rate is an upper bound rather than an estimate.

Finally, a detection rate over a corpus is not an escape rate in production. It says what fraction of seeded defects a check rejects, which is a statement about the checks, and the fraction of real defects that reach users depends additionally on what your agent generates, what reviewers catch, and what your traffic exercises. The corpus is for ranking checks against each other under one consistent question. Turning those rankings into a decision about which check runs on every attempt requires putting the rates next to the latencies and multiplying by how many attempts a task takes, which is the next chapter.