Validation depth has to match the blast radius of the edit

The ticket asked for a field rename: OrderResponse.total_cents had been a lie since the Japanese launch, because yen has no cents, and the field should be total_minor_units. The agent edited platform/schemas/order.py, updated the six references inside the module, and ran the loop. The patch applied at offset zero, the file parsed, ruff was clean, mypy on the package was clean, and pytest tests/platform/schemas reported 96 passed in 12 seconds. The loop accepted the edit, wrote a summary, opened a pull request and exited.

CI, eleven minutes later, produced this:

✗ generate-check          packages/api-client/src/generated/order.ts is stale
                          run `make generate` and commit the result
✗ build //services/web-checkout
    src/cart/summary.tsx:44:31 - error TS2339:
      Property 'total_minor_units' does not exist on type 'OrderResponse'.
✗ build //services/ops-console
    src/orders/table.tsx:78:19 - error TS2339:
      Property 'total_minor_units' does not exist on type 'OrderResponse'.
2 targets failed, 1 check failed

Every verdict the loop collected was accurate. platform/schemas/order.py really did parse, really was type-clean within its package, really did pass its tests, and the six references really were the six references in that module. The seventh consumer was a TypeScript file generated from the schema and committed to the repository, and the eighth and ninth were two services that import it. The loop had already exited, so nothing retried; a human picked the failure up the next morning, spent twenty minutes working out what had happened, and ran the generator by hand.

The graph the checks did not consult#

The suite that ran covers one edge out of five, and it is the only edge that does not leave the package. Nothing about that is a bug in the suite; a schema package's tests are supposed to test the schema package. The mismatch is between where the checks reach and where the change reaches, and the second of those is a property of the file that was edited rather than of the edit itself. How far a change can reach is its blast radius: the set of code that could stop working because of it. For platform/schemas/order.py that set contains a generated TypeScript file and the two services built on it; for a private helper in a leaf module it contains nothing outside the module, and the same six checks are the right answer in one case and the wrong one in the other.

Costing the obvious fix#

The suggestion that follows this incident, reliably, is to run the full build and the full suite on every loop iteration. It is worth costing rather than dismissing, because the argument for it — correctness beats speed — is not wrong on its own terms; it is wrong about what it buys and what it spends.

In this repository bazel build //... takes 8 minutes 20 seconds cold and 2 minutes 10 seconds with a warm remote cache, and bazel test //... adds 6 minutes. Agent tasks in this repository average 3.4 edit iterations before the loop is satisfied, so a full build inside the loop adds between 7 and 28 minutes per task depending on cache warmth, against a current mean task wall time of 4 minutes 10 seconds. The median accepted edit is 12 lines in a single file. Making a 12-line edit cost half an hour does not make the agent more careful; it makes the agent unusable for the class of work that is most of its value, and teams respond to that by turning the checks off rather than by waiting.

The deeper problem with the uniform answer is that it spends the same budget on every edit when the edits differ in reach by orders of magnitude. A docstring fix in a leaf module and a field rename in a shared response model get the same eight minutes, which is simultaneously far too much for one and — because the full build still does not re-run the code generator — not actually enough for the other.

A rule that keys on where the file sits#

What the loop needs is a classification computed from the path being edited and the repository's own dependency graph, producing a validation tier that the loop then executes. The classification runs in milliseconds; only the tier it selects costs anything.

CONTRACT_GLOBS = (
    "platform/schemas/**", "**/openapi/*.yaml", "**/*.proto",
    "packages/*/src/index.ts", "**/migrations/*.py",
)

def blast_radius(path: str, graph: DepGraph) -> int:
    if fnmatch_any(path, ("**/migrations/*.py", "infra/**", "**/flags/*.yaml")):
        return 3                       # runtime or deploy-time contract
    if fnmatch_any(path, CONTRACT_GLOBS):
        return 2
    external = graph.rdeps(path) - graph.package_of(path).members
    if external:
        return 2                       # imported from outside its own package
    if graph.is_package_exported(path):
        return 1
    return 0

TIERS = {
    0: ["parse", "lint", "module_tests"],
    1: ["parse", "lint", "typecheck_package", "package_tests"],
    2: ["parse", "lint", "typecheck_package", "regenerate_and_diff",
        "compile_rdeps", "rdep_tests"],
    3: ["parse", "lint", "halt_for_human"],
}

The rdeps call is the part that has to be fast, and build systems already answer it. bazel query 'rdeps(//..., //platform/schemas:order)' returns the transitive reverse dependencies of a target — everything that depends on it, whether directly or through a chain of intermediates, which is the ordinary dependency relation read backwards — in about 1.5 seconds against a warm graph; the equivalents are pnpm why, turbo run --filter='...[HEAD]', go list -deps, or a cached import index built once per session with grep-grade tooling if the repository has no build graph at all. Computing it once at session start and caching it costs a few seconds and serves every edit in the run.

A repository with no build graph at all can still answer the question, and the fallback is cheaper than it sounds. One pass over the tree, parsing each file and recording its import statements, produces a forward edge list that inverts into rdeps; on this repository's 12,400 Python and TypeScript files that pass took 4.1 seconds using the same parsers from the syntax gate, and the index only has to be rebuilt when a file's import block changes, which the loop already knows because it wrote the edit. The result is less precise than a build system's answer — it resolves module paths rather than build targets, and it misses re-exports through index files unless you follow them — and it is precise enough to distinguish "nothing outside this package imports it" from "eleven things do", which is the distinction the tiers are built on.

The classification also has to be legible after the fact, which means recording it rather than only acting on it. An acceptance record that says tier 2 (matched platform/schemas/**; 2 reverse deps outside package: services/web-checkout, services/ops-console) lets a reviewer disagree with a specific clause, and gives whoever maintains CONTRACT_GLOBS the data to notice that a glob has been matching a directory that moved six months ago. Tier assignments drift the way every path-based rule drifts, and the cheapest defence is that every decision leaves a sentence explaining itself in a place people already read.

Applied to the incident, platform/schemas/order.py matches CONTRACT_GLOBS, comes back tier 2, and the loop runs make generate, diffs the working tree, finds packages/api-client/src/generated/order.ts changed, includes it in the edit, compiles web-checkout and ops-console, and hits the same two TS2339 errors CI hit — inside the loop, with a model still attached, twelve minutes before a pull request exists.

The shares come from classifying one month of merged agent edits in this repository after the fact, and they are the number that makes the scheme affordable: the expensive tier fires on roughly one edit in seven, so the mean added cost per edit is on the order of 30 seconds rather than 8 minutes. That is the trade being made — not fewer checks, but checks proportioned to reach, with the expensive ones reserved for the edits that can reach far enough to justify them.

Tier 3 deserves the separate treatment it gets. A migration that applies cleanly and passes its tests can still be irreversible against a production database, and there is no check the loop can run in a sandbox that establishes it is safe to run against 400 million rows. Escalating validation depth is the right response to a wide static blast radius; refusing is the right response to a blast radius that leaves the repository entirely.

Where this rule is too strict, and the fix that goes wrong#

Path-based classification over-escalates in one obvious way: a typo fixed in a docstring inside packages/api-client/src/index.ts matches a contract glob and buys a five-minute build for a change that cannot affect anything. Over a month those cost about 40 minutes of loop time on this repository, which is not nothing.

The refinement is to look at the diff rather than only the path, and demote when the change provably cannot alter behaviour — an edit confined to comments, docstrings, or whitespace outside string literals. Deciding that requires the parse tree the syntax gate already built: compare the token streams of the before and after content with comments and whitespace filtered out, and demote by one tier when they are identical. That is a sound test in the sense that a token-identical change cannot alter compiled behaviour in any language considered here, which is exactly why the tempting looser versions of it are dangerous. A regex for lines starting with # demotes an edit that changed a docstring which happens to be a doctest, or a comment that is a # type: annotation, or a // @ts-expect-error directive, all of which are load-bearing. The demotion has to come from the grammar, and a shortcut that comes from a pattern match will eventually demote an edit that changes the program.

Where this rule is too permissive#

The classification keys on the static dependency graph and on a list of paths someone wrote down, so every contract that is not visible to either is invisible to it.

Dynamic references are the first and most common gap. A serializer named by a string in a settings module, a Celery task addressed by dotted path, a Django get_model("orders", "Refund"), a plugin registry populated by entry points — none of these appear in an import graph, so rdeps returns nothing outside the package and the edit is classified tier 0. The reference is real and resolves at runtime, and the failure surfaces when that code path runs, which may be a nightly job.

Data-level contracts are the second, and they are worse because no amount of graph analysis reaches them. If total_cents is also the key serialised into a Kafka topic that another team's repository consumes, or the key in a JSONB column with four years of history in it, then renaming the field breaks consumers that this repository has never heard of and cannot compile. The static graph is bounded by the repository; the contract is not. The mitigation is not clever — a rule that treats any change to a serialised field name as tier 2 regardless of what the graph says, plus a human-maintained list of paths whose contents cross the repository boundary — and it is a rule maintained by people, which means it goes stale in exactly the places nobody is thinking about.

The third gap is the one where the rule is right about reach and wrong about risk. A tier 0 file with no external importers can still change behaviour that another team depends on through an HTTP call, and validation depth measured in compile units has nothing to say about it. That case is not solvable by classification, and pretending otherwise is how a scheme like this acquires more trust than it has earned. What the tiers buy is that the failures which are mechanically visible get seen while the loop can still act on them, and that is a narrower promise than "the edit is safe".

None of this helps once a check has already failed. A tier 2 edit that compiles its reverse dependencies and finds two of them broken has produced exactly the outcome the scheme was built for, and it has also left a modified generated client and a modified schema sitting in the working tree of a machine that belongs to somebody. Getting those back out — precisely, without touching anything else in that tree — is a mechanism with its own way of going badly wrong, and it is the subject of the next chapter.