An impacted subset is a heuristic, not a proof of irrelevance

LEDG-2301 changed the signature of format_money(amount, currency) to format_money(amount, currency, *, locale=None) and updated eleven call sites. The harness computed a 187-test subset from the changed paths, the run came back green in 34 seconds, and the pre-merge run on the merge result failed with four errors in tests/exports/test_ledger_csv.py. The CSV exporter does not import format_money. It looks up formatters by name out of a registry populated from an entry-points table in pyproject.toml, calls them with positional arguments only, and the fourth positional argument it passes is a column width that used to be silently ignored and is now a TypeError.

There is no arrangement of static analysis that would have put those four tests in the subset. The dependency is real, it is exercised on every export in production, and the only trace of it anywhere in the repository is a string in a TOML file. The subset was not wrong about the imports; it was answering a question about imports and being read as an answer about relevance.

What a coverage map is actually made of#

The phrase "impacted subset" hides two separate mechanisms, and both of them have edges. The static half walks import edges from changed files outward: if tests/billing/test_settlement.py imports ledger.services.settlement, a change to that module puts the test in the set. The dynamic half is a recorded artefact — a coverage run from some earlier point in time, storing which test executed which lines — so a test that touched line 88 of settlement.py during the recording is selected when line 88 changes. Between them they cover a great deal, and everything they cover, they cover because a name was written somewhere a parser or a tracer could see it.

Every dependency expressed without such a name is invisible to both halves:

The last row is the one that quietly does the most damage, because it degrades rather than fails. A coverage map rebuilt weekly is, on a repository merging 40 pull requests a day, describing a codebase that no longer exists by the end of the week, and the tests it knows least about are exactly the newest ones — which are attached to the code most recently changed. This team rebuilds the map nightly from the full suite run they were doing anyway, and any test node id absent from the map is added to every subset unconditionally, which costs about 30 extra tests on a normal morning and is the cheapest safety property in the whole system.

Two things follow from the table that are worth stating plainly. Improving the map is worth doing and does not converge — each mitigation is a rule someone wrote after an escape, so the set of rules is a history of past failures rather than a theory of the language. Determining the true impact set of a change in a language with runtime name resolution is not a solved engineering problem with a budget attached; it is undecidable in the general case, meaning no analysis exists that gets it right for every program, and the practical versions are all approximations that trade recall for speed. The right conclusion is not to distrust the subset but to stop asking it for a guarantee it was never computing.

Changes with no impact set worth computing#

Some diffs make the whole exercise meaningless, and recognising them is a policy decision rather than an analysis. A change to a shared settings module, a dependency version bump, a regenerated protobuf or OpenAPI client, a schema migration, or a repository-wide codemod — a scripted edit applied mechanically to every file matching a pattern — all have the same property: the set of things that could be affected is approximately everything, so a subset computed from changed paths will be either enormous or absurdly small, and neither answer is informative. A requirements.txt bump of sqlalchemy from 2.0.29 to 2.0.34 touches one file, imports nothing, and has a computed impact set of zero tests.

That is what no_subset_paths in the harness config is for, and it is worth writing out because most teams discover it one incident at a time:

selection:
  no_subset_paths:
    - "ledger/settings/**"
    - "ledger/db/migrations/**"
    - "**/conftest.py"
    - "pyproject.toml"
    - "poetry.lock"
    - "requirements*.txt"
    - "ledger/generated/**"       # protobuf and OpenAPI clients
    - ".github/workflows/**"
  full_suite_timeout_min: 35
  unmapped_tests: always_run

Refusing a subset means the loop runs the 22-minute suite for those tasks, so the loop is slow for them, and that is the correct outcome — a migration ticket that takes three attempts at 22 minutes each is 66 minutes of runner time against a defect class that reaches production. What should not happen is the compromise that gets proposed instead, which is running the subset for these changes anyway and relying on the pre-merge run to catch problems. That converts an in-loop signal into an after-the-fact rejection, and everything from the first chapter about the difference applies with more force here, because the agent iterating against an empty test set for four attempts produces a patch shaped by nothing at all.

Measure the gap on your own repository#

The size of the gap between subset and full suite is a property of your codebase, not of the technique, and two experiments will give you the number in a fortnight.

Before running either, be clear about what the answer changes, because a 4% miss rate is not automatically an argument for abandoning subsets. The subset is 40 seconds against 22 minutes, which is a factor of 33, and that factor is what makes three or four attempts possible inside a single job. Replacing it with the full suite would turn a four-attempt budget into 88 minutes of runner time per ticket and push the loop into overnight batches, at which point the timing argument from the first chapter starts to erode — an agent iterating once a night is closer to the reviewer at 16:40 than to a loop. The subset earns its place by being fast and approximate, and the experiments tell you how approximate, so that the pre-merge run and the review policy can be sized against a measured number rather than a feeling.

The shadow run is the cheaper one. For every task that ends green, run the full suite as well, record both verdicts, and count the disagreements. Only one direction of disagreement matters: subset green, full suite red. Over 214 completed tasks this team found 9 such disagreements, or 4.2%, with a median of one failing test each and a maximum of 26 — that maximum being a conftest.py change from before the no-subset rule existed. Six of the nine traced to registry and entry-point lookups of the format_money kind, which is what turned that pattern into a standing always_run_for rule.

Defect seeding gives you a sharper number and costs more to run. Take the computed impact set for a change, then inject a plausible defect outside it — flip a comparison operator, drop a boundary case, off-by-one a slice — and see whether anything catches it. The interesting statistic is not the raw catch rate but the split of what escapes:

60 seeded defects, injected outside the computed impact set
  caught by the subset anyway .................. 12   (the map was over-conservative)
  caught only by the full pre-merge suite ...... 34
  caught by no test in the repository .......... 14

Twelve caught by the subset anyway means the map is doing better than its own theory, usually because a broad integration test happened to be selected. Fourteen caught by nothing is a different finding entirely and has nothing to do with selection — those are places where the repository has no verifier at all, and no scheduling of the tests you own will produce a signal there. That number is a coverage problem, and it comes back with force in the final chapter.

Two checks before you accept a green subset#

Given all of the above, a subset that comes back green on attempt three is worth something specific and bounded, and there are two things worth looking at before you treat it as a pass.

The first is whether the run actually executed the change. A subset can be green because it exercised the modified lines and they behaved, or because it exercised none of them, and those are different events wearing the same colour. The number that measures it is diff coverage, the share of the patch's own changed executable lines that the run actually executed, and computing it is a small amount of work since you are already collecting coverage:

def diff_coverage(diff, run_coverage):
    changed = diff.executable_lines()               # file -> {line numbers}
    hit = {(f, l) for f, lines in changed.items()
           for l in lines if run_coverage.was_executed(f, l)}
    total = sum(len(v) for v in changed.values())
    return len(hit) / total if total else 0.0

A patch of 60 executable changed lines with a diff coverage of 0.08 has been graded on five lines, and the green means almost nothing regardless of how many tests ran. This team gates at 0.7 for auto-acceptance and routes anything below it to review with the number attached, which is a far better use of a reviewer than reading a diff that the suite already covered.

The second is whether the attempt history is consistent with a fix. A patch that goes green on attempt three after failing on one and two should show the failures moving — different tests, or the same test with a different actual value — because that is what progress looks like. Three attempts where the same assertion fails identically twice and then passes, with the third diff touching a fixture or a configuration file rather than the code path in the failure record, is a pattern worth stopping on. What that pattern usually indicates is that the third patch changed the verifier rather than the behaviour, which everything so far has assumed the agent could not do. It can, the loop as built gives it every opportunity, and the concrete form it takes is narrow enough to catch mechanically.