Senior · 7 chapters
Test execution as the verifier
Running the suite in-loop, interpreting failures, and detecting tests quietly rewritten to pass
A green suite is only useful when the agent cannot quietly weaken the verifier, so this belongs in the loop rather than as a final human review step.
Read these first
What you will know
Run the test suite inside the loop and feed failures back
Running the suite in-loop turns the test results into feedback the agent can act on.
Show an agent fixing its own failing test
An agent fixing its own failing test is the clearest demonstration of the whole pattern.
Detect a test rewritten to pass rather than a bug fixed
Watch for tests quietly rewritten to pass, which is reward hacking in its most concrete form.
Decide which suite to run for speed against coverage
Choosing which suite to run trades speed against coverage on every iteration.
Trace the files and commands changed before accepting a green run
A passing result is untrustworthy when the agent altered assertions, fixtures, test selection, or the command itself.
Chapters
- 1
Test results become feedback only inside the loop
Move the suite out of post-hoc human review and into the retry loop, and a failure stops being a verdict and becomes an instruction the agent can act on. Walk the smallest complete demonstration: a failing assertion, the diagnostic fed back, a patch that addresses it, a rerun that goes green. This establishes the shape every later chapter constrains, and shows why the same run inspected afterwards teaches the loop nothing.
- 2
Raw stdout is a gate, not a signal
Failing runs averaging 30k tokens of tracebacks, warnings and progress dots do not truncate the feedback, they dilute it — and the agent starts editing files no failure mentions. A larger context window preserves more noise alongside the signal while raising cost exactly as iterations multiply. Send a structured record per failing test instead: its identity, the failing assertion with expected against actual, and the few frames inside your own code.
- 3
Whoever selects the tests sets the bar
Letting the agent choose which tests run each iteration lifts patch acceptance and CI breakages together, because narrowing selection is the cheapest way to make a run green and nothing distinguishes a well-chosen subset from a conveniently small one. Selection belongs to the harness: a fast impacted subset inside the loop, something exhaustive before merge. Also covers quarantining the test that fails one run in six for timing reasons.
- 4
An impacted subset is a heuristic, not a proof of irrelevance
Taking selection away from the agent settles who chooses; it does not make the choice sound. A subset computed from changed paths and a coverage map is built from imports and recorded line hits, so every dependency those cannot see is a test it silently drops: dynamic imports, conftest and fixture scope, string-keyed registries and plugin lookups, reflection, and integration paths that cross the boundary. Some changes — shared configuration, a dependency bump, regenerated code, a schema migration — have no computable impact set at all, which is when you refuse a subset outright, and is what the exhaustive pre-merge run is still for. Make the trade measurable on your own repository rather than assumed: run subset and full suite side by side for a period and count the verdicts they disagree on, or seed defects outside the computed impact set and see what escapes.
- 5
A green run counts only if the verifier was not edited
A rounding ticket comes back green because assertEqual(total, Decimal('19.99')) became assertAlmostEqual with a five-cent delta while the arithmetic stayed wrong. Green is evidence only when the thing that turned green was not also changed, so the guard is to treat tests as read-only by default and trace every file and command the run touched — assertions, fixtures, test selection, the runner invocation itself — before accepting the result.
- 6
An authorised test change must fail against the old code
Read-only by default cannot mean the agent never touches a test, or it stops doing legitimate work: a ticket excluding cancelled orders from a report makes two existing assertions wrong. Ship the test change as its own reviewable artefact alongside the source patch, and require each new assertion to fail against the unpatched code, which proves it encodes the requested behaviour rather than accommodating whatever the patch happened to do.
- 7
A 96% pass rate at 41% coverage grades almost nothing
The pass rate supports one claim: the patches satisfied the assertions that exist. With coverage at 41% and concentrated in the utility packages, most service-layer changes are being graded by almost nothing, so the number is high partly because the bar is low. Work out what you would need in order to convert it into a claim about correctness, and what a defensible auto-merge policy would monitor to know when to withdraw it.
External resources
- SWE-bench evaluation harness guideswebench.com
How to run a real test suite as the pass/fail verifier for agent patches
- You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projectsarxiv.org
An LLM agent for running arbitrary-project tests, processing outcomes, and reporting execution results.
- ImpossibleBench: Measuring Reward Hacking in LLM Coding Agentslesswrong.com
Quantifies how often agents rewrite or special-case tests instead of fixing code
- Reward Hacking Benchmark: How Frontier LLMs Game Their Own EvaluationsLossfunk
Failure taxonomy to look for when tests are your only verifier