Four checks catching four disjoint failure classes

Two days after the caching patch merged, the orders-api pods began emitting AttributeError: 'NoneType' object has no attribute 'get' at roughly forty occurrences per minute, all from the same line of the pricing handler, and all for tenants whose price tables had been sitting untouched long enough to go stale. The patch had been produced by the agent, reviewed by a human in about six minutes, and merged on a green pipeline. The loop that produced it ran ruff check, then black --check, then pytest, and stopped when all three reported success, which they did on the fourth attempt, 41 seconds after the agent started.

The defect is visible in nine lines:

# cache.py
from time import monotonic

_STORE: dict[str, tuple[float, dict]] = {}

def put_entry(key: str, value: dict, ttl_s: float = 300.0) -> None:
    _STORE[key] = (monotonic() + ttl_s, value)

def get_entry(key: str) -> dict:
    record = _STORE.get(key)
    if record is None:
        return {}
    expires_at, value = record
    if monotonic() > expires_at:
        del _STORE[key]
        return None
    return value

Three of the four exits from get_entry return a dictionary, and the fourth returns None while the signature promises a dict. Every caller in the service does something like get_entry(key).get("unit_price"), which is correct against the annotation and fatal against the implementation. The path is reachable only when a key exists and has passed its TTL — the time-to-live, the number of seconds after which a cached entry is meant to be treated as gone — and the existing tests cover a hit and a miss, so nothing in the suite ever holds a key long enough — or fakes a clock far enough forward — to walk it.

Here is what the agent saw before it declared itself finished:

$ ruff check .
All checks passed!
$ black --check .
All done! 14 files would be left unchanged.
$ python -m pytest -q
..........................................                        [100%]
42 passed in 6.10s

And here is what a fourth check says about the same tree, without executing a single line of the service, without a clock, and without a key that has expired:

$ mypy cache.py
cache.py:16: error: Incompatible return value type (got "None", expected "dict[str, Any]")  [return-value]
Found 1 error in 1 file (checked 1 source file)

That took 0.6 seconds on a warm cache. The interesting part is not that mypy is clever, because the check it performed is close to trivial. The interesting part is that no amount of investment in the other three would have produced this verdict. Adding fifty tests to the suite does not help unless one of them happens to construct an expired key. Enabling every optional rule in ruff does not help, because ruff is matching patterns inside a single file's syntax tree and return None inside a function is a legal pattern that appears thousands of times in any codebase. Reformatting is orthogonal to behaviour by construction, since black is a function from source text to equivalent source text.

What each verdict is a verdict about#

The word "check" has been carrying four different jobs, and the disagreements between them are where the value is. Each of the four answers a different question about the tree, and each is deaf to the questions the others answer.

The compiler answers whether this is a legal program in the language: the tokens parse, the symbols resolve, the arity is right, and in a language with a serious static front end, whether ownership, exhaustiveness and effect discipline hold. In Rust or Go or TypeScript this is the heaviest of the four and it subsumes a great deal of what other languages leave to a separate type checker. In Python it is thin — python -m compileall catches syntax errors and nothing else, and a misspelled attribute survives happily until the line executes — which is exactly why the type-checker column matters more in a Python service than a Go one, and why an agent loop copied from a Go repo to a Python repo silently loses a whole failure class.

The type checker answers whether the program is internally consistent across every path, including paths no input will ever take today. It reasons over the control-flow graph — the map of every route execution could take through the code, built by reading it rather than by running it — so an unexercised branch is worth exactly as much to it as a hot one, and it composes claims across module boundaries: get_entry promises dict, the pricing handler consumes a dict, and the promise is broken in one branch. This is a fundamentally different observation from the suite's, because the suite has to reach a line to say anything about it and the type checker never has to reach anything.

The linter answers whether the code matches known-bad or non-idiomatic shapes, almost always within one file and usually within one function. Ruff ships something on the order of 800 rules and runs over a 40,000-line codebase in around 40 milliseconds, which makes it the cheapest signal available and the one an agent can afford to consult after literally every edit. What it buys is a real defect class — mutable default arguments, bare except, an f-string with no placeholders, a comparison against a tuple that is always truthy, an unused import that means the agent deleted the wrong line — and what it cannot buy is any claim that involves two files or a value at runtime.

The test suite answers whether the program produces expected values on the specific paths some test actually constructs. It is the only one of the four that observes behaviour, which makes it the only one that can catch a proration calculation that is off by three cents, and it is also the only one whose coverage is written by hand and therefore stops exactly where somebody's attention stopped. On the orders-api service the suite took 6.1 seconds; on the payments monolith it takes 11 minutes; either way it is the slowest of the four and the one whose blind spots are least predictable, because they are shaped by the history of what previous engineers thought to write down.

Read the fourth column rather than the second. The reason to run all four is not that four opinions are better than one, and it is not redundancy in the engineering sense of two systems covering for each other's downtime. It is that the fourth column of each row is non-empty and the rows do not fill each other's gaps by accident — they fill them because the questions were different to begin with.

Twenty defects, three buckets, and one wrong inference#

The way to stop arguing about this is to seed defects deliberately. Take twenty known defects — commits reverted from your own history are the best source, since they are defects your codebase actually produced — and run each one through the agent's accept-or-reject loop — the cycle in which the agent produces a patch, the checks run over it, and the patch is either accepted or handed back with the diagnostics for another attempt — with all four checks wired up. A representative outcome on a well-tested Python service looks like six rejected by the type checker but passing the suite, four rejected by the suite but type-checking cleanly, seven caught by more than one check, and three that clear all four and are still wrong.

Each bucket describes the verifier set rather than the agent. The six say that a large slice of the defects this codebase produces are contract violations on paths nobody exercises, which is a statement about the shape of the code and the shape of the test suite, not about how good the model is at Python. The four say that behaviour still escapes static reasoning, mostly as arithmetic, ordering, and off-by-one boundaries where every type is correct and every value is wrong. The three say the set has a hole, and the size of that hole is the number you should quote when someone asks how much of review the agent has taken over.

The tempting inference from those numbers is that the suite is the strongest single verifier and the rest can go, and a teammate will make it in a specific form: the type checker missed ten of the twenty defects, the suite rejected four of those ten, so the suite is doing the type checker's job anyway. That reasoning treats the buckets as overlapping samples of one underlying quantity called "defect-catching power", and they are not samples of one quantity at all. The six the type checker rejected are exactly the ones the suite structurally cannot see, because seeing them would require a test to walk a branch that no test walks — which is not a gap you close by running the suite more often or caring about it more. Deleting the type check does not lose you six-tenths of a redundant signal; it loses you all of a distinct one, and it trades an 8-second gate for an 11-minute one that does not cover the same ground.

Where the four do overlap, honestly#

The disjointness is a strong tendency rather than a theorem, and it is worth knowing where it leaks. Both mypy and ruff will flag a call to a function that does not exist, from different directions. A test that exercises a branch with a None in it will fail with a TypeError where the type checker would have refused the annotation, so the two do sometimes catch the same instance by different means, usually with the suite arriving later and with a worse message. Linters have grown rules that encroach on type checking — ruff's flake8-bugbear set catches some things pyright also catches — and type checkers have grown flow analysis that encroaches on tests. On the twenty-defect run above, seven of the twenty were caught by more than one check, which is a substantial fraction and does not change the argument, because the decision about a check is made on the defects only it catches.

What the four together do not give you is any claim about whether the change was the right change. The get_entry incident is a story about a hole in the verifier set, and it has a tidy ending: install mypy, put it in the loop, and that entire class stops reaching production. The three defects that cleared all four checks have no such ending, and they are the ones worth the next chapter.