Detect the unmoving failure set, recover once, then stop

The run is on attempt fourteen of a twenty-attempt budget. The failing check set has been the same nine identifiers since attempt ten, the patch has been rewritten from scratch each time and now stands at 291 lines, and 400,000 tokens have gone through the loop. A teammate looking at the trace proposes raising the attempt limit to thirty, on the grounds that the agent was clearly still trying.

It was. The record from the previous chapter says exactly what the trying consisted of: four materially different patches, none of which changed what the check objects to. That is evidence about the space being searched. If four substantially different attempts produced no movement in the failure set, the property those attempts share — the framing of the step, the information in the context, the check's own output — is where the obstacle lives, and six more samples from the same distribution have no reason to land differently. The extra ten attempts cost roughly 300,000 more tokens and end with a larger diff for a human to unpick.

Two signatures, and the different bills they run up#

The cheapest signature to detect is a tool call repeating verbatim and failing verbatim. It appears as three or more consecutive iterations in which the normalised call and its error are byte-identical, with no intervening edit that could have changed the outcome.

iter 11  bash("go test ./internal/api/ -run TestTokenRoundTrip")  exit 1  "token_test.go:88: got \"\", want \"abc123\""
iter 12  bash("go test ./internal/api/ -run TestTokenRoundTrip")  exit 1  "token_test.go:88: got \"\", want \"abc123\""
iter 13  bash("go test ./internal/api/ -run TestTokenRoundTrip")  exit 1  "token_test.go:88: got \"\", want \"abc123\""

That is a loop with no state change, and it is usually a harness problem rather than a reasoning problem — an edit that silently failed to write, a stale build cache, a container whose filesystem is read-only, a tool wrapper swallowing an error. It is worth catching first because it is unambiguous and because it is cheap: three identical iterations cost perhaps 40,000 tokens before the detector fires.

The costly signature is the one in the opening. The failing set is constant across attempts whose patches differ substantially, so nothing looks stuck — every iteration produces new code, new reasoning, new tool calls, and a growing diff. It runs up a much larger bill precisely because it looks like work.

The detector#

The rules are small enough to read in one screen, and the important design choice is that they operate on the per-iteration records rather than on the model's own account of how it is doing.

def classify(records, step, cfg):
    r = [x for x in records if x["step"] == step["id"]][-cfg.window:]
    if len(r) < 3:
        return "running"

    # 1. identical call, identical error, no edit in between
    tail = r[-3:]
    if (len({(x["last_call"], x["last_error"]) for x in tail}) == 1
            and all(x["patch_lines"] == 0 for x in tail[1:])):
        return "stuck_environment"

    # 2. tool errors dominating
    calls = sum(x["tool_calls"] for x in r)
    if calls and sum(x["tool_errors"] for x in r) / calls > cfg.tool_error_rate:
        return "stuck_environment"

    # 3. unmoving failure set under changing diffs
    sets = [frozenset(x["failing"]) for x in r[-cfg.flat_attempts:]]
    if len(sets) == cfg.flat_attempts and len(set(sets)) == 1:
        churn = mean(x["patch_distance"] for x in r[-cfg.flat_attempts:])
        if churn < cfg.refine_threshold and not step.get("monotone_metric"):
            return "stuck_search"
        if step.get("monotone_metric") and not decreasing(
                [x["metrics"][step["monotone_metric"]] for x in r]):
            return "stuck_search"

    return "running"

Defaults that have held up in practice: a window of six iterations, flat_attempts of three for cheap steps and four for steps whose checks take minutes, a refine threshold of 0.5, and a tool error rate of 0.10. The thresholds are not delicate — the difference between three and four flat attempts is one attempt's worth of spend, and every value in that config should be per-step overridable, because a step certified by a four-minute integration suite has a different economics from one certified by a nine-second build.

Recovery is one bounded pass, not a mode#

Detecting the signature is the easy half. What the policy does next is where most implementations either do nothing useful or do the wrong thing repeatedly.

Recovery has four actions and then it is over. The uncertified patch is set aside — stashed under the run id, not deleted, so nothing is lost and nothing contaminates the retry — and the tree returns to the step's rollback boundary. The context is rebuilt from durable state as if this were a resume, which it effectively is: the assembled sections, the tree read fresh, the ledger's record of what each of the last four attempts tried and what each one's failure was. The step itself is re-planned narrower, which usually means splitting it into two steps with their own certifying checks, so that the next attempt is aimed at a target small enough for a check to localise. Then exactly one attempt runs under the new framing.

If that attempt moves the failure set, in either direction, the run returns to normal operation with the narrower steps. If the failure set is unchanged, the policy stops and hands off, meaning it ends the run and passes the task to a person along with the state of the tree and the evidence behind the stop, in the form the last chapter of this topic specifies. The one-shot bound is the part people are tempted to relax, and relaxing it recreates the original problem one level up: a recovery procedure invoked repeatedly under unchanged evidence is itself a form of sampling, and the second recovery has no more reason to succeed than the ninth attempt did. A second recovery on the same step is permitted only when something has actually changed, which the records can answer — the failure set differs, or a new failure appeared and was resolved, or the human supplied information.

The environment signature gets a different and shorter recovery, because its diagnosis is different. There is no point re-planning a step when the evidence says the harness is not doing what it was told, so the recovery is a set of assertions about the world: does the file the last edit claimed to write actually contain the change, does the command run outside the agent's wrapper produce the same error, is the working directory what the run thinks it is, is the disk full. Those checks take a few seconds and they resolve into one of two outcomes. Either the environment is broken, in which case the run stops immediately and the handoff says so, and the human's decision is about infrastructure rather than about code. Or the environment is fine and the run genuinely repeated an action for no reason, which is a context problem and reduces to the same rebuild-from-durable-state that the search recovery performs.

Separating the two matters because the responses are opposite in cost. A broken sandbox that gets fed into a re-planning recovery produces a narrower step that also cannot be edited, a second recovery, and a stop forty minutes later with a handoff blaming the code. Checking the environment first costs seconds and turns that into an accurate two-line diagnosis. It is also the case where stopping early is unambiguously correct, since no amount of model capability compensates for a filesystem that is not accepting writes.

Against the proposal to raise the limit to thirty, the numbers make the case without any appeal to principle. Eight attempts under one framing produced zero movement in the failing set; treating those as trials, the optimistic estimate of the per-attempt chance of movement is about one in ten, and six more attempts at roughly 30,000 tokens each buys 180,000 tokens of spend for a coin-flip's worth of chance, plus a diff that grows another few hundred lines. Redirecting the same budget into a bounded recovery costs one attempt and changes the thing that all the failed attempts had in common. The change worth making is to the framing, the check's diagnostic output, or the information the step has — not to the number of samples drawn from a distribution that has already been sampled eight times.

The run this detector stops for no reason#

Rule three fires on a legitimate case, and it is worth naming precisely because it is common enough to matter. A codemod converts forty packages to a new interface. The shared package cannot be converted until all consumers are, so the type checker reports the same nine errors in internal/shared after every batch, unchanged, while each attempt produces a large and completely different patch — different packages, different files, hundreds of lines each time. Four attempts in, the failing set is identical and the patch distance is low, and stuck_search fires on a run that is doing exactly what it should.

The fix is the monotone_metric escape in the detector. A step is allowed to declare a secondary progress counter that the harness can evaluate against the tree — remaining unconverted call sites, remaining files matching a pattern, remaining TODO(migration) markers — and while that counter is strictly decreasing, an unmoving failure set is not evidence of thrash. The counter has to be computed by the harness rather than reported by the model, for the reason established earlier: a progress number the model supplies is the model grading itself, and a stuck run is exactly the situation where that number is least reliable.

Two other legitimate runs get caught. A step certified by a check with a quarantined flaky test produces an unmoving failure set consisting entirely of that test, which is why the certifying check has to be hermetic and why quarantined tests belong outside it. And a run that is deliberately bisecting — inserting logging, running an experiment, reverting it, running another — produces large, different, low-similarity patches and an unchanged failure set by design, because none of those attempts were trying to fix anything. That one is worth supporting explicitly: an attempt the model marks as an experiment is recorded but excluded from the flat-attempt count, and the marking is cheap to abuse only in the direction of running longer, so it needs its own small cap.

What stopping does not do#

The detector is not a correctness signal in either direction. A run it stops may have been one attempt from a correct patch, and a run it never fires on may be converging steadily on something wrong. It measures whether the loop is moving, which is a much narrower claim than whether the loop is right.

It also does not replace budget caps. Token, wall-clock and attempt limits stay, as the backstop for failures the signatures do not describe — a run making steady, expensive progress on a task that turns out to be far larger than anyone estimated will never trip a thrash detector and should still stop at some point. The detector's job is to convert the common case from "burn the whole budget and produce nothing" into "spend about a fifth of it and produce a diagnosis".

What it produces at the end is a stopped run, a tree in some state, and a set of records. Someone now has to be told, and what they are told, along with what the run did with the work before it stopped, is the difference between a stop that saves money and a stop that transfers the cost onto a person at three in the morning.