A diff that keeps changing is activity, not progress

Two runs on the same modernisation backlog report the same headline at the same moment: 380,000 tokens, 22 iterations, still running. A dashboard that shows totals shows them as identical. One of them has certified nine of twelve steps and is four minutes from finishing. The other has certified two, has been on step three since iteration eight, and will burn its entire remaining budget without certifying a third.

The totals were never going to separate them, because a total is an integral over a trajectory and the two trajectories differ everywhere except in their sum. What distinguishes them is available at every iteration, cheap to record, and usually discarded — the harness runs the check, reads the exit code, and throws away the output that says which checks failed.

The record one iteration owes you#

Two units are in play here and they are close enough to blur. An iteration is one turn of the loop: the model acts, its tool calls run, and something reports back. An attempt is a complete try at one step, ending when that step's certifying check is run against the result, so on a step cheap enough to check every turn the two coincide, while on a step whose check takes four minutes an attempt spans several iterations. An iteration ends with a validation run, and the thing worth keeping is the identity of what failed, not the count. Counts are a projection that loses the property everything downstream needs, which is whether this iteration's failures are the same failures as last iteration's.

def record_iteration(run, step, attempt, result, patch, usage):
    failing = normalise_failures(result.stdout)        # a set of stable ids
    prev    = run.last_failing_set(step)
    emit({
        "run": run.id, "iter": run.iter, "step": step["id"], "attempt": attempt,
        "check": step["check"][-1], "exit": result.exit, "secs": result.secs,
        "failing": sorted(failing),
        "n_failing": len(failing),
        "resolved":  sorted(prev - failing),           # gone since last attempt
        "persisting":sorted(prev & failing),           # unmoved
        "new":       sorted(failing - prev),           # introduced by this attempt
        "patch_lines": patch.added + patch.removed,
        "patch_distance": levenshtein_ratio(patch.text, run.last_patch(step)),
        "tool_calls": usage.tool_calls,
        "tool_errors": usage.tool_errors,
        "tokens_in": usage.tokens_in, "tokens_out": usage.tokens_out,
        "usd": usage.usd,
    })

normalise_failures is the part that decides whether any of this works, and it is easy to get wrong in a way that makes every set comparison return "everything is new". Raw check output carries line numbers, absolute paths, durations, goroutine addresses and timestamps, all of which shift between attempts even when the failure is identical.

FAIL_RE = re.compile(r"^\s*--- FAIL: (\w+)")
ERR_RE  = re.compile(r"^(\S+\.go):\d+:\d+: (.+)$")

def normalise_failures(out):
    ids = set()
    for line in out.splitlines():
        if m := FAIL_RE.match(line):
            ids.add(f"test:{m.group(1)}")
        elif m := ERR_RE.match(line):
            msg = re.sub(r"\b0x[0-9a-f]+\b", "0xADDR", m.group(2))
            msg = re.sub(r"\b\d+(\.\d+)?(ms|s)\b", "DUR", msg)
            ids.add(f"build:{m.group(1)}:{msg}")       # file + message, no line no.
    return ids

Dropping the line number from a compiler error is deliberate. session.go:22:14: not enough arguments in call to parseSessionToken and the same error at session.go:26:14 after an unrelated edit shifted the file are the same unresolved problem, and treating them as different failures is how a thrash detector — the rule that decides a loop has stopped getting anywhere and keeps acting regardless, which the next chapter builds — reports healthy churn for an hour.

Reading the two runs side by side#

With per-iteration records, the two runs stop looking alike immediately. Below are the iterations that fall on step three in each, which is where they diverge.

Run A's failing set shrinks monotonically, each patch is smaller than the last, and the resolved column accounts for the drop. That is what buying correctness looks like: successive attempts are addressing a shrinking remainder, and the patch converging in size says the model is narrowing rather than rewriting.

Run B's failing count is flat at nine, and the two columns that explain why are persisting — not shown, but it is the same nine identifiers in every row — and patch_distance, which stays near 0.2, meaning each attempt discards roughly eighty percent of the previous one and writes something substantially different. The patch is growing, the tool error rate is climbing, and the token cost per iteration is rising because the accumulated diff being reasoned over is larger each time. Row 11 and row 12 are the giveaway: something moved, three new failures appeared, one attempt cleared them, and the original nine never budged. That is search, not progress.

The instrumentation does not say which iteration to stop at. It says, with evidence, that run B's iterations 8 through 13 produced 111,000 tokens of spend, a diff that grew from 180 to 291 lines, and zero net change in what the check objects to.

Five signals, and what each one is blind to#

The failing-set trajectory is the primary signal, and it needs all three of its derived columns rather than the count. A count that stays at nine is compatible with nine unmoved failures and with nine entirely different failures each time, and those are opposite situations — the first says the attempts are missing something structural, the second says the attempts are breaking roughly as much as they fix. What the trajectory cannot see is a step whose check is coarse enough that a genuine improvement does not move it, which is the case for any step certified by a build, since a build fails identically until it does not. On steps certified by compilation, the failing-set columns carry much less information than they do on steps certified by a test suite, and the honest response is to know which kind of step you are watching rather than to pretend the signal is uniform.

Patch distance between consecutive attempts on the same step separates refining from restarting. A distance near 1.0 alongside a shrinking failing set is convergence — the model is keeping what worked and adjusting the remainder. A distance near 0.2 means each attempt discards most of the previous one, which is a fresh guess rather than a refinement, and a run doing that repeatedly is sampling the space rather than reasoning about it. The measure is blind to a run refining a patch in a corner of the file that has nothing to do with the failure, so it is only ever read alongside the failing set; on its own it would score a stubbornly wrong run as beautifully convergent.

Diff volume against the base is the cost the human eventually pays, and it deserves its own line rather than being inferred from patch size. Run B's step-three patch at 291 lines is not simply seven times the review work of run A's 44-line one, because it is 291 lines about which nothing is known: no check passes over it, and no attempt in the record explains what any part of it was for. Tracking git diff --stat against the last certified commit at every iteration gives an early warning that the review bill is growing while the certified count is not, which is the specific shape of a run that will end by handing a human more work than the task contained.

Tool error rate is the cheapest signal to collect and the one most often left out of dashboards. Run B's climb from zero to five tool errors an iteration is a run losing its grip on the environment — reading paths that no longer exist, running commands with arguments assembled from a stale picture of the tree, hitting timeouts it then retries verbatim. A rate above roughly ten percent of calls tends to precede the point where a run stops being able to help itself, and it is worth alerting on independently because it often rises before the failing set stops moving. It says nothing at all about a run making confident, well-formed tool calls toward the wrong goal.

Spend per certified step is the number to put in front of anyone asking whether the agent is worth running at all. Run A has certified nine steps for 380,000 tokens, about 42,000 per step. Run B has certified two, about 190,000 per step, and the second of those was certified at iteration seven, so its marginal cost since then is unbounded and its marginal yield is zero. The metric only means anything with the step definitions from chapter one behind it, since a run whose steps cannot be certified individually has no denominator, and it lags by construction — it summarises what already happened, which is why the per-iteration signals exist alongside it.

None of this is expensive to collect. The check is being run anyway at the end of each attempt, so the only added cost is parsing its output and appending a record, which is about 1.2 KB of JSON per iteration and under fifty milliseconds. The one decision that costs something is cadence: on steps where the certifying check takes four minutes, running it after every model turn rather than after every attempt turns a cheap signal into most of the run's wall clock, so the record is emitted per attempt and a much cheaper proxy — compile only, or a single test file — runs in between.

What this instrumentation is not#

It is not a stopping policy. Everything here produces evidence, and none of it makes a decision. The gap matters because the obvious decision rule — stop when the failing count stops falling — is wrong in a way the next chapter has to handle carefully, since there are legitimate runs whose failing set is constant for a long stretch by construction.

It is also not a substitute for the evaluation work that came before it. Trajectory metrics are within-run signals about one task; they say whether this run is converging and nothing about whether the agent converges on tasks of this shape generally. A run that thrashes on step three and recovers is a data point for the eval suite, not a verdict on the agent.

And these are not quality signals. A run whose failing set drops monotonically to zero in four iterations has satisfied its checks, and chapters one and three both put the same ceiling on what that means. Perfect trajectory metrics over weak checks describe a run efficiently converging on something nobody has verified is right.

The signals are now available at every iteration, in a form a policy could read. Nothing in the loop reads them. Run B, on the strength of these records, will continue to iteration twenty-two and beyond, because the only thing stopping it is an attempt limit that knows nothing about what the attempts contained.