Eviction is the other half of the budget

The ceiling from the previous chapter was in place — 70k for the first turn's fill, derived from a p90 diagnostic block of 24k — and the run still died. The trigger was a signature change in a base class:

attempt 2/4  tool=typecheck  status=error
mypy: found 431 errors in 118 files (checked 2,847 source files)
tool_result_tokens=61,204   budgeted=24,000   overflow=+37,204
harness: transcript would reach 231,204 / 200,000 — trimming
harness: evicted 4 file blocks (retrieval ranks 5,6,7,8) to fit

Eviction is what the last two lines of that log describe: the harness removing material it has already placed in the transcript so that the next request fits back inside the window, a decision it makes on its own, mid-run, with no view of the task. Four files that the builder had selected on the first turn were gone by the second, and one of them was the module defining the base class whose signature had just changed. The next attempt patched a subclass to match a signature it could no longer see, guessed the argument order, and produced 431 errors again in a slightly different arrangement.

Every number in the ceiling calculation was right. What was missing is that a budget you can only set at turn 0 is not a budget at all, because the material that overflows arrives later and arrives at a size nobody controls. The fill is the half of the window a builder chooses; tool results, diagnostics and the agent's own superseded diffs are the half it inherits, and by attempt three the inherited half is larger. Holding a budget through a run that misbehaves is a different discipline from sizing one, and it needs its own machinery.

Nothing here has a retrieval score#

The reason the harness above evicted the base class is that it was the only thing it knew how to rank. Retrieval scores exist for file blocks and for nothing else, so a trim step that sorts by score and drops the tail will always eat the repository context first, regardless of whether the 61k of type errors it is protecting has any value. There is no score for a stale diff, no score for the tail of a pytest run, no embedding that says the 300th mypy error is less useful than the 3rd. The ranking has to come from somewhere else, and the thing that supplies it is the block's role in the loop rather than its similarity to the task.

Sorting by role gives a small number of tiers, which is enough.

The order that matters most is that superseded material goes before repository context. An attempt that has already been replaced by a later attempt has served its purpose, which was to stop the model from trying the same thing twice, and that purpose survives compression to a single line — attempt 1: added tz-aware conversion in _apply_vat; failed test_vat_rounding[JP]. Twelve tokens do the work that 6,000 were doing.

Truncating at arrival, not at overflow#

The 61k block from mypy, the type checker this repository runs over its Python source, should never have entered the transcript at full size. A tool result is the one piece of context that arrives with a known structure, which means it can be reduced by that structure instead of by a character count, and the reduction can happen at the boundary where the result is received.

MAX_TOOL_RESULT = 24_000

def compress_typecheck(raw: str) -> str:
    """Keep the first error per file, plus a count of what was dropped."""
    errors = parse_mypy(raw)                       # [(path, line, msg), ...]
    by_file: dict[str, tuple[int, str]] = {}
    for path, line, msg in errors:
        by_file.setdefault(path, (line, msg))

    head = [f"{p}:{ln}: {m}" for p, (ln, m) in list(by_file.items())[:40]]
    dropped = len(errors) - len(head)
    if dropped > 0:
        head.append(f"... {dropped} further errors in {len(by_file) - 40} files")
    return "\n".join(head)

Keeping the first error per file is not an arbitrary heuristic. A type checker reports cascades: one wrong signature in a base class generates an error at every call site, and the 430 downstream errors say nothing the first one did not. The same shape holds for pytest, where the useful content is the assertion and the last few frames of each distinct traceback, and for a build log, where it is the first error and the target that emitted it. Blind truncation to the last N tokens is the version that is easy to write and wrong for every one of these formats, because the tail is where the summary lives and the head is where the cause lives.

Where a structural rule does not exist — a tool returning an opaque blob, or a fetched HTML page — the fallback is head-and-tail with a marked gap, and a note in the transcript that says how much was removed so the model can ask for the rest through a narrower query. Silent truncation is the version that produces confident nonsense, because a model reading a traceback that stops mid-frame has no way to know it was cut.

The budget itself belongs on the tool rather than on the harness, declared next to the schema, because the person who knows that a type check can return 431 errors is the person who wrote the type-check tool. A tool definition that carries max_result_tokens: 24000 alongside its parameters lets the compression step be chosen per tool and lets the total loop growth be computed by summing over the tools a task will call, which is what turns the ceiling arithmetic of the previous chapter from an estimate into something checkable at configuration time.

Truncation is also not the only response to an oversized result. A tool can paginate — return the first 24k with a cursor, and expose a continue parameter the agent calls when it needs more — which converts a decision the harness has to make blindly into one the model makes with the task in view. This costs a round trip, so it suits results where the tail is occasionally the point, such as a long integration-test log where the interesting failure is the last of forty. It suits type-checker output poorly, since the model has no way to know whether error 41 matters before reading it, and would burn four round trips finding out.

The copy that went stale#

There is a second failure that the tiers above do not describe, and it is a correctness bug rather than a budget one. When the agent successfully writes to billing/invoice.py at attempt 1, the copy of that file sitting in context is now wrong — it describes the state of the repository before the edit. If the transcript also contains the diff, the model is being asked to reconstruct the current file by mentally applying a patch to an older version, and it does this badly in exactly the way you would expect: line numbers in the next hunk are computed against the stale copy, and the apply step rejects them.

attempt 2/4  tool=apply_patch  status=error
patching file billing/invoice.py
Hunk #1 succeeded at 87 (offset 12 lines)
Hunk #2 FAILED at 212
1 out of 2 hunks FAILED -- saving rejects to billing/invoice.py.rej

The fix is to treat every successful write as an invalidation. The block holding that file is marked stale, and before the next attempt the harness re-reads the file from disk and replaces the block in place. It costs one tool call and the same token count as the block it replaces, which is why it is worth doing eagerly rather than waiting for the model to ask. The alternative that teams reach for first — leaving the old copy and trusting the diff — saves nothing, since both the stale copy and the diff are occupying the window anyway, and it introduces a class of failure that only shows up after the second edit to the same file.

Putting the three rules together, the pass that runs before every model call looks like this.

The loop back from drop bulk files to the fit check is where an important property lives: the pass keeps removing until it fits or until only pinned material is left, and if only pinned material is left and it still does not fit, the correct behaviour is to fail the run with a clear reason rather than to start trimming the pins. A harness that will evict anything will eventually evict the task statement, and an agent that has lost the task statement does not stop — it keeps working on whatever the most recent diagnostics imply, which is how a run ends with a green build and a patch nobody asked for.

What this costs, and what it is not#

Eviction is not free in the way its name suggests. Removing a block from the middle of a transcript invalidates the cached prefix from that point onward, so every subsequent token has to be prefilled again. Prefill is the pass in which the model reads its input before writing anything, and a cached prefix is the leading run of input tokens a provider will reuse at a reduced price and latency when it is byte-identical to the previous call's, which together mean that where an eviction falls in the transcript costs as much as how large it is. In this harness, a mid-transcript eviction at attempt 3 added between four and seven seconds to the turn and multiplied the input cost of that call by roughly eight relative to a cache hit. That is the argument for doing the work at arrival — compressing a tool result before it is appended costs nothing, because the prefix has not been built yet — and the argument against a policy that trims aggressively on every turn.

Cache behaviour also settles an argument about ordering within a tier. Dropping the oldest bulk file first is the intuitive choice and the most expensive one, because everything after it has to be prefilled again; dropping the newest evictable block invalidates the least. The two rules disagree about which file to lose, and the disagreement is small in practice, since bulk files are by definition the ones no diagnostic has mentioned. Where they conflict, the cheaper rule wins on any turn that is not the last, and a run that is one eviction away from failing should stop optimising for cache and take whatever fits. Re-reading a stale file has the same property, which is a reason to replace the block in place at the same position rather than appending the fresh copy at the end where it would read as newer evidence than the diff that changed it.

There is one ambiguity in the pin set worth resolving before it bites. "The most recent diagnostics" is unambiguous when one tool runs per attempt and stops meaning anything when three do — a type check, a lint pass and a test run produce three results, and pinning only the last of them pins whichever finished most recently rather than whichever matters. The rule that survives contact with a real toolchain is to pin the most recent result per tool, and to let the previous result from each tool become superseded material as soon as a newer one arrives. That also gives the compression step something useful to do, since the diff between the last two type-check results is usually three lines and says more than either result alone.

It is also worth being precise that this is not compaction. Compaction rewrites the whole transcript into a summary and restarts, which is a reasonable move at the end of a long session and a bad one mid-task, because the thing being summarised is exactly the detail the next patch depends on. Eviction is selective and structural: named blocks leave, the rest is untouched, and the ordering is knowable in advance so you can reason about what a run will look like at attempt four before you launch it. Nor is it a sliding window, which drops the oldest material and would therefore drop the task statement first.

The honest limitation is that every rule here is about material the loop produced. The tiers assume the initial fill was worth keeping and only argue about the order in which to give it up, and the truncation rules assume the tool output is worth having at all. Neither addresses the possibility that the 70k of repository code chosen at turn 0 was actively making the patch worse — that a file which the model could see, and which had nothing to do with the task, is what put the wrong import in the diff. That possibility is not hypothetical, and it is what the next chapter takes apart.