Raw stdout is a gate, not a signal

The quarterly numbers on the agent loop looked like a scaling problem. Mean attempts per completed task had gone from 2.1 in January to 3.8 in April, cost per task from $0.19 to $0.36, and the only thing anyone could point at was that the suite had grown by about 900 tests over the same period. The theory in the room was that harder tickets were arriving. Then somebody pulled the diffs from the failed attempts and counted where the edits landed: across 140 attempts, 31% of the changed hunks were in files that no failing test mentioned anywhere in its traceback. On LEDG-2251, a settlement rounding ticket, attempt two edited ledger/utils/dates.py — a module the failing test does not import, which had appeared in the run output only inside a DeprecationWarning about utcnow().

That is the shape of the problem, and it is not a shortage of context. The failing runs averaged 30,400 tokens and the model's window was 200,000, so nothing was being cut. Everything the agent needed was in front of it, buried in everything else, and the everything else was full of file paths.

Where thirty thousand tokens go#

It is worth breaking a real failing run down, because the intuition that "the output is mostly the failure" survives only until someone counts. This is one run of the impacted subset on LEDG-2251, 214 tests selected, one failing, captured with default pytest settings and the project's usual plugins:

The two rows that carry the defect are 1,010 tokens out of 30,400, which is 3.3% of the payload. Roughly 78% of the output names files, and almost all of those files are irrelevant to this failure. A model asked to produce a patch after reading that is doing selection under a strongly adversarial prior: the most frequent identifiers in its recent context are dateutil, sqlalchemy.orm.session, and ledger/utils/dates.py from the deprecation warning, and the identifier that matters appears twice.

The mechanism connecting volume to wandering patches is that mention is a weak form of instruction. Everything in the context is evidence about what the task is; a file path that appears eleven times in warnings is, statistically, part of what the run is about. There is no marker in raw stdout that separates "this line explains the defect" from "this line is a deprecation notice from a library you do not control", so the separation has to be done by inference, and inference over 30,000 tokens of near-uniform-looking terminal text is exactly the operation that degrades as the volume climbs. The effect compounds across iterations: attempt two's off-target edit to ledger/utils/dates.py broke nothing, so attempt three received a failing run that now also contained the original defect plus a slightly different warnings summary, and the agent had one more file in its history to justify touching.

The bigger window buys the wrong thing#

The proposal that follows from a diagnosis of "not enough context" is to move to a model with a much larger window, and it is worth taking seriously enough to say precisely what it changes. It changes whether the output fits. On this repository the output already fits, and had fit every single run of the quarter, so the change purchased is zero. What a larger window does not change is the ratio of signal to noise in the payload, which stays at 3.3% no matter how much room there is around it.

The cost side is where the proposal actively hurts. Iterations were multiplying — 2.1 to 3.8 over the quarter — and every iteration pays the full input cost of the feedback again, because the failure record is re-sent with each new attempt along with everything that came before it. Larger-window models are typically priced above their smaller siblings, and long-context pricing tiers on several providers step up once a request passes a threshold, so the same run gets more expensive per attempt at the moment the attempt count is rising. Multiply a 1.7× iteration increase by a per-token price increase and the cost curve gets steeper rather than flatter.

The row that matters when someone pushes back is the last one. A larger window is the correct fix for a different failure, which is a suite whose output legitimately exceeds what you can send — a 4,800-test full-suite run with 60 failures after a bad dependency bump will produce something in the hundreds of thousands of tokens, and no amount of formatting makes all of it fit. That case exists. It is not this case, and the diagnostic that separates them is simply whether anything was dropped.

A record instead of a transcript#

There is an intermediate proposal that deserves dismissing on the way past, which is to hand the 30,400 tokens to a cheap model and ask it to summarise the failure. It costs a model call and 2 to 4 seconds per iteration, it is stochastic in exactly the place you want determinism, and it can drop the single frame that mattered while producing a fluent paragraph that reads as if nothing was lost. The information you want is not hidden in the output in a way that requires interpretation; it is present in the runner's own data structures with the fields already separated, and a summarising model is an expensive way to recover something you threw away by capturing stdout instead of a report object.

What goes back to the agent should be assembled from the runner's structured output rather than scraped from its terminal rendering. Every mainstream runner exposes one: pytest has report hooks and JUnit XML, Go has test -json, Jest has --json, JUnit has its XML report. Using them removes the parsing fragility as a side effect, but the reason to do it is that the structured form has already separated the fields that raw text mashes together. In pytest the collecting happens in conftest.py, the file the runner loads automatically for every test beneath its directory:

# conftest.py — collect one record per failing test
import json, pytest

PROJECT_ROOT = "ledger/"

def _own_frames(entries, limit=4):
    frames = [e for e in entries if PROJECT_ROOT in str(e.path)]
    return [
        {"file": str(e.path), "line": e.lineno + 1, "code": e.statement.lines[-1].strip()}
        for e in frames[-limit:]
    ]

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    report = (yield).get_result()
    if report.when != "call" or not report.failed:
        return
    repr_ = report.longrepr.reprtraceback
    RECORDS.append({
        "test_id": report.nodeid,
        "exception": repr_.reprentries[-1].reprfuncargs.__class__.__name__,
        "message": str(report.longrepr.reprcrash.message).splitlines()[0][:400],
        "frames": _own_frames(repr_.reprentries),
        "duration_s": round(report.duration, 2),
    })

The record for LEDG-2251's failure comes out at 118 tokens:

{
  "run": {"selected": 214, "passed": 213, "failed": 1, "duration_s": 38.4},
  "failures": [{
    "test_id": "tests/billing/test_settlement.py::test_partial_refund_nets_to_zero",
    "message": "AssertionError: Decimal('-0.02') != Decimal('0.00')",
    "frames": [
      {"file": "ledger/services/settlement.py", "line": 88,
       "code": "net = self._credits() - self._debits()"},
      {"file": "tests/billing/test_settlement.py", "line": 54,
       "code": "assert ledger.net_position() == Decimal('0.00')"}
    ],
    "duration_s": 0.31
  }]
}

Four decisions inside that format are load-bearing. The test identity is kept verbatim and in full, because the agent needs to be able to re-run exactly that test and because the node id — the runner's address for a single test, its file path, class and function plus the parameter set when a test is run once per input case — encodes the parametrisation that often is the defect. The assertion message is truncated to the first line at 400 characters, since a diff of two 900-element lists teaches nothing that the first mismatching element does not. Frames are filtered to the project root, which is what removes the eleven sqlalchemy frames, and capped at the last four, because the deepest frames in your own code are where the defect lives and the shallow ones are the call path the agent can reconstruct from the repository. Warnings and captured logs from passing tests are dropped entirely.

Caps have to be applied at the run level too. Fifty failing tests at 120 tokens each is 6,000 tokens of nearly identical records, and past a handful of failures the useful information is the pattern rather than the list. A run with more than eight failures should send the first three records in full, then group the remainder by exception type and top project frame with counts, which turns "48 more failures" into one line that says ImportError: cannot import name 'Money' — 48 tests, all under tests/billing/.

What the record cannot carry#

The compression is lossy by design, and the losses are not evenly distributed. Collection errors are the clearest case: when a module fails to import, there is no per-test report to build a record from, the run has zero failures and zero passes, and a naive implementation reports success or reports nothing. The fallback for a collection error, a segfault, a killed worker, or a run that exceeds its timeout is to send the tail of raw output, because in those cases the raw output is the only artefact and it is short.

Flaky failures — the ones that come and go without the code changing, usually because of timing, ordering or state shared between tests — come through the record looking exactly like real ones, which is a problem the format cannot solve — the same assertion, the same frames, the same message, and no field that says "this passed twice this morning". That distinction has to come from run history rather than from the run.

Some defects also genuinely need the wide view. A test failing because a fixture in a distant conftest.py changed shape — a fixture being a named piece of setup that a test receives by asking for it by name — will produce frames inside ledger/ that look fine on their own, and the agent, given only those frames, will edit correct code. The mitigation is to include any project-root frames from setup and teardown phases rather than only the call phase, and to accept that a small share of tasks will need a second pass with more output. Measured on this repository, that share was 6 of 140 attempts, against a 31% off-target hunk rate before the change.

Measurement is what turns this from a plausible story into a decision you can defend. Four numbers were enough here: mean attempts per completed task, tokens per completed task, off-target hunk rate — the share of changed hunks in files no failure record names, which is the direct measure of wandering — and the CI breakage rate on merged patches, watched to make sure the compression had not removed something the agent was quietly relying on. Over the six weeks after the switch, attempts fell from 3.8 to 2.4, tokens per task from 41k to 16k, and off-target hunks from 31% to 9%, with CI breakages flat at 4%.

That last number stayed flat because the loop was still running the same tests it had been running before, and nothing in this chapter touched which ones. The off-target hunk rate is now low enough that the remaining failures look like genuine ambiguity rather than distraction, which makes it a good moment to ask a question the feedback format cannot answer: of the 4,812 tests in the repository, why were those 214 the ones that ran, and who decided?