A whole-file rewrite ends at the output cap in silence

The change was small — add a currency check to one method in payments/reconcile.py, a 1,400-line module that has been in the repository for six years. The agent was set to whole-file rewrite, so it read the file, produced the new version, and the harness wrote it — the harness being the program wrapped around the model that assembles the prompt, calls the API, runs the tool calls the model asks for, and puts the resulting bytes on disk. The tool call reported success. The agent moved on to the next step, ran the test suite, got a SyntaxError from the collector, spent two iterations trying to fix a syntax error it could not find, and gave up. What was on disk when a human finally looked:

    def _match_settlement(self, batch, ledger_rows):
        matched = []
        for row in ledger_rows:
            if row.settled_at is None:
                continue
            candidate = batch.find(row.external_ref)
            if candidate is None:
                self._unmatched.append(row)
                cont

The file ends there, at line 806 of an intended 1,400, in the middle of the word continue. There is no error in the agent's log, no exception in the harness, no warning from the file write, and nothing in the tool result to suggest anything went wrong. The write did exactly what it was told: it took the string the model produced and put it on disk. The string was 594 lines short because generation stopped at the output token limit, and the harness never looked at the field that says so.

{
  "id": "msg_01Hq…",
  "stop_reason": "max_tokens",
  "usage": { "input_tokens": 17204, "output_tokens": 8192 },
  "content": [{ "type": "text", "text": "…                cont" }]
}

stop_reason is max_tokens rather than end_turn, and that single field is the whole diagnosis. Most harnesses read content[0].text and throw the envelope away, which turns a documented, deterministic, entirely predictable outcome into a mystery syntax error two iterations downstream.

It is worth tracing where the silence comes from, because every layer behaved correctly. The provider returned a well-formed response and labelled it accurately. The SDK — the provider's client library, which the harness calls instead of building HTTP requests by hand — deserialised it without complaint, since a truncated completion is a normal outcome and not an error condition. The harness extracted the text, which was a valid string. The tool wrote the string, and a write of 24 KB is not distinguishable from a write of 41 KB by anything the filesystem knows. No component was in a position to say the output was incomplete except the one that had the stop_reason in hand and was not looking at it. Under streaming the same information arrives as a message-delta event carrying the stop reason, and the same discarding happens for the same reason — the consumer is accumulating text.

The cap is also frequently lower than people believe, because max_tokens — the output cap, the ceiling on how many tokens a single response may contain, entirely separate from how much text the model can read — is a request parameter as well as a model ceiling. A harness that sets max_tokens: 4096 because that was the example in the quickstart has an effective cap half the model's, and the arithmetic below should be run against the number in the request rather than the number in the model card. Raising it is one line and moves the boundary from around 400 lines to around 800; it does not move it to 3,000.

The arithmetic that decides this before you run anything#

Code tokenises at roughly three to four characters per token, and a typical line of application code runs 30 to 50 characters including indentation, so a line of Python or Java costs somewhere between 8 and 14 tokens. Ten to twelve tokens per line is a serviceable planning figure, and it is worth measuring on your own repository once because the spread matters: dense Java with long identifiers and package-qualified types runs high, terse Python with short names runs low.

The 3,000-line row is the one that settles arguments. Against an 8k output cap the model would need four to five times the budget it has, so the rewrite cannot be emitted at all — not occasionally, not on a bad day, but structurally, on every attempt, for every prompt, regardless of how good the model is. And because the truncated text is still written, the outcome is worse than a refusal: the file is destroyed rather than edited. The original 3,000 lines are gone, replaced by the first 700-odd lines of a plausible-looking rewrite that no longer parses.

This is the arithmetic to run first when someone proposes a format, because it rules options out without any experiment. Given a single method to change inside a 3,000-line file and an 8k output cap, whole-file rewrite is eliminated on the numbers: 30k to 40k tokens required against 8k available. A search-and-replace block naming the old method body and the new one costs the size of the method twice, which for a 40-line method is roughly 800 to 1,000 tokens, comfortably inside the cap with room for the model to think. The tempting counter-argument is that a rewrite cannot fail to apply, since there is no anchor to miss and no coordinate to get wrong. That is true, and it is irrelevant when the format cannot be emitted in the first place. A format that cannot apply incorrectly is no use if it cannot be produced at all.

Truncation is one of two ways a rewrite loses code#

Hitting the cap is the mechanical failure, and it is the easier one because it leaves evidence — a stop_reason, a file that does not parse, a line count far below the original. The second way is elision, meaning the model leaves parts of the file out and gestures at them instead of reproducing them, and it is quieter.

Asked to reproduce 1,400 lines of which four are changing, a model will frequently decide that reproducing the other 1,396 is not a good use of anyone's time, and will write something like this:

    def _load_batches(self, cursor):
        # ... rest of the file unchanged ...

This is a complete response. stop_reason is end_turn, the output is well under the cap, the file may even parse if the elision lands between top-level definitions, and 900 lines of the module have been replaced by a comment. Aider's write-up "Unified diffs make GPT-4 Turbo 3X less lazy" measured this behaviour directly and named it laziness, and the finding worth carrying is that the format itself changes how often it happens: asking for a whole file invites the model to economise on the parts it considers boring, while asking for a diff of the change makes the boring parts something it never has to type.

Elision has a family of variants that are subtler than a comment saying so. A model reproducing a long file from context will occasionally drop a method it read past, reorder the import block into what it considers canonical order, normalise string quoting across the file, silently revert a line that a previous edit in the same session changed, or reformat a block to a different line width. Each of these produces a file that parses, passes review at a glance, and differs from the intended change in ways the diff will show but nobody may read, because a whole-file rewrite of a 1,400-line module produces a 1,400-line diff in which the four intended lines are somewhere in the middle.

The costs that do not show up as failures#

Output tokens are generated serially, one at a time, and no amount of hardware changes that for a single request. At 60 tokens per second — a reasonable figure for a large model under load, though it varies by provider and by time of day — a 1,400-line rewrite takes about four and a half minutes of wall clock, and a 3,000-line rewrite would take ten if the cap allowed it. A search-and-replace block for the same change is 800 tokens and takes about thirteen seconds. That is a factor of twenty on latency for an identical edit, and it lands inside an agent loop that may make five or six edits before the task is done.

The token bill scales the same way. At a representative $15 per million output tokens, a 1,400-line rewrite costs about 25 cents to emit once; a 40-line search-and-replace costs about one and a half cents. Reading the file into context costs the same under either format, so the multiple applies only to output, which is the expensive half.

The retry row is where the numbers turn from annoying into decisive. Every format has a failure rate, and every failure costs a fresh attempt at the same price. When an attempt costs four and a half minutes, a format with a 5% failure rate and a format with a 15% failure rate are separated by real money and real latency; when an attempt costs thirteen seconds, the same gap is affordable. Retry cost is a property of the format, and it multiplies the failure rate rather than adding to it.

Where a rewrite is the right answer#

None of this makes whole-file rewrite a bad format. It is the correct choice in several situations, and they are common enough that an agent with only one format will handle them worse.

Creating a new file has no anchor to match against and no existing content to preserve, so a rewrite is the only sensible shape for the output. Small files — under about 300 lines, comfortably inside any modern cap — rewrite cleanly, and the apply step becomes a write() that cannot fail on ambiguity, which removes an entire class of retry. Changes that genuinely touch most of a file, such as migrating a module from callbacks to async or renaming a type that appears forty times, produce a search-and-replace patch nearly as long as the file with forty separate opportunities for an anchor to miss; a rewrite is both cheaper and safer there. And a weak model that cannot reliably reproduce an anchor byte for byte may do better with a rewrite of a small file than with any format that requires exact quotation, which is a point worth remembering when the agent has to run on more than one model.

Two cheap guards make the rewrite path safe enough to keep. Check stop_reason on every response and treat max_tokens as a hard failure that never reaches the filesystem, and compare the emitted line count against the original before writing, rejecting anything that shrinks by more than a threshold the task did not ask for. Both are a handful of lines and both catch the failures in this chapter before they touch disk.

The arithmetic pointed at search-and-replace for the surgical edit in a large file, and that format has its own failure modes rather than none. It replaces a coordinate the model must compute with a string the model must reproduce exactly, which trades an arithmetic problem for a copying problem. The next chapter is about what happens when the copy is not exact, including the case where the search text is rejected as absent from a file in which a human can see it plainly.