Backoff answers a transport failure; a bad request needs a new one

The nightly reconciliation agent had a bad week in March. Its job was ordinary: for each account flagged by billing, pull the invoices for a date range, compare them against the payment ledger, and write a short note explaining any gap. It ran 24,000 times that week and returned nothing at all on 1,900 of them, which finance noticed before engineering did. The trace of a failed run was the same every time. Step three called list_invoices with {"account_id": "acc_7741", "start_date": "03/14/2026", "end_date": "03/21/2026"}, and the tool answered:

{
  "status": 400,
  "error": {
    "code": "invalid_argument",
    "field": "start_date",
    "message": "invalid date format: expected YYYY-MM-DD"
  }
}

The HTTP client under the tool was configured the way the whole service was configured, with five attempts and exponential backoff at 1, 2, 4, 8 and 16 seconds. It made all five, took 31 seconds, and received the identical 400 each time, because nothing about a malformed date changes while a client sleeps. The wrapper then raised, the loop caught the exception and appended a tool result reading tool call failed, please try again, and the model — which had been told the call failed and nothing else — emitted the same call with the same American-format date. That cycle repeated until the step cap fired at twenty. Each of those twenty steps was a full model round trip over a context that had grown to roughly 11,400 input tokens by the end, so a failed run cost about $0.85 and 4.7 minutes of wall clock, and 1,900 of them came to roughly $1,600 for the week with zero useful output. The string expected YYYY-MM-DD was in the response body the whole time and never reached the model once.

Everything that week turned on error recovery, which is the part of an agent that decides what happens after a step goes wrong — a tool returning a rejection, a dependency timing out, a model producing something the system cannot use. Ordinary server code hands that decision to a person, because the lines after the except block were written in advance and say what to do. An agent chooses its own next action from whatever its context contains, so a failure is at once a condition to be handled and information the model needs in order to choose differently, and the reconciliation agent's client handled the condition competently while discarding the information. The machinery is spread across a system rather than gathered into one module: the retry policy around each tool, the message the loop appends when a call raises, the ceiling that ends a run at twenty steps, the alert when a nightly job returns nothing. Those pieces tend to be written at different times by different people, which is how a service-wide backoff setting comes to be applied to a malformed date.

That double role is the whole difficulty and it is worth stating as a sequence rather than as a sentence. A tool call fails. In ordinary code the failure is a condition, and handling it means restoring the program to a state it can continue from. In an agent the failure is also an observation, because the next action is chosen by a model reading a transcript, and whatever the harness writes into that transcript is the entirety of what the model knows about what just happened. Handling the condition well therefore does not imply handling the observation at all, and the two are done by different pieces of code that were usually written by different people. A wrapper can retry correctly, give up correctly, raise correctly, and still leave the model with strictly less information than the network delivered.

Two failures that look alike in a status code and nowhere else#

A 503 from the same tool has a completely different structure underneath it. The request was well-formed, the arguments were correct, the dependency was momentarily unable to serve it, and the passage of time is the only thing that has to change for the identical bytes to succeed. Retrying the same call after a jittered delay is the whole fix, and it is a fix that costs one HTTP round trip and no model tokens at all, because the agent loop never has to know the failure happened. Topic 2.11 developed that policy in detail — the jitter, the ceiling, the attempt cap, the reset the provider reports — and none of it changes when the caller is an agent.

The shape of that policy is worth one line of arithmetic, because the arithmetic explains why the numbers are always chosen the way they are. Each wait doubles the one before it, so the waits form a run of doublings, and a run of doublings sums to one short of the next doubling: one, two, four, eight and sixteen come to thirty-one, which is one less than thirty-two. That is a general fact about doubling rather than a property of these five numbers, and what it says is that the total time a backoff spends sleeping is always about twice its longest single wait. The last interval sets the budget. The attempt count barely moves it, which is why raising a retry ceiling from five attempts to six roughly doubles the worst case rather than adding a fifth to it.

What changes is that an agent has a second kind of failure available to it, one a plain HTTP client does not: a request that is well-formed at the transport layer, accepted by the network, and semantically wrong. The model chose the arguments. A malformed date, a customer ID from the wrong namespace, a filter value not in the tool's enum, a search query that returns zero rows because it was phrased for a different index — all of these arrive as clean HTTP responses carrying a rejection, and every one of them will produce a byte-identical rejection on every subsequent attempt until the arguments change. There is no delay long enough. The only actor in the system that can change the arguments is the model, and the model can only change them if it is told what was wrong.

The reason the March week was silent rather than noisy is visible once you write down what the retry wrapper actually reads. Its decision is a function of one input: it takes the status code, checks it against a set, and either sleeps or gives up. Nothing else about the response is an argument to that function. Not the error code inside the body, not the name of the offending field, not the sentence stating the exact format the argument should have had. Because none of those enters the expression, the wrapper's behaviour cannot vary with them — five attempts and five identical sleeps, whether the body named the fix or was empty. A quantity that cannot vary with the thing you are worried about reads the same on a healthy system and a broken one, and that is the property being exploited here, unintentionally, by the malformed date. It licenses one sentence and no more: a retry policy certifies that time has passed, and never that anything about the request has changed.

Every row of that table is one distinction asked in a different place, and the row that generates the rest is the second. Once you know that elapsed time is the only thing that has to change on the left and the call itself is the only thing that has to change on the right, the remaining rows follow without being learned separately. Elapsed time is something a wrapper can produce on its own and it costs a round trip, so retry cheaply, often, and out of the model's sight. A different call is something only the model can produce and getting one costs a full model round trip over the whole context, so retry rarely, expensively, and only after putting into the context something the model can actually act on. The two columns are not two policies that happen to differ; they are one question about who is able to change the thing that has to change.

The last row is where the money went. Applying a transport policy to a semantic failure produced 31 seconds of guaranteed-useless sleeping per step and then hid the diagnosis behind a generic message. Applying a semantic policy to a transport failure is the mirror mistake and is not cheaper: feeding 503 Service Unavailable back to the model as a tool result invites it to reason about the outage, and a model that has been told the search tool is broken will happily invent a workaround — calling a different tool with worse data, or answering from memory — when the correct response was to wait 800 milliseconds and call the same tool again.

That mirror mistake is tempting for a reason worth naming, because it presents itself as caution: showing the model more of what happened cannot make things worse. It can, and the mechanism is the model's competence rather than any weakness in it. A model's job inside an agent loop is to find a path to the goal. Tell it the search tool is broken and it will do that job, because there is nearly always some path, and the path it finds is a different tool holding worse data or an answer assembled from memory. Neither is what you wanted, and both look like progress in the transcript. Waiting is the correct move and no amount of reasoning arrives at it, because waiting is not a path and a model asked for a path will not return one.

Returning the error text is the mechanism#

The correction is small in code and large in behaviour. The tool wrapper keeps its backoff for the retryable statuses and stops swallowing everything else; the dispatch layer catches the domain error and turns it into a tool result the model can read, with the provider's own message preserved.

RETRYABLE_STATUS = {408, 429, 500, 502, 503, 504, 529}

class ToolFailure(Exception):
    def __init__(self, message: str, retryable: bool):
        super().__init__(message)
        self.message = message
        self.retryable = retryable

def call_tool(name: str, args: dict) -> str:
    """Transport-layer retries only. Semantic failures escape immediately."""
    for attempt in range(5):
        try:
            resp = http.post(f"/tools/{name}", json=args, timeout=20)
        except (ConnectionError, ReadTimeout) as exc:
            if attempt == 4:
                raise ToolFailure(f"{name} unreachable: {exc}", retryable=True)
            sleep(jittered(attempt))
            continue

        if resp.status_code in RETRYABLE_STATUS and attempt < 4:
            sleep(retry_after(resp) or jittered(attempt))
            continue
        if resp.status_code in RETRYABLE_STATUS:
            raise ToolFailure(f"{name} unavailable after 5 attempts "
                              f"({resp.status_code})", retryable=True)
        if resp.status_code >= 400:
            body = resp.json().get("error", {})
            raise ToolFailure(compact(body), retryable=False)
        return resp.text

The retryable flag on ToolFailure is the classification made once, at the only place in the system that can see the status code, and carried forward as data. Everything downstream branches on it rather than re-deriving it, which matters because the status code is gone by the time the loop sees the exception. A design that raises one exception type for everything and asks the caller to parse a message string is the same design that produced tool call failed, please try again, one layer up.

compact is doing more work than its size suggests. Whatever it returns is appended to the message list and is then resent on every remaining step of the run, so a 4 KB stack trace at step three costs its full token count six more times before the run ends, and it crowds the context that the model needs for the actual task. Keeping the code, the field name and the human-readable message — about 90 characters here — carries every bit of information the model can act on. Dropping the internal trace, the SQL, the request ID and anything resembling a credential or a customer's personal data matters for the same reason: the model will quote the error back to a user often enough that you should assume anything you put in the context can be read aloud.

The general shape of that cost is worth having, because it decides how the trade-off runs on any run rather than on this one. A tool result is appended to the message list once and is then part of the prompt of every model call the run still has to make, so what it costs is its size multiplied by the number of calls remaining. Two consequences follow. The same message is expensive when it arrives early and nearly free when it arrives on the last step, so the worst place for a large error is the beginning of a long run. And the decision about what an error message contains is a decision about the rest of the run rather than about one step, made once, in compact, by whoever wrote it — which is why it is worth writing deliberately instead of passing the provider's body through whole.

The loop's side of the change is the part that decides what happens next.

try:
    output = call_tool(call.name, call.input)
    results.append({"type": "tool_result", "tool_use_id": call.id,
                    "content": output})
except ToolFailure as failure:
    if failure.retryable:
        # The dependency is down, not the call. Do not ask the model to be
        # clever about it; the breaker in the next chapter owns this case.
        raise
    semantic_failures[call.name] += 1
    if semantic_failures[call.name] > 2:
        content = (f"{failure.message}\n"
                   f"This tool has now rejected three calls in this run. "
                   f"Do not call it again; use another approach or report "
                   f"what you have.")
    else:
        content = failure.message
    results.append({"type": "tool_result", "tool_use_id": call.id,
                    "content": content, "is_error": True})

The is_error flag is not decoration. Providers surface it as a distinct block type, which means the model reads "this tool ran and rejected me" rather than "this tool returned a string that happens to describe a problem", and the difference shows up in how often the next call is a genuine correction rather than a rephrasing of the same intent. The pattern is written up as Factor 9: Compact Errors into Context Window in the 12-factor agents material, and the compaction half of the name is as load-bearing as the error half.

Four things a changed strategy can change#

Reformatting an argument is the easiest case because the error names the fix. The general move is broader, and it is worth having all of it in mind, because the model will choose among these and your tool descriptions and system prompt are what steer the choice. The arguments can change, which covers formats, enum values, identifiers from the wrong namespace, and ranges that were too wide to return anything. The tool can change: a search_orders that returns zero rows for a phrase-matched query is a signal to try get_order_by_id with the number the user actually gave. The instruction can change, in the sense that the model's own plan is revised — decomposing one call that keeps timing out into three narrower ones over a week each. And the path can change entirely, dropping to a fallback that is worse but works, such as answering from a cached summary and saying so, or handing the task to a human queue with the partial findings attached.

Those four are not a list somebody happened to collect, and seeing why settles what to expect from a tool you have not met yet. A tool call is made of three things: a tool, a set of arguments, and a position in whatever plan the model is working to. Change the arguments and you have the first. Change the tool and you have the second. Change the position — what happens before the call, whether it is one call or three narrower ones — and you have the third. Give up on the plan the call belonged to and you have the fourth. There is no fifth item because there is no fourth part, and any recovery you can describe is one of these or a combination of them.

Only the first of those is deterministic enough to consider doing in code rather than through the model. A wrapper that normalises dates before they reach the tool is strictly better than a wrapper that lets the model discover the format, because it costs no round trip and it cannot fail twice. That is worth saying plainly: a semantic failure the model must recover from is a semantic failure your schema, your enum, or your argument coercion did not prevent. Every recovery path in this topic is the second line of defence, and the traces that fire it most often are usually pointing at a tool description that omits the format or a schema that accepts a string where it should accept an enum.

The step under that claim is short and worth walking, because it decides where you spend your time after reading a trace. The model picks arguments from the tool's description and schema. A schema that accepts a free string accepts every string, including the wrong ones, so the rejection happens at the far end where it costs a round trip. A schema that accepts an enum rejects the wrong ones before the call is made, at no cost at all. So a recurring semantic failure is evidence about the tool's surface and not only about the run it appeared in, and the cheapest fix for a class of failures is almost never in the recovery path that catches them.

Where the changed-strategy retry stops helping#

It is not a general-purpose escape hatch, and it has two failure modes of its own. The first is that each attempt costs a full model round trip over a context that is now larger by the error message, so three semantic attempts on a 9,000-token context cost about as much as nine ordinary steps and take three times as long as the naive path would have. The cap in the code above exists for that reason, and two to three is not a number derived from theory — it is the point past which replayed traces stopped producing genuinely different calls and started producing cosmetic variations on the same wrong idea, which is a loop with extra steps.

The second is that a changed strategy assumes there is a strategy that works. When a tool is wrong about the world rather than wrong about the request — an inventory service returning stale counts, a search index that silently lost a shard — the model receives no error at all, retries nothing, and produces a confident answer built on bad data. Nothing in this chapter touches that case, and the chapter that closes this topic is about how it stays invisible for a month.

Those two limits share a boundary with the one the opening incident sat on. Everything in this chapter is a rule about who can change the thing that has to change: elapsed time for a transport failure, which a wrapper can supply on its own, and the call itself for a semantic failure, which only the model can write and only when the harness has handed it the provider's own words. Getting the classification right is worth more than getting either policy right, because a correct policy applied to the wrong class is what turned ninety characters of perfectly good diagnosis into a fortnight of silent, expensive sleeping.

There is also a case sitting between the two columns of that table. A tool that returns 503 five times in a row is not having a transient; it is down, and the transport policy has already spent its attempts finding that out. Retrying with a changed strategy does not help either, because the strategy that changed still points at a dependency that is not answering. Both policies so far assume the tool eventually responds, and when that assumption fails the agent needs a way to stop asking.