Exhausted budgets and client deadlines

Deadline handling and generation limits live in different places. An HTTP client, or a framework's request timeout, enforces the deadline and raises on expiry; the provider enforces the token budget and reports it in the response. When the client raises first there is no response to read, so stop_reason is never observed and the provider's account of what happened is discarded along with the connection. Everything downstream sees is one exception type.

That last step is worth slowing down, because "discarded along with the connection" sounds like withholding and is nothing of the kind. A streamed response arrives progressively over one connection, and the fields describing how generation ended — the stop reason, the usage totals — sit in the final message, after all of the content. The account of what happened is therefore the last thing to arrive, by construction rather than by policy. Cancelling on a deadline closes the connection while that message is still queued behind data you have stopped waiting for. Nothing refused you the information. It was in transit, and you hung up.

This chapter separates the cases that exception type conflates. There are three worth distinguishing, they occur in roughly comparable numbers on a reasoning workload with a tight deadline, and only one of them is helped by more time. The chapter covers the instrumentation that tells them apart, the arithmetic that decides whether a thinking budget is reachable inside a deadline at all, and the result type a fallback needs so that a truncated answer cannot be mistaken for a finished one.

Three failures behind one log line#

The first is a request where generation was healthy and the clock simply ran out. Thinking finished at 19 seconds, the answer was streaming steadily at around 70 tokens per second, and the client cancelled at 30 with two-thirds of a plan received. Given 40 seconds these would have returned a good answer.

The second never had an answer at all. The thinking allowance was consumed entirely, the provider stopped generation at the limit, and what came back was either empty or a single truncated sentence. Some of these return in 22 seconds, comfortably inside the deadline, and are counted as successes by a latency dashboard and as garbage by whatever consumed them. More time changes nothing here, because time was not the binding constraint; the token budget was.

The third appears in no failure metric. An answer was produced, the request returned HTTP 200 inside the deadline, and the budget ran out partway through the model's own checking pass, so what you received is a first draft with its verification cut short. These are not counted anywhere and they are the most expensive, because a subtly wrong plan that executes is worse than a plan that never arrives.

The belief that makes one label look like one problem is that a timeout is a statement about slowness. It is not. A timeout is a statement about a clock your own code started, and it fires whether the thing being waited on was slow, was stuck, had already finished, or was never going to produce anything better than a short, unhelpful answer. Only the first of the three cases above is about time at all. The other two are about a token allowance, and a token allowance is enforced somewhere else, in a different unit, by a system with no view of your clock. The remedy the label suggests — wait longer — is aimed at one case in three, and it charges the other two for the attempt.

Telling D from G matters because they have different fixes. In D the model spent the whole deadline thinking and produced nothing, so extending the deadline is a bet that it would have finished, and you have no evidence either way — the request was killed before the provider could say whether the thinking pass had converged. In G you have direct evidence, since tokens were arriving at a measurable rate when you cancelled, and you can compute how much longer the request needed from the answer length distribution.

The instrumentation that separates them reads the stream rather than the final object. Every provider supporting extended thinking streams the thinking phase as distinct events, which gives a timestamp for the moment thinking ended, and that single number resolves most of the tree above.

async def traced_call(prompt, budget, deadline_s):
    t0 = time.perf_counter()
    thinking_done_ms = None
    first_text_ms = None
    text_chars = 0

    try:
        async with asyncio.timeout(deadline_s):
            async with client.messages.stream(
                model="claude-sonnet-4-5",
                max_tokens=budget + 1500,
                thinking={"type": "enabled", "budget_tokens": budget},
                messages=[{"role": "user", "content": prompt}],
            ) as stream:
                async for ev in stream:
                    now = (time.perf_counter() - t0) * 1000
                    if ev.type == "content_block_stop" and ev.block_type == "thinking":
                        thinking_done_ms = now
                    elif ev.type == "text":
                        first_text_ms = first_text_ms or now
                        text_chars += len(ev.text)
                final = await stream.get_final_message()
                return Outcome("complete", final.stop_reason, thinking_done_ms,
                               first_text_ms, text_chars, final.usage)
    except TimeoutError:
        phase = "deadline_in_thinking" if thinking_done_ms is None else "deadline_in_answer"
        return Outcome(phase, None, thinking_done_ms, first_text_ms, text_chars, None)

Four fields do the work. thinking_done_ms being None at cancellation means the model never reached the answer. first_text_ms being set with a non-zero text_chars at cancellation means generation was healthy and the deadline was the constraint. A completed response with stop_reason == "max_tokens" and text_chars near zero is the budget-exhausted case. A completed response where thinking_done_ms sits within a few percent of the budget's expected duration is the silent third case, where the model used everything it was given and stopped checking because it had to rather than because it was satisfied.

Two timestamps carry that much because each answers one yes-or-no question, and two such questions separate four states. Whether the thinking phase ended splits the requests that never reached an answer from the ones that did. Whether any visible text arrived splits an answer that started from an answer that did not. The tree above is those two questions asked in an order chosen so that the provider's own stop_reason — the field you cannot always obtain, for the reason given at the top of this chapter — is consulted only where it is genuinely needed.

Why the third case is silent rather than merely rare follows from what a response is able to express. The provider reports how generation ended, and generation did end normally: the model was moved off the thinking pass when its allowance ran out, wrote an answer, and finished the answer. Every field in that response is telling the truth. What is missing is any field for "the answer is here and the work behind it was cut short", because from the provider's side nothing went wrong — a budget was set and the budget was spent. That is why the only evidence available is timing, why the inference drawn from it is a prior rather than a reading, and why the result type at the end of this chapter needs a field the API does not supply.

There is a limit to how far this reaches, and it is worth stating rather than pretending the tree is complete. Where a provider returns a summary of the reasoning rather than the reasoning itself, the number of summary tokens you receive bears no fixed relation to the tokens generated, so progress cannot be inferred from text arriving during the thinking phase — only from the usage figures at the end and the timestamps of the phase boundaries. Distinguishing the third case therefore relies on inference from timing rather than on observation, and the inference is imperfect: a model that finishes exactly at its budget may have finished, or may have been cut off. Treat the classification as a strong prior for triage and confirm it by rerunning the same input with a larger budget to see whether the answer changes.

Squeezing the budget on purpose#

You cannot classify failures you have never seen, and a 3% failure rate is too thin a sample to develop the instinct on. Run a frozen task set against a deliberately hostile configuration — a thinking budget of 1,024 tokens, the provider minimum, against items you know need several thousand — and watch what the responses look like when the allowance runs out. Then do the same with a generous budget and a five-second deadline. The two runs produce visibly different artefacts, and after reading twenty of each you will recognise them in production logs without the tree.

The hostile run also gives you the number the real configuration needs, which is the decode rate. Measure tokens per second on the thinking phase specifically, since it is generated at roughly the same rate as visible output — for a mid-sized model on a typical hosted endpoint, somewhere between 60 and 90 tokens per second, and worth measuring rather than assuming because it varies with load and with the deployment. At 75 tokens per second, a 30-second client deadline with 3 seconds reserved for network, queueing and retries leaves 27 seconds of generation, which is about 2,025 tokens in total. If the answer needs 500, the thinking budget cannot exceed roughly 1,500 without making the deadline unreachable by construction.

That arithmetic is worth doing explicitly, because a great many production configurations set a thinking budget of 10,000 tokens behind a 30-second deadline. At 75 tokens per second, 10,000 thinking tokens is 133 seconds, so the budget was never reachable and its only real effect was to guarantee that any request needing serious deliberation would be killed mid-thought. The configuration reads as generous and behaves as a trap: it permits the model to start work it cannot finish, and every such request costs full price for the tokens generated before the client hung up.

A decode rate is the number worth measuring because it is the exchange rate between the two units this chapter is about. A deadline is set in seconds, usually by whoever owns the calling service. A thinking budget is set in tokens, usually by whoever owns the prompt. Nothing in either system converts one into the other, so the two get chosen by different people, in different files, without either being checked against the other even once. Tokens per second is that conversion. Once it has been measured, the two settings stop being independent knobs and become one constraint with two ways of writing it down.

Retries deserve a line in the same arithmetic, because they are usually configured independently and then compete for the same 30 seconds. A client with three retries and a 30-second per-attempt timeout has a 90-second worst case the calling service does not know about, and a client with three retries inside a shared 30-second budget has an effective per-attempt allowance of ten seconds, which no thinking budget above about 500 tokens fits into. Decide which of those you have and write it down next to the budget, since a retry policy inherited from an HTTP library was designed for requests that fail in milliseconds and behaves very differently in front of a call that fails after twenty seconds of billed generation.

The five hundred tokens in that sentence come out of the same division, and it is worth running rather than reading. Ten seconds of allowance, less the reserve for network and queueing, at a measured decode rate, is a few hundred tokens of generation in total — and the answer has to be paid for out of the same few hundred. What is left over for thinking is whatever the answer does not need, which on a planning task is very little. A retry policy therefore does not simply multiply the worst case. It divides the budget, and the division happens silently inside a library that was never told a budget existed.

What a fallback is allowed to return#

Every routing and budget decision eventually needs an answer to what downstream code receives when the model does not produce something usable, and the wrong answer is to pass along whatever text arrived. A truncated plan is syntactically a plan. A fallback returning the partial answer with no marking converts a visible failure into a silent one, and the consumer executes half a plan.

The danger in that sentence is that syntax is exactly what downstream code checks. A consumer validates against a schema, finds a list of steps where a list of steps was promised, and proceeds. Nothing in a well-formed structure records how much of the work behind it actually happened, because completeness of form and completeness of work are independent properties and only the first one is expressible in a schema. So the second has to be carried deliberately, in a field whose only job is to carry it, or it is not carried at all.

The contract that works returns the same schema as success, with an explicit field saying it is degraded and why.

@dataclass
class PlanResult:
    plan: list[Step] | None
    status: Literal["ok", "degraded", "unavailable"]
    reason: str | None          # "budget_exhausted" | "deadline" | "verifier_rejected"
    verified: bool              # false when the checking pass did not complete
    source: str                 # which model and configuration produced this

verified is the field that earns its place, because the third failure group is precisely the case where plan is populated, status is ok by any naive test, and the model's own checking never finished. Downstream code can then make its own decision: an automated executor refuses anything with verified == False, while an interface shows it to a human with a warning. Both behaviours are correct for their context, and neither is available if the result type has no way to express the distinction.

What the fallback contains depends on the workload, and there are three defensible options. It can be the output of a cheaper, faster model run without thinking, which gives you something rather than nothing at a known lower quality. It can be a deterministic default — the conservative route, the unmodified record, the human queue — which is right when a wrong answer is worse than no answer. Or it can be an explicit unavailable the caller is required to handle, which is the honest choice when there is no acceptable degraded output and the alternative is inventing one. Choosing among them is a product decision that needs making before the fallback is written rather than discovered from what the code happens to do.

One exception type is not one failure, and the cost of treating it as one is that the most common fix — more time — helps at most a third of the cases and charges the rest of the traffic for the privilege. The three causes are separable with two timestamps and a stop_reason, the budget that makes a deadline reachable is a division you can do on paper from a measured decode rate, and the answer that arrives with its verification truncated needs a field in the result type or it will be indistinguishable from a good one.

Running a cheaper model as the fallback raises an obvious question about the ordering. If a fast model can produce an acceptable answer when the expensive one fails, and something can tell you whether that answer is acceptable, it is not clear why the expensive model should go first.