Timeouts for streaming and batch calls

A timeout is the client's decision to stop waiting. Written as a single duration attached to a call — timeout=60 — it answers exactly one question: has the whole thing taken too long? That question is adequate for a request-response call, where the only two outcomes are a complete response and no response. A streamed generation, where the provider sends the answer as a sequence of server-sent events rather than one body at the end, has a third outcome, which is a partial response followed by silence. A total budget is blind to it by construction: from the outside, five seconds of output followed by fifty-five seconds of nothing and sixty seconds of steady output look identical until the sixtieth second.

The distinction the code needs is between a call being slow and a call being dead, and that is a statement about the gaps between events rather than about their sum. What a bulk path needs is different in kind rather than in magnitude: nobody is watching an individual item, so a gap between deltas is not a symptom of anything, and the deadline that matters belongs to the job rather than to the call. The same number on both paths is wrong twice, for unrelated reasons.

Phased budgets, and the stall a total budget hides#

HTTP client libraries already know that one number is not enough, which is why their timeout configuration is not one number. In httpx, Timeout(connect=..., read=..., write=..., pool=...) splits the budget by phase, and the read value is the maximum time to wait for the next chunk of data rather than for the whole body. The same is true of the read timeout in requests. A stalled stream trips a five-second read timeout in five seconds, and a stalled stream under asyncio.timeout(60) trips nothing for a minute; the difference between those two behaviours is entirely in which knob the number was written on.

What separates the two is where the clock is started and what restarts it. A total budget starts one clock when the call begins and compares it against one number, so the only quantity it can ever report is elapsed time since the start. A read timeout starts a clock when the client asks the socket for more data and cancels it the moment any data arrives, then starts a fresh one for the next read. The first arrangement measures a sum. The second measures the largest gap. A stalled stream and a healthy long generation have very similar sums and completely different largest gaps, which is why one of these configurations can tell them apart and the other cannot, whatever number is written on it.

import httpx

# Total budget: blind to stalls. One number, one question.
httpx.Timeout(60.0)

# Phased budget: 5 s to connect, 10 s of silence tolerated between
# chunks, no total cap — a healthy long generation runs as long as
# it keeps producing.
httpx.Timeout(connect=5.0, read=10.0, write=10.0, pool=2.0)

The second configuration fails a stalled request in a few seconds rather than at the total budget, which frees the worker and lets a fallback path run while the user is still watching. It is a better default for any streaming call, and on its own it is still not enough, for a reason that only shows up on providers that keep their connections warm.

Keep-alive pings defeat a byte-level deadline#

Server-sent event streams from these APIs carry more than content. Alongside the content deltas there are lifecycle events and periodic ping events whose entire purpose is to keep intermediaries from closing an idle connection. Those pings are bytes on the socket, and a read timeout measured in bytes is satisfied by them. A generation that has stopped producing text while the connection is still being pinged every few seconds keeps a 10-second read timeout permanently happy, and the request hangs exactly as it did under the total budget.

The reason this cannot be fixed by choosing a better read timeout is worth following down one layer, and exactly one layer. A server-sent event stream is a byte stream with a framing convention on top of it: events are delimited by blank lines and typed by a field inside them. The transport layer beneath the client is delivering bytes, and it has no interest in what those bytes say — that is what makes it reusable for every protocol. So the timer the read timeout resets is reset by the arrival of any bytes. A ping event is bytes. A content delta is bytes. The two are the same event as far as the socket is concerned, and no value of read can distinguish them, because the distinction does not exist at the level where the number is applied.

That places the deadline that can tell them apart in the only layer that parses the events, which is the loop consuming the stream. The general shape recurs well beyond this API: a deadline can only discriminate between conditions that are visible at the layer it is installed in. Moving a number up or down does not add information to a layer that never had it.

The deadline that catches this has to be measured on content rather than on traffic, which means it lives in the event loop rather than in the transport configuration. Two clocks are worth running there, and they measure different failures. The first is the time to the first content delta, the quantity most dashboards label time to first token (TTFT), which covers the case where the provider accepts the connection, sends message_start, and then never produces anything — a failure mode with its own distinct signature, because the request has been admitted and is consuming a rate-limit slot while producing nothing. The second is the gap between successive content deltas, which covers a generation that dies part-way through.

import asyncio, time

async def consume(stream, ttft_s=5.0, gap_s=10.0):
    deadline = time.monotonic() + ttft_s
    first_seen = False
    async for event in stream:
        now = time.monotonic()
        if event.type == "ping" or not is_content(event):
            continue                      # pings do not reset the clock
        if now > deadline:
            raise StreamStalled("no content before deadline",
                                first_seen=first_seen)
        if not first_seen:
            first_seen = True
            METRIC_TTFT.observe(now - (deadline - ttft_s))
        yield event
        deadline = now + gap_s            # reset only on real content

The check that pings do not reset the clock is the whole point of the function, and it is the line most implementations of this pattern are missing. The deadline is also evaluated when an event arrives, so a stream that goes completely silent — no pings either — is still caught by the transport-level read timeout underneath, and the two mechanisms cover each other.

Numbers for these two deadlines come from a latency histogram rather than from a guess. On one support-chat path, measured time to first content delta was 640 ms at p50 and 2.1 s at p99, so a 5-second TTFT deadline sits about two and a half times above p99 and will essentially never fire on a healthy call. Inter-delta gaps on the same path were 40 ms at p50 and 180 ms at p99 during ordinary generation, which makes a 10-second gap deadline extremely generous — deliberately, because extended thinking, tool-use blocks and provider-side batching all produce legitimate pauses an order of magnitude longer than a typical inter-token gap. Setting the gap deadline near the p99 of ordinary token spacing would kill working requests every time the model paused to think.

Both numbers come from treating a deadline as what it actually is, which is a classifier. It sorts calls into "still working" and "dead" using one observation, and like any classifier it has two ways to be wrong. Set it too low and healthy calls are killed; set it too high and dead calls are detected late. A percentile from the measured histogram tells you the cost of the first kind directly: a deadline placed exactly at p99 kills, by definition, one healthy call in a hundred, because that is what the p99 of a distribution means. Five seconds against a p99 of 2.1 seconds is about two and a half times that, which pushes the false-positive rate far down the tail while still detecting a genuine stall within a few seconds. The cost of getting it wrong is asymmetric too, and in the direction that argues for generosity: a killed healthy call is a user who sees a failure for no reason, and a stall detected two seconds late is a user who waits two more seconds.

The gap deadline is set from a different reading of the same principle, because the gaps are not one population. Ordinary token spacing produces one cluster of values, tens of milliseconds wide. Structural pauses — extended thinking, a tool-use block, provider-side batching — produce another cluster, seconds wide, and both are perfectly healthy. A threshold derived from the p99 of the first cluster classifies the whole of the second as death, which is why 180 milliseconds is the wrong basis for the number even though it is a correctly measured p99. Ten seconds is chosen to sit above everything the healthy second cluster does, and it is worth being honest that it is not a tight bound: a stall on this path is caught in ten seconds because no shorter number can be told apart from thinking.

Two paths, six numbers#

A bulk path has none of these problems and a different set of its own. Nobody is watching an individual item, so time to first token is meaningless; what matters is whether the whole job finishes before the window in which its results are useful.

The job-level deadline is the row with no counterpart on the left, and the one most batch systems omit. The capacity arithmetic shows where the number comes from: 50,000 documents at a mean of 8 seconds each, across 40 concurrent workers, is 10,000 seconds of wall clock, or about 2.8 hours. Each step there is worth seeing separately. Fifty thousand documents at eight seconds each is four hundred thousand seconds of work, which is the total the job has to get through and has nothing to do with how it is arranged. Forty workers running at once divide that by forty, giving 10,000 seconds, because concurrency does not reduce the work — it only reduces the wall clock the work is spread over. Ten thousand seconds, divided by the number of seconds in an hour, is about 2.8 hours. If the results are needed by 07:00 and the job starts at 01:00, there are 6 hours available and roughly 3.2 hours of slack. A deadline set at 5.5 hours means the job gives up with half an hour to spare, emits the count of unprocessed items, and leaves a cursor a human or a follow-up run can resume from. Without it, a job that hits sustained trouble keeps retrying items until someone notices at 09:00 that the report is missing, having spent eight hours and a great deal of budget discovering that.

The per-item retry allowance on the right is far larger than the interactive one, and this is where the difference in kind shows up as a difference in number. Ten minutes of waiting spread over five attempts is nothing in a job with three hours of slack, and unthinkable in a request a person is waiting on. That asymmetry only makes sense because the job-level deadline exists to bound the total: without it, 50,000 items each allowed 10 minutes of retrying is a theoretical worst case of 347 worker-days.

That figure is worth computing rather than taking on trust, because it is the argument for the row above it. Fifty thousand items, each permitted ten minutes of waiting, is five hundred thousand minutes. Divided by the minutes in an hour that is a little over eight thousand hours, and divided by the hours in a day it is the 347 days. No fleet size changes it: forty workers get through it forty times faster in wall clock, but the worker-days are worker-days. The per-item allowance is generous precisely because something else is bounding the total, and if that something else is removed the per-item number is revealed as a promise the job cannot keep.

Swapping the two policies#

Take the batch settings and apply them to the chat path. The result is a 180-second per-call ceiling, five attempts, and no first-content deadline at all, which means a stalled generation hangs for three minutes and then does it four more times. Every one of those numbers is individually defensible in the job it came from. The failure is that the chat path's real requirement is not expressible as any value of the settings the batch path uses, because the batch path has no concept of noticing that nothing is arriving.

Take the chat settings and apply them to the batch. Now every item gets one attempt with about two seconds of retry budget and a 5-second first-content deadline. On a healthy night this mostly works, and during any rate-limit episode the job fails thousands of items that a thirty-second wait would have completed, exhausts its dead-letter queue — the side channel a queue keeps for messages it has stopped trying to deliver — and produces a report with gaps. The chat settings are correct for a caller who will leave, and a batch item has nowhere to go.

The general shape is that an interactive deadline exists to protect a person's attention while a batch deadline exists to protect a delivery commitment, and those two purposes generate different instrument panels. Tuning one path's numbers to the other path's magnitudes leaves the instrument missing.

Two things these deadlines explicitly do not do are worth stating. A client-side deadline does not stop the provider's work: abandoning a stream leaves the generation running and billed, so a timeout is a decision about your own resources rather than a way to save money. The step under that is worth having, because the belief that a timeout saves spend is both common and expensive. The generation is running on the provider's hardware under the provider's scheduler, and the deadline that expired is a comparison between two numbers on your machine. Nothing about that comparison crosses the network. The provider finds out something changed when it next tries to write to a socket that has gone away, if it tries, and the tokens produced up to that point were produced whatever happens next. Deadlines buy back workers, connections and a user's attention. They do not buy back tokens. And a deadline is not a retry policy. Deciding to give up after 15 seconds says nothing about whether to try again, how long to wait, or whether trying again is even safe, which is the subject of the last chapter of this topic.

A deadline, then, is not one number but a small set of them, chosen to match the failure each one can see. On a streamed call that means a phased transport configuration underneath a content-level clock, with a first-content deadline sized from a measured p99 and an inter-delta deadline set generously enough to survive a model that pauses to think. On a bulk path it means a per-item ceiling that tolerates slowness, sitting under a job-level deadline derived from the window the results are needed in and the slack that window leaves.

Every deadline in that table is scoped to one request or one job, and each assumes that the thing it protects against is momentary. When a provider degrades for twenty minutes rather than twenty seconds, a per-request policy applied fifty thousand times becomes its own problem, and both paths need something that operates above the level of a single call.