Proxy timeouts on long calls

Software that fails because of a bug fails at times drawn from a distribution. A memory limit is hit sooner on a bigger input, a deadlock resolves or does not, a downstream call times out somewhere near its own limit with jitter around it. Failures clustered inside 100 ms of a round number in the client's wall clock come from a configured limit instead, and 60 seconds is the most heavily defaulted number in the HTTP stack.

That is a whole class of failure that has no counterpart over stdio, because a pipe between a parent and its child has no third party with an opinion about duration. Everything that makes HTTP deployable — proxies, load balancers, TLS termination, centralised logging — is another participant in the conversation, and each participant is one more place where an exchange can be ended by something other than the two parties to it. A streamable HTTP response is unusually exposed to this, because it is allowed to stay open and quiet for as long as a handler takes.

Long calls are what expose it, and it is worth being precise about what a pipe lacks that an HTTP path has, because the three absences map one-to-one onto the three symptoms in this chapter. A pipe has no intermediary with an opinion about duration, so no exchange over it is ever ended by a party that was not in it. It has no buffering layer that can be misconfigured independently of either side, so bytes written by the server are bytes the client sees. And it has no access log in which a third party records its own version of events, so there is never a second account of the call to reconcile against the server's. Add the three back and you get, in order: a call killed at a round number, a stream that arrives all at once at the end, and two logs that disagree about whether anything succeeded.

Idle is measured from the last byte, not from the start#

The relevant setting in nearly every proxy is an idle timeout: the maximum time the connection may go without transferring data in either direction. A normal HTTP response satisfies that trivially, because the body arrives in a burst. A streamable HTTP response for a tools/call that does 90 seconds of work before it has anything to say sends its headers, then nothing, and the connection sits silent from the proxy's point of view for the entire duration of the call.

Why the setting measures silence rather than work is the step that makes everything after it predictable. A proxy holds a socket, a buffer and a slot in a connection table for every exchange in flight, and it has to be able to reclaim them, because an upstream that crashed or a client that walked away would otherwise hold them forever. It has no way to ask whether the upstream is still working. It is a program in the middle: it did not issue the request, it cannot see the handler, and it is not told when work begins or ends. The only evidence available to it is whether bytes have moved. So "has this exchange stalled" gets implemented as "has this connection been quiet for longer than the limit" — a substitution that is exactly right for the traffic the defaults were chosen for, where a response body arrives in a burst, and exactly wrong for a response that is entitled to think for a minute and a half before it says anything.

The defaults are worth having in front of you, because the number the client observes tells you which hop you are fighting.

Two rows there are worse than an idle timeout, since they cap the total duration of the response regardless of how much traffic it carries. Envoy's route timeout and Google's backend timeout will kill a stream that has been chattering happily the whole time, which is why the standard advice to send heartbeats fixes some deployments and not others. A deployment that fails at 15 s with an Envoy sidecar in the path is not going to be rescued by a keepalive; that hop needs its timeout raised or disabled for the MCP route.

The reason a heartbeat cannot rescue those two rows is definitional rather than empirical, and seeing it saves an afternoon spent adding more heartbeats. A heartbeat works by resetting an idle timer, and an idle timer is the only thing it can reset, because the arrival of a byte is evidence of precisely one fact: this connection is not quiet. A total-duration cap is not measuring quiet. It starts when the request starts and it expires on the wall clock however much traffic crossed in the meantime, so a byte carries no information it is willing to act on. Reading the last column of that table is therefore the first move rather than the last: if the hop that is failing counts total time, the only options are to raise it, to route around it, or to stop holding responses open at all.

There is a companion failure at the same layer that presents differently. If the proxy buffers the response instead of streaming it, the client receives nothing at all until the handler finishes, at which point everything arrives at once — progress notifications that are now historical, followed by the result. The session appears to work and every long call appears to hang, and no amount of timeout tuning changes it. Buffering is on by default in nginx, and the response needs both a configuration change and, for belt and braces, a header that tells the proxy to leave this response alone.

location /mcp {
    proxy_pass         http://repo_index_upstream;
    proxy_http_version 1.1;

    proxy_buffering    off;      # stream through instead of accumulating
    proxy_cache        off;
    chunked_transfer_encoding on;

    proxy_read_timeout    3600s; # silence tolerated from upstream
    proxy_send_timeout    3600s;
    keepalive_timeout      75s;
}

return StreamingResponse(
    sse_events(),
    media_type="text/event-stream",
    headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)

The same header is understood by several proxies derived from or configured to mimic nginx, and it costs nothing on the ones that ignore it. It is the same class of fault as the block-buffered stdout from earlier in this topic — bytes written by the application and held by a layer beneath it — relocated from a libc buffer to a reverse proxy, and it responds to the same diagnostic instinct of asking where the bytes stopped rather than why the work is slow.

Keeping the stream from going quiet#

The fix that is available without touching infrastructure is to make the response stream produce traffic while the handler works. There are two forms, and the difference between them is whether the traffic is protocol or not.

An SSE comment is a line beginning with a colon, which the event-source framing defines as a no-op. It travels the connection, resets every idle timer along the path, and never reaches the client's JSON-RPC parser.

async def sse_events(work):
    task = asyncio.create_task(work)
    while not task.done():
        done, _ = await asyncio.wait({task}, timeout=15.0)
        if not done:
            yield ": keepalive\n\n"          # comment frame: not a message
    for event_id, message in emitted_messages():
        yield f"id: {event_id}\ndata: {json.dumps(message)}\n\n"

A progress notification is a real protocol message, sent when the client supplied a progressToken with the call, and it does double duty: it resets the same timers and it tells the user's interface that the index is 57 files into 84. The choice between them is not exclusive, and a reasonable server sends progress when it has genuine progress to report and comments when it does not.

The interval matters more than which form you pick. A 15-second heartbeat against a 60-second idle timeout leaves three chances to miss one before the connection dies, and it costs 6 bytes every 15 seconds per open call. Setting it to 55 seconds to be economical means a single delayed event loop iteration kills the call, and the resulting failure is intermittent and clusters near 60 seconds all over again, which is a worse bug to own than the original.

Those three chances are worth counting out, because "send a heartbeat" shipped without an interval attached is a half-done fix. Sixty divided by fifteen is four, so four heartbeat slots fit inside one idle window. Every beat that does arrive resets the window, so what kills the connection is never a single missed beat but a run of consecutive misses long enough for the clock to reach the limit. Miss one and the gap is thirty seconds; miss two and it is forty-five; miss three and it is sixty, which is the boundary. Three consecutive misses is the margin the interval buys, and the margin is the whole point of choosing an interval well under the limit rather than just under it. Run the same arithmetic at fifty-five seconds and there is no margin left: the first slot already sits five seconds short of the limit, so one late event loop iteration is fatal on its own.

Four fixes, and what each of them actually buys#

Once the mechanism is clear the options are easy to state, and it is worth being explicit about what each one does not cover, because teams commonly ship the cheapest one and rediscover the problem from a different hop.

Raising timeouts is the honest fix when you own every hop, and the failure mode is that you rarely do. A corporate proxy on the client side, a CDN in front of the ingress, or a platform like API Gateway with a hard 30-second ceiling will each be discovered one at a time, usually by one customer whose network path differs from everyone else's. Resumption, which the previous chapter described as a property of event ids on the stream, converts a fatal drop into a reconnect that replays the tail — and it only works if the reconnect reaches a process that still remembers the session and the buffered events.

The fourth option changes the tool's contract rather than its transport. An index_repo tool becomes a call that returns {"job_id": "idx_9f21", "status": "running"} in 40 ms, plus a second tool index_status that the model calls until the job is done. Every timeout in the table becomes irrelevant, because no request is ever open for more than a second. The costs are real: the model now has to be told to poll, which consumes turns and tokens; the job state has to live somewhere both replicas can see; and a user watching the interface gets a worse experience than a progress bar. For work measured in minutes rather than seconds — a full reindex, a large migration, a batch export — it is usually the right answer anyway, because holding an HTTP response open for four minutes is a bad plan regardless of whether you can make it work.

Round numbers that are yours#

Not every clean 60 comes from a proxy, and assuming it does will send you to configure infrastructure that was innocent. Postgres statement_timeout, a requests call with timeout=60, an HTTP client default in an SDK, and a Kubernetes liveness probe that restarts the pod after a failure threshold all produce their own round numbers. The evidence that separates them is whether the handler completed. If the server logged index_repo done at 93.5 s and then failed to write, the work finished and the channel was gone; if the server logged an exception at 60.0 s from inside the handler, the limit is in your own stack and no ingress change will help.

Three instruments are available in that evidence and they agree, which is what makes the reading decisive rather than suggestive. Subtracting the scenario's start timestamp from its done timestamp gives a minute and a half of wall clock, crossing the minute boundary, and that difference is the same 93.5 s the handler's own elapsed counter reported — one figure from the logging framework's clock and one from a timer inside the handler, neither derived from the other. The client's three timings are the third instrument, and they cluster inside a tenth of a second of sixty. Two clocks on the server say the work took a minute and a half; a clock at the other end says the exchange ended at sixty seconds, so the work outlasted the exchange by about a third of its own duration. A handler that crashed, hung or ran out of memory cannot produce that pattern, because in each of those cases the work stops at or before the exchange does.

The instrumentation that makes this a five-minute question is not elaborate. Log the start and end of every tool call with the session id and the elapsed time, log write failures on the response stream separately from handler exceptions, and read the proxy's own access log for the same request, where the upstream response time and the connection close reason are recorded.

# ingress access log, same request
09:42:02 "POST /mcp HTTP/1.1" 200 upstream_time=60.001 bytes_sent=214 \
         upstream_status=- reason=upstream_timed_out session=7f3a1c02

upstream_status=- with a 200 already sent to the client is the fingerprint: the proxy committed to a successful response when the headers arrived, then gave up on the body. The client sees a truncated stream and reports a network error; the server sees a successful call. That asymmetry is the reason these reports are confusing, since the two sides genuinely disagree about what happened and both are describing their own evidence accurately.

One figure in that access log line repays a close read. upstream_time=60.001 is the proxy's own account of how long it waited on the upstream before giving up, and it sits a thousandth of a second past the configured limit rather than anywhere near the ninety-three and a half seconds of work — which is what a timer expiring looks like, as against work finishing. The client, one hop further out, recorded a few hundredths more than the proxy did, and those hundredths are the leg between proxy and client plus the client's own bookkeeping. Three numbers from three vantage points: the handler timed the work, the proxy timed its own patience, and the client timed the round trip. Only the first is a statement about the server, and it is the one nobody had been reading.

The habit worth building from all of this is to read a duration before reading a stack trace. A failure at 60.04, 60.02 and 60.11 seconds is a configured limit somewhere in the path, and the only question left is whose — which the server's own log answers in one line by saying whether the handler ever finished. Everything after that is a choice between the four fixes, made on the basis of which hops you control and how long the work honestly takes.

Heartbeats and generous timeouts get a single-replica deployment working. The reconnect path does not survive the next step, because the moment there are three pods behind the load balancer, a client resuming a stream has to land on the process that issued its session, and nothing so far guarantees that it will.