Propagating a client abort upstream

Everything in this topic so far has been about bytes going one way. This chapter is about the one signal that travels the other way, from a user pressing Stop or closing a tab, through your relay, to the provider that is still generating. The signal has a name in each runtime — an AbortSignal in the Web platform and in Node, a closed context manager in Python — and in every case the mechanism at the far end is the same: there is no cancel endpoint for a streaming completion at any major provider, so the way to stop one is to destroy the HTTP connection carrying it. The provider observes the transport close and stops the decoder.

What makes this worth a chapter rather than a line is that every hop except the last one works by default. The browser's fetch promise rejects, the socket closes, the route handler's request emits a close event, the handler returns. Each of those is real and none of them touches the upstream call, so a team can ship a Stop button, watch every part of it behave, and change nothing about what they are charged.

The reason they do not touch it is the one fact this chapter rests on, and it is worth stating before any code. A relay holds two connections. One runs from the browser to your server and one runs from your server to the provider, and they are separate TCP connections, opened at different times, to different addresses, with independent lifetimes. Nothing in either connection's protocol carries the information that the other exists. The provider was handed a prompt over a socket and has no idea whose question it was; the browser opened a request to your host and has no idea which upstream it caused. The only component anywhere in the path that knows the two are related is the relay, which is why the only component that can carry a signal between them is the relay. A signal nobody sends does not travel, however obviously it ought to.

That is also why the fix reads as small out of proportion to its effect. Adding signal: req.signal is not switching cancellation on. It is the first time anything in the system has been told that these two connections belong to the same piece of work.

Following the abort#

In a Node process the upstream response object is still referenced by the in-flight fetch, the body is still being written into a stream nobody is reading, and the socket stays open until the provider finishes the message or something times out. Garbage collection is not a cancellation mechanism and will not run promptly enough to be one even when the last reference does drop. From the provider's side, absolutely nothing has changed: the request is in a decode batch, it decodes to its stop_reason or to max_tokens, and the usage record is written for the full completion.

Two beliefs die in that paragraph and both are worth naming. The first is that returning from a handler ends the work the handler started, which holds for anything running inside the function and fails for anything that outlives it — and an in-flight fetch is exactly that, a live object owned by the runtime rather than by your stack frame. The second is that an unreferenced connection gets tidied up. A collector runs when memory pressure says so, not when a socket becomes pointless, and the collector's job when it does run is to reclaim memory, which is a different job from sending a reset to a peer. Neither belief is careless; both are true of ordinary synchronous code, and the streaming case is the one where the work and the frame that started it come apart.

There are two places to attach the abort. The relay built earlier in this topic already has one of them: the cancel callback on the client-facing ReadableStream, which fires when the runtime tears that stream down and calls reader.cancel() on the upstream body. That is the first hop, and it depends on the runtime actually invoking cancel when the socket closes, which is a thing to verify rather than assume. The second attachment point is the request itself, and it is the one that survives a hand-rolled handler. On the Web-standard Request used by Next.js route handlers, Deno, and Cloudflare Workers, the client disconnect is exposed as req.signal, and forwarding it is one property:

export async function POST(req: Request) {
  const upstream = await fetch("https://api.openai.com/v1/chat/completions", {
    method: "POST",
    headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}` },
    body: JSON.stringify({ model: "gpt-4.1", stream: true, messages }),
    signal: req.signal,          // the hop that was missing
  });
  // ...relay as before: buffer, split on the blank line, re-emit
}

On the older Node HTTP API there is no signal, so you make one and wire the disconnect to it:

const controller = new AbortController();
req.on("close", () => controller.abort());   // fires on client disconnect

const upstream = await fetch(providerUrl, { ...init, signal: controller.signal });

Provider SDKs accept the same signal, and the streaming helpers expose their own abort:

import httpx
from anthropic import Anthropic

client = Anthropic()

with client.messages.stream(
    model="claude-opus-5",
    max_tokens=2048,
    messages=messages,
) as stream:
    for text in stream.text_stream:
        if client_gone.is_set():
            break          # exiting the with-block closes the HTTP response
        yield text

In the Python case the mechanism is the context manager: breaking out of the loop and leaving the with block closes the underlying httpx response, which closes the connection, which is what the provider observes. Code that iterates a stream without a context manager, or that reads it into a list before processing, has no equivalent moment and cannot cancel.

Whichever form you use, put the partial-text persistence from the previous chapter in a finally, because an abort unwinds the loop the same way an error does and the half-answer is still worth storing. Tag it as cancelled rather than failed while you are there, so that the availability ratio of sentinels to starts does not count a user pressing Stop as an outage.

Two library behaviours will defeat all of this quietly. The synchronous requests library with stream=True holds the response open until you call close() or exhaust it, and abandoning the object leaves the connection in the pool with the body still arriving; the with form or an explicit close() in a finally is required. Connection pooling generalises the problem: a pooled client may keep the TCP connection alive for reuse and drain the remaining body in the background rather than sending a reset, which looks like cancellation from your code's point of view and looks like an ordinary completed request from the provider's.

The pooling case deserves the extra step, because it is the one that survives a correct-looking fix. A connection pool exists to avoid paying for a TCP handshake and a TLS negotiation on every call, and a connection can only be reused if the previous response was read to its end — anything less leaves bytes in the pipe that would be mistaken for the start of the next response. So a pool handed a response nobody wants has two options: reset the connection and throw it away, or read the rest of the body quietly and keep it. The second is cheaper, and cheaper is what a pool is for. From your process the request is finished and the object is gone. From the provider's the socket is open and being drained at the usual rate, so the decoder runs to its natural stop and the usage record is written in full. This is why cancelling has to be verified against what the provider was paid for rather than against what your own code did.

The other place an abort commonly dies is a hop you did not write. Teams routinely put a gateway between their application and the provider — an internal LLM proxy, LiteLLM, an API gateway doing key management and cost attribution — and that gateway is itself a relay with all the framing obligations of one, plus this one. If it reads the provider stream into its own buffer, or if it does not propagate a client disconnect to its own upstream call, then your abort stops there and every hop before it looks correct. Walking the chain means checking each process that holds two sockets, not just the two ends.

The rule that falls out of it generalises past this one signal. Anything that has to be propagated across a pair of connections has to be propagated by a process that holds both, and every such process is a place the chain can end without saying so. That is why "we added the signal" is a statement about one hop rather than about the system, and why the count of hops to audit is the count of relays in the path rather than the count of teams involved.

Showing that it worked#

Reading the diff is not evidence. The failure being fixed is precisely that a chain of correct-looking hops had a gap in it, and the next gap will look correct too — a framework that does not wire req.signal to client disconnect, an HTTP client that ignores the signal once headers are received, a connection pool that keeps the socket alive and lets the provider finish anyway. The check has to be on the outcome.

The deeper reason is that the failure has no per-request observable. Take one cancelled request and ask what distinguishes a working abort from a broken one, and the answer is nothing you can see: the text stops at the same word either way, the socket closes at the same instant either way, the handler returns at the same instant either way. The single quantity that differs is how many tokens the provider went on to produce after you stopped caring, and that quantity is invisible to every log line the request wrote. Worse, it is not decisive even when you can see it, because a short completion on one cancelled request is also what a model finishing early looks like. One request cannot separate the two explanations. A thousand can, because the two explanations predict different shapes.

That is where the distribution comes from, and the prediction is specific. If the abort never leaves the process, every cancelled request runs to its natural stop, so the cancelled cohort's lengths are drawn from the same distribution as everyone else's and the two rows of the table match. If the abort lands, cancelled requests are cut at the moment a human lost patience, and human patience has nothing to do with where the model would have stopped — so the cancelled cohort's lengths are drawn from an entirely different process and the rows separate. The comparison works because the null hypothesis is not "no effect" but "the same distribution", which is a much easier thing to see in a table.

Instrument each request with four fields at stream termination, all of which the finally block from the previous chapter already has access to: the configured max_tokens, the number of output tokens actually delivered, whether the stream ended by client disconnect, and the terminal state.

The output-token figure has to come from a local tokeniser over the text you received, for the reason the accounting chapter earlier in this topic gave: a cancelled stream ends before the usage-bearing event is written, so the provider-reported count is null on exactly the requests you are trying to measure, and any dashboard that sums provider usage will show cancelled requests contributing zero whether or not the fix worked. Counting locally is fine here because the comparison is between two distributions of the same quantity measured the same way, and a systematic 3% offset from the billed figure cancels out of both sides.

Record max_tokens per request rather than assuming a constant, since the cap is often set per route or per model and the ratio is meaningless if the denominator varies within a group. Comparing a fixed cap, as the query below does, keeps that simple.

SELECT cancelled,
       count(*)                                              AS n,
       percentile_cont(0.5) WITHIN GROUP (ORDER BY out_tok)  AS p50,
       percentile_cont(0.9) WITHIN GROUP (ORDER BY out_tok)  AS p90,
       avg(out_tok::float / max_tokens)                      AS frac_of_cap
FROM stream_events
WHERE created_at > now() - interval '7 days'
  AND max_tokens = 800
GROUP BY cancelled;

The shape of the answer tells you which world you are in:

Cancelled requests that are statistically indistinguishable from uncancelled ones are the signature of an abort that never left the process. Users do not press Stop at a moment that happens to reproduce the natural length distribution; if they are stopping a third of the way through, the completions should be a third as long. After the fix the cancelled cohort separates cleanly, and the mean fraction of the cap falling from 0.76 to 0.24 is the claim you can put in front of finance.

There is a second, faster check for a single request. Send one call with max_tokens: 4000 and a prompt that will use all of it, abort at two seconds, wait a minute, and read the provider's own usage export for that request. Billed output tokens near 4,000 means the abort did not land; a few hundred means it did. That confirms one path in about the time it takes to run it, and the distribution query confirms that it holds across the fleet.

What cancelling does not reach#

Cancellation is not instantaneous, and expecting otherwise leads to reading the numbers wrong. Between the click and the provider halting there is a network round trip, the provider's own detection of the closed connection, and whatever decode steps are already in flight in the current batch — realistically 100 to 400 milliseconds, during which tokens are produced and billed. At 60 tokens per second that is 6 to 24 tokens of unavoidable overshoot per cancellation. Sixty tokens a second is six tokens per tenth of a second, so a hundred milliseconds of lag costs six and four hundred costs four times that. Some providers also state plainly that tokens generated before the cancellation takes effect are billed, and that is the honest contract rather than a bug to escalate.

It does nothing for input tokens, and the ordering that guarantees this is worth seeing rather than accepting. Prefill runs over the whole prompt before any token can be decoded, because decoding the first token requires the prompt to have been processed. The first delta is therefore evidence that prefill completed. And the first delta is also the earliest moment at which a user can have seen anything worth cancelling — nobody presses Stop on an empty box for the reason this button exists. So every request a human cancels is, by construction, a request whose input was fully paid for before the human had grounds to act. The entire prompt is paid for on any request that produced any output at all, which on a retrieval-augmented prompt of 12,000 input tokens and 800 output tokens is most of the cost regardless. Cancellation targets the tail, and if your bill is dominated by prompts then the levers are the ones the accounting chapter listed: shorter prompts, a cheaper model, a cached prefix.

It also only fires when there is a human to fire it. Roughly half the wasted generation on a chat product comes from people who close the tab, navigate away, or lock their phone, and those produce a silent close rather than a deliberate Stop — which is fine, because the same propagation handles both, and it is a reason to make the fix at the transport level rather than binding it to a button. What it does not cover is the request whose client is a script with no timeout, or a background job that started a generation and crashed. For those, a server-side ceiling on generation duration — abort the upstream call after 90 seconds regardless of who is watching — is the only backstop, and it belongs in the same place as the stall watchdog from the previous chapter.

It is also not a substitute for max_tokens or a stop sequence, the string that makes the provider halt decoding the moment the model produces it. Those bound the generation server-side without anything having to travel; cancellation is a reaction to something a human did, arriving over a network, and it can only ever be slower. A route that relies on user cancellation to keep its answers short will be expensive on every request nobody bothers to stop.

The shape of the whole mechanism is therefore one property and one query. The property carries a signal your framework already produces across a hop it does not know about, and it is three tokens of code in the Web-standard form and a context manager in Python. The query is what tells you the property is doing anything, because every intermediate hop reports success either way and the only place the difference is visible is in the distribution of how much text cancelled requests produced.

All of this rests on an assumption that is easy to miss: the abort travels promptly because bytes are flowing, and the disconnect is detected on a socket that is actively being written. When something between you and the browser is holding the body rather than passing it along, both the first token and the disconnect are hostage to it, and the first symptom you will see is not the billing.