Choosing a disconnect policy

A disconnect policy is the rule your handler applies when it learns that the client is gone. It is a decision that most services make by accident, because doing nothing is itself a policy — the one where the generation finishes into a socket nobody is reading and nothing is written down. Naming the two real options is what turns an accident into a choice.

Everything that sounds like a third option collapses into one of them. Pausing is not available, because a decoder holds no resumable state you can address; the provider is either producing tokens or it is not. "Finish only if it is nearly done" is detach with a predicate. "Cancel but keep what we have" is cancel plus persistence, which is worth doing under both policies and is the next chapter's subject.

The reason there is no third option is a mechanism rather than a list of things that happen not to work. A running generation is addressable only through the request that started it: the provider issued no identifier for a half-finished decode, exposes no endpoint that would take one, and offers no way to hand the partial state back to you and pick it up later. The single lever your process holds is therefore the upstream socket. Holding it open keeps the decoder running and keeps the meter running; destroying it stops both. Every proposal that sounds like a compromise turns out to be one of those two levers with a condition attached to when it is pulled.

What actually decides it#

The deciding quantity is the expected cost of not having the answer, compared against the cost of finishing it. Both are computable from numbers you already have.

Take a long-form generation of the kind that produces a document: an evidence summary for an audit, carrying about 12,000 input tokens of retrieved material and producing about 7,000 output tokens over ninety seconds. At $3 per million input and $15 per million output, one run costs roughly $0.036 + $0.105 = $0.141. A disconnect at 60 seconds has already burned the input and about two thirds of the output, so cancelling recovers about $0.035. Against that, measure the rate at which users re-run after a cancelled attempt. On a workflow whose whole purpose is the artifact, that rate is high — 74% in one measured case — because the user came to get the document and will come back for it. Expected cost of cancelling is then $0.035 saved against $0.104 spent generating it again, before counting the user's ten minutes. Detach is not a close call.

Each figure there is one multiplication, and they are worth doing rather than trusting, because the comparison at the end turns on the smallest of them. Twelve thousand input tokens at $3 per million is 12,000 divided by a million and multiplied by 3, which is $0.036. Seven thousand output tokens at $15 per million is the same operation on the other side and gives $0.105, and the two add to $0.141. A disconnect at 60 seconds of a ninety-second run has already produced about two thirds of the output, so what cancelling avoids is only the remaining third, and a third of $0.105 is $0.035. The input is untouchable at that point, because it was charged in full when the request was accepted. Then the other side of the ledger: 74% of users come back for the document, and each of those re-runs costs the whole $0.141 again, so the expected cost of having cancelled is 74% of $0.141, which is $0.104. Spending $0.104 to save $0.035 is the comparison, and no plausible correction to the inputs brings the two within reach of each other.

Now take an in-editor code assistant: about 800 input tokens and 300 output, one generation costing roughly $0.0069, and a disconnect at token 60 recovering about $0.0036. Users there disconnect because they changed their mind, moved to a different file, or fixed it themselves, so the finished answer is asked for again in about 8% of cases, and when it is asked for again the prompt has usually changed. Detaching would spend $0.0036 per abandoned generation to produce an artifact with a 92% chance of never being read, and would hold a worker for seven seconds each time it did.

That is the same arithmetic with the sign reversed, and the second half of it is the part that decides. A disconnect at token 60 leaves two hundred and forty of the 300 output tokens unproduced, which is why cancelling recovers $0.0036 — four fifths of the output side, against the one third the long-form case recovered. On the tokens alone, cancelling looks better here than it did there. What differs is what happens afterwards: the re-ask rate is 8%, so an artifact produced by detaching is read eight times in a hundred and paid for a hundred times out of a hundred. The 92% is not an estimate of anything, it is one minus that measured rate, and it is the figure that turns a larger token saving into the wrong decision rather than the right one.

Three properties travel with high-value artifacts and are worth checking directly, because they predict the answer more reliably than the token arithmetic: the generation takes long enough that a user will look away from it, the output has a home in the product that is not the streaming view (a document, an export, a record), and the user cannot cheaply reconstruct it by asking again. When all three hold, detach. When none holds, cancel. Mixed cases go to whichever of the first and third dominates, and the second is the one that quietly kills detach implementations, because a finished answer nobody can navigate back to is the same as no answer with a bill attached.

The policy therefore belongs to the route rather than to the service, and a gateway shared by several products needs it resolved per request, from tenant configuration or from the endpoint being called. One product commonly wants both: the same assistant that answers a chat turn under cancel also generates an exportable summary under detach, and the difference between those two calls is not visible anywhere in the networking code. Where nobody has chosen, cancel is the safer default, because a detach policy without a retrieval path in the product is the worst cell in the table above — full price paid, artifact produced, nothing that leads a user back to it.

The stumble underneath that default is a belief about which policy is the careful one. Cancel reads as the frugal option, because it is the one that stops spending, and a reviewer weighing "stop paying for output nobody reads" against "pay for it anyway" will take the first every time. What the framing leaves out is that not having the answer is also a cost, charged to the same account a few minutes later at full price and with the user's time added on top. Cancel is frugal exactly when the answer is cheap to reproduce. Where it is not, cancelling is a decision to buy the same artifact twice, and the saving on the first purchase is what pays for part of the second.

The handler under both policies#

Detaching means the generation's lifetime stops being the request's lifetime, which is a structural change and not a flag. The read loop has to move somewhere that outlives the response.

type DisconnectPolicy = "cancel" | "detach";

export async function POST(req: Request): Promise<Response> {
  const { tenantId, params } = await req.json();
  const policy: DisconnectPolicy = policyFor(tenantId);
  const generationId = crypto.randomUUID();

  // The upstream call is owned by the generation, never by the request.
  const controller = new AbortController();
  const upstream = await fetch(PROVIDER_URL, {
    method: "POST",
    headers: providerHeaders,
    body: JSON.stringify({ ...params, stream: true }),
    signal: controller.signal,
  });

  const pump = runGeneration(generationId, upstream.body!); // writes durably

  req.signal.addEventListener("abort", () => {
    if (policy === "cancel") {
      controller.abort();
      metrics.increment("generation.cancelled", { tenantId });
    } else {
      metrics.increment("generation.detached", { tenantId });
      // pump is already running and is not tied to this response
    }
  });

  return new Response(tail(generationId, pump), {
    headers: {
      "content-type": "text/event-stream",
      "x-generation-id": generationId,
    },
  });
}

The shape that matters is that runGeneration is started once and referenced by the response, rather than being driven by it. On a long-lived server that is enough: the promise keeps running in the event loop after the response object is destroyed. On a serverless platform it is not, because the execution environment is frozen the moment the response completes, and a detached pump is simply suspended mid-decode with its upstream socket dropped underneath it. Platforms expose an escape hatch — ctx.waitUntil(pump) on Cloudflare Workers and Vercel's runtime, after() in Next.js — that keeps the invocation alive until the promise settles, and each comes with its own wall-clock ceiling, commonly 30 to 300 seconds. If your detached generations can run longer than that ceiling, the handler cannot own them at all, and the generation has to be started by a durable worker that the HTTP request merely subscribes to.

Why the environment freezes rather than simply carrying on is worth one step, because it settles whether those escape hatches are a nicety or a requirement. A serverless platform charges for the time your code is executing and reclaims the container the moment it believes there is nothing left to execute, and the signal it uses for that belief is the response completing. A promise still pending at that instant is not something the platform is watching; it is state inside a runtime that is about to stop being scheduled. ctx.waitUntil and after() exist to move that promise into the platform's own bookkeeping, which is the only place a decision to keep running can be taken. It is also why they come with a ceiling: the platform is now billing wall-clock time that no request is waiting on, and it will not do that indefinitely.

The shape each policy leaves on a dashboard#

A disconnect policy fails quietly, because the code path only runs when something has already gone wrong, and neither failure raises an error. What each policy leaves on a cost dashboard is specific enough to alert on.

Under cancel, the tell is the ratio of billed to delivered output tokens. A working policy sits a few percent above 1.0, from the in-flight tokens described in the previous chapter. When the abort stops propagating — a library upgrade that no longer forwards the signal, an intermediary that holds the connection open after the browser leaves — the ratio climbs to 2 or 3 and correlates with the share of traffic on mobile networks, because that is where disconnects live. Nothing errors; the bill just grows.

Under detach the tell is on the other side. A working detach policy costs the full answer every time, so cost per generation is deliberately flat and tells you nothing. What tells you something is the count of detached generations that reached complete, compared against the count that started detaching. A gap there means the pump is dying with the process — a deploy, a scale-in event, a frozen serverless invocation — and the user experience is identical to having no policy at all. The second signal is generations started per session: if detach were failing invisibly, users would be re-running, and re-runs show up as two generations with near-identical prompts inside ten minutes.

The ratio's reading in the broken-detach column follows from what dies with the process. A pump killed mid-decode takes its upstream connection down with it, so the provider observes the transport close, stops the decoder and stops billing at roughly the moment delivery stopped — and billed over delivered settles near 1.0, which is the number a healthy cancel policy produces. Broken detach and working cancel are the same picture on a cost dashboard. That is why the detach row in the table is a pair of counts rather than a ratio: only detach_started measured against detach_completed can separate a generation that finished into storage from one that was cut off on the way there, because the difference between them is not visible in money at all.

Where detach stops being enough#

Detach solves ownership. It says the generation belongs to the system rather than to a socket, and it makes the bill defensible by turning paid tokens into a stored artifact. It does not, on its own, put the answer back in front of the user, and it does not survive the thing most likely to kill it.

The gap is that the detached pump still holds the only complete copy of the answer in memory until it finishes and writes. A pod eviction at second 70 of a 90-second report loses everything, including the 55 seconds of output that existed before the user's laptop even slept, and the cost dashboard records a fully billed generation with no artifact at the end of it. That failure is more likely than it sounds, because the events that recycle a pod — a deploy, a scale-in, a node drain — are routine and are concentrated in working hours.

A user who reconnects at second 40 is in the same position for a different reason, because there is nothing to hand them except the finished document that does not exist yet. Detach as described gives you one moment of usefulness, at the end, and nothing in between; a reader who returns before that moment is told to wait with no way to see how far along the work is.

So the two policies are the whole decision space, and the way to make it is to price the artifact rather than the tokens: how likely is the user to want this answer, and what does it cost to make it again. Cancel where the answer is cheap to reproduce and nobody will come looking for it. Detach where it is not, and accept that detach is a larger change than a flag, because it requires a generation id, a durable place to put the output, and a route in the product that leads back to it.

Making detach worth its cost requires one more thing: the output has to exist durably while it is being produced, chunk by chunk, rather than only once the decoder stops. That changes what the read loop writes, and in what order.