Retry belongs above the layer that owns the visible effect

The trace is 68 seconds long and every span in it is green. The gateway authenticated the request at t=0 and started a 60-second deadline. The orchestrator read six prior turns from Postgres, queried the vector store, composed a 3,100-token prompt and opened a streaming call. Tokens began arriving at t=1.4s. At t=42s, after roughly 400 tokens had been written to the SSE connection — server-sent events, the long-lived one-way HTTP stream a browser holds open to receive an answer fragment by fragment — and rendered in the browser, the provider's connection dropped mid-stream. The shared provider client caught the transport error, applied its failover policy, opened an identical call against the second provider, and began yielding deltas again from the start of a fresh completion. The user watched the answer they were reading disappear and restart with a different opening sentence. Meanwhile the orchestrator's stream handler, which appends on interruption so that a half-finished answer is not lost, had already written the 400-token fragment to Postgres as an assistant turn. When the second completion finished, it wrote a second turn. Two rows, one question. The usage recorder wrote one receipt per provider call, so the tenant was billed for both generations, 2,730 tokens in total. At t=60 the mobile client's own timeout fired and it re-POSTed the whole request, which produced a third generation and a third turn.

Error rate for that minute: 0.0%. Every span reported success, because from each layer's point of view nothing failed. The client retried and succeeded. The orchestrator received a complete stream and committed it. The recorder recorded what it was given. The defect is not in any of those behaviours individually; it is that the layer performing the retry could not see the effects of the call it was retrying.

The rule, and where the line actually falls#

A layer may retry an operation whose effects have not escaped it. That is the whole rule, and the work is in knowing what "escaped" means for each layer.

For a provider client, the effects of a call are an outbound HTTP request, a parsed response object, and possibly a token count. For a unary JSON extraction — send a document, get back a structured object — none of those effects escape the client until it returns. A connection reset at 8 seconds means the caller never saw anything, no state moved, and nothing was shown. Retrying inside the client is not merely safe there, it is the right place: the retry has access to the exact request, the vendor-specific error taxonomy, and the connection pool, and putting it anywhere higher duplicates that logic into every caller. This is the case the shared-client argument is built on, and for that case the argument is correct.

For a streaming call the escape happens at the first delta. The moment the client yields a token that the orchestrator writes to the transport, the effect is outside the client's blast radius and inside the user's browser, and no error handling inside the client can undo it. The client at that point has two facts it cannot know: what the user was shown, and whether anything was written down. Both belong to the orchestrator.

The refinement worth implementing is that streaming is not uniformly unsafe to retry — it is safe up until first byte, which in practice covers a good share of transient failures. Connection setup errors, 429s before the stream opens, and 5xx responses to the initial POST all happen with zero deltas emitted. A client that tracks whether it has yielded anything can keep the free case and hand the rest up.

class StreamInterrupted(Exception):
    def __init__(self, cause: Exception, emitted: int, provider: str, text: str):
        self.cause, self.emitted = cause, emitted
        self.provider, self.text = provider, text


async def stream(self, req: ChatRequest) -> AsyncIterator[Delta]:
    emitted = 0
    text: list[str] = []
    for attempt, provider in enumerate(self._failover_order):
        try:
            async for delta in provider.stream(req):
                emitted += 1
                text.append(delta.text)
                yield delta
            return
        except TransportError as exc:
            if emitted > 0:                      # an effect has escaped
                raise StreamInterrupted(exc, emitted, provider.name,
                                        "".join(text)) from exc
            if attempt == len(self._failover_order) - 1:
                raise
            # nothing escaped: failover is invisible and free
            log.info("failover", extra={"from": provider.name, "cause": str(exc)})

Eight lines of condition, and the client stops making a decision it has no information for. What replaces the transparent retry is an exception carrying the two facts the caller needs: how much escaped, and what it was.

The orchestrator can then choose, and it has three real options rather than one.

try:
    async for delta in client.stream(req):
        await sse.send(delta.text)
except StreamInterrupted as exc:
    if exc.emitted < RESUME_MIN_TOKENS:            # 40 tokens: cheap to discard
        await sse.send_event("restart", {"reason": "provider_interrupted"})
        req2 = req                                  # same prompt, clean slate
    else:
        await sse.send_event("resuming", {"after_tokens": exc.emitted})
        req2 = req.with_assistant_prefix(exc.text)  # continue, do not repeat
    async for delta in client.stream(req2):
        await sse.send(delta.text)

Resumption by prefilling the assistant turn with what was already delivered — sending the 400 tokens back as the opening of the assistant's message, so the model continues that message rather than composing a new one — is supported by every major chat API, and it is the option that preserves what the user has read. It is not free: the continuation is a second billable call whose prompt now includes the 400 tokens already produced, so the wasted spend is the prefix, not the whole answer. On the incident above, resuming rather than restarting would have cost roughly 400 extra prompt tokens — about $0.0012 at $3 per million input tokens — against the 1,200 completion tokens that were regenerated and thrown at the user twice.

Resumption has caveats that are worth knowing before it is designed in. Assistant prefill is well supported for plain chat completions and is restricted or unavailable when the interrupted call was in the middle of a tool use block, because the partial tool call is not something you can meaningfully continue by prefixing text. Some providers reject a prefill ending in trailing whitespace, so the prefix has to be right-stripped. Prompt caching, where a provider charges a reduced rate for a leading stretch of the prompt it has processed recently and holds in memory, interacts with it favourably, since the resumed call repeats the same system prompt and retrieved context and will usually hit the cache, which is why the marginal cost of resuming is dominated by the delivered prefix rather than by the 3,100-token context. Failing over to a different vendor while resuming is the case to avoid, because the second model continues a passage written by the first and the seam is often visible in the register of the prose; resuming on the same provider and failing over only when nothing escaped keeps the two mechanisms from interfering.

The restart event matters as much as the resume path. A client that is told the stream restarted can clear the rendered buffer; a client that is told nothing renders two answers concatenated, which is how "the model repeated itself" bugs get filed against a prompt that is fine.

Which layer for which call#

The answer genuinely differs by call class, which is why "put failover in the client" and "put failover in the orchestrator" are both wrong as blanket policies.

The tool-loop row is the one that generalises the rule beyond streaming. A retry inside the client of a call that already triggered issue_refund re-runs the model's decision to issue a refund, and the client has no idea a refund exists. Streaming is just the most visible instance of a general condition: the moment a call's output has driven an effect the caller committed to, the call stopped being a pure function and retrying it is a business decision.

The opposite arrangement fails too, and it fails quietly enough that teams who have been burned by the incident above sometimes over-correct into it. Strip the provider client of all retry, make it a thin transport that surfaces every error, and the logic it was doing does not disappear — it reappears in each caller, in inconsistent forms. One team implements 429 handling by reading retry-after, another sleeps a flat two seconds, a third treats 429 as fatal and pages on-call. Connection pooling and timeouts drift apart per caller because there is no longer one place that owns them. The vendor error taxonomy leaks upward, so feature code starts containing except openai.APIConnectionError, which is the exact leak the port in topic 2.18 exists to prevent, and adding a third vendor becomes a change to every feature rather than to one adapter. The thing that silently breaks is the abstraction's substitutability: it still compiles, still passes tests, and the day you want to shift 10% of traffic to a new vendor you discover that three of your seven callers have vendor-specific handling that does not survive the move.

The workable arrangement gives each layer the half it can see. The client owns everything determined by the transport and the vendor — connection reuse, timeouts, the mapping from vendor errors to a common taxonomy, retry-after honouring, and failover for calls whose effects have not escaped. The orchestrator owns everything determined by the operation — whether a partially-delivered answer is resumed or restarted, how many total attempts the deadline allows, what the user is told, and what is written down.

Detecting a violation in a running system is worth setting up before you need it, because the failure produces no errors. Two counters catch nearly all of it. The first is the ratio of assistant turns to user messages per conversation, which should sit at 1.0 and which drifted to 1.06 on the service in the trace above; anything above 1.01 sustained means something is writing turns that no user asked for. The second is a stream_restarts_total counter incremented on the orchestrator's restart path, labelled by provider and cause, which turns an invisible user-experience defect into a graph. Neither is expensive, and both are the kind of instrumentation that only gets built after an incident unless someone puts them in the design.

Ownership of the receipt follows the same logic. A usage record per provider call is a correct record of provider calls and an incorrect record of user operations, and billing is charged for operations. The recorder should write one receipt per operation with the attempt detail as fields — attempts: 2, providers: ["a","b"], billable_tokens: 2730 — so that the finance rollup and the incident review read the same row.

What this does not fix#

None of this is an argument against the shared provider client. The client keeps everything that is genuinely about transport: connection pooling, timeout configuration, the vendor error taxonomy, 429 handling with retry-after, and pre-first-byte failover. What moves up is the decision that depends on state the client cannot see, and the surface area of that move is one exception type. If the orchestration code in three different features ends up writing the same except StreamInterrupted block, that block belongs in a shared orchestration helper rather than pushed back down into the client — the fix for duplication is a shared function at the right layer, not the wrong layer.

The rule also has nothing to say about retries that originate outside your system. The mobile client's re-POST at t=60 was not a violation of any layer boundary, because the mobile client is not one of your layers; it is a caller that timed out and did what callers do. No amount of correctly-placed internal retry prevents an external retry from arriving as a brand-new request with a fresh trace, a fresh prompt composition and a fresh commit. That case produces a duplicate that costs real money and corrupts real state, and it is worth being precise about which half of it is the expensive half.