Retrying a cut-off stream

A retry around a streaming request is standard treatment for a network call that fails halfway, and dispatching a tool the moment its call reports itself finished is the rule the previous four chapters have been building toward. Each is correct on its own. Together they produce a system in which a connection reset a second after a payment was accepted causes the payment to be made again, with the model as the willing accomplice, because the turn it is asked to redo contains no record that any of it happened.

This chapter is about the window that makes that possible, why removing the retry does not close it, and where the guarantee has to live instead.

The window between the side effect and the end of the stream#

Two pieces of a handler combine to produce a duplicate, and each is defensible in isolation: the dispatcher fires on the per-call completion event, and the streaming request is wrapped in a retry that re-issues it once on any stream-level error.

The reset at t=3.1s arrived after the completion event and after the payment had already been accepted, in the window where the stream was still open because the provider had yet to send its message-level stop. That window is short and it is not negligible: on this deployment the gap between the last content block stopping and message_stop averaged 240 milliseconds, and the dispatcher's own call to the payments API took 820, so the process spent about a second in a state where a side effect had happened and the stream could still fail.

Add those two figures rather than reading them separately, because the sum is the quantity that matters and neither number is it. The dangerous interval opens the instant the dispatcher is called and closes when the stream is finished with. Its length is the time the tool takes plus the time the provider then spends before sending its message-level stop — 820 milliseconds and 240 milliseconds, a little over a second in total. Addition is right here rather than taking the larger, because the two intervals happen one after the other: the payment call has to return before the handler can go back to waiting on the stream, and the provider's final event has not arrived either way.

Two things follow that are worth carrying to your own system. The window grows with tool latency, so the slowest write tool you have is the one that defines your exposure, not the average. And the window exists even if your tools are instant, because the provider's own gap before message_stop is not yours to shorten. There is no way to make this interval zero from inside the handler, which is the argument for putting the guarantee somewhere the interval cannot reach.

What the retry then sent was the conversation as it stood before the turn, because the assistant turn and its tool result are only committed once the turn finishes — the reason for that is chapter 8's, and it is correct. The model received input identical to the first request, and produced the obvious continuation, which was the same payment. It had no way to know it was repeating itself, since nothing in its input mentioned a payment having been made. The second call also carried a different toolu_ identifier from the first, because call ids are generated per response, which matters for the fix.

That chain is worth walking one link at a time, because each link is a rule from earlier in this book behaving exactly as designed. The provider holds no state between requests, so everything the model knows about the situation is in the message list you send. The message list is only extended once a turn completes, because a half-finished turn is not a valid conversation. The turn did not complete, so nothing was appended. The retry therefore sends byte-for-byte what the first request sent. A model given identical input produces materially the same output, which is the property that makes the system useful and here makes it dangerous. There is no defect anywhere in that sequence. The duplicate is what correct components produce when the record of the side effect lives somewhere the next request does not read.

The counter-argument that this cannot happen is that a cut stream means nothing was dispatched, so a retry is free. That holds for every cut that lands before the completion event, which is most of them — on this system, 89% of stream failures occurred while arguments were still streaming or before any call was announced. The remaining 11% landed in the window above, and 11% of a failure rate of 0.6% of turns is roughly one turn in fifteen hundred, which on a few thousand payment turns a month is several duplicates.

That figure is two multiplications and one reciprocal, and it is worth doing because the same shape appears whenever a rare condition sits inside another rare condition. Streams fail on six turns in every thousand, which is six thousandths. Of those failures, eleven in a hundred land after dispatch, which is eleven hundredths. Multiplication is the right operation because the second proportion is taken of the first — the failure has to happen and then has to land in the window — and six thousandths of a turn multiplied by eleven hundredths comes to about seven parts in ten thousand. The reciprocal of that is roughly fifteen hundred, which is the interval between duplicates stated as one turn in that many.

Notice what the multiplication does to intuition. Each of the two rates sounds like a rounding error on its own, and their product sounds like an impossibility. But the denominator is turns, and a system doing thousands of payment turns a month crosses that interval several times over. The same figure would be genuinely negligible on a hundred turns a month and a weekly incident on a hundred thousand, without anything about the code being different. A rate is only alarming or not alarming once it has been multiplied by volume.

Where the stream died determines what is safe#

The accumulator already knows which of these situations you are in, which is the practical reason chapter 6's finalisation retains state rather than discarding it on error.

The two rows in the middle are the ones the naive retry gets wrong, and the third row is worse than the fourth. When the connection drops while the POST to the payments API is still outstanding, you do not know whether the payment was accepted; the request may have reached the provider and the response may have been lost. That is the classic distributed-systems uncertainty, and no amount of care in the streaming layer resolves it, which is exactly why the resolution has to live downstream of the streaming layer.

Removing the retry does not fix it#

Stopping retries on stream errors removes this specific path, and it is the wrong place to intervene for two reasons.

The retry is doing real work on the 89% of failures that land before dispatch. Removing it converts a class of invisible, self-healing hiccups into visible failed turns, and the user's response to a failed turn is to ask again — which sends a second request with the same conversation input, producing precisely the duplicate the change was meant to prevent, now initiated by a human instead of by a loop. The same substitution happens through every other route that can replay a turn: a client that reconnects and re-sends its pending message, a job queue with at-least-once delivery, which redelivers a message rather than risk dropping one, a load balancer retrying an idempotent-looking POST, a user on a flaky train connection hitting send twice. A retry policy is one of many things that can cause a turn to be issued twice, and it is the only one you can delete.

The deeper problem is that the dispatcher has no memory. It will execute any well-formed call handed to it, as many times as it is handed one, and nothing about it distinguishes a call it has already performed from a new one. A system with that property produces duplicate side effects under enough different conditions that enumerating and blocking them is not a strategy.

The key that survives the retry#

Idempotency in the dispatcher means that dispatching the same logical call twice performs the effect once and returns the same result both times. The whole design question is what "the same logical call" is keyed on, and the obvious candidate does not work: the call id is generated by the provider per response, so the retried turn carries a different one, and keying on it deduplicates nothing.

The key has to be derived from things that are identical across a re-request of the same turn and different across a genuinely new one. That gives you the conversation, the position of the turn within it, the tool name, and the arguments in a canonical form:

import hashlib, json

def idempotency_key(conversation_id: str, turn_index: int,
                    name: str, args: dict, ordinal: int = 0) -> str:
    canonical = json.dumps(args, sort_keys=True, separators=(",", ":"))
    material = f"{conversation_id}|{turn_index}|{name}|{ordinal}|{canonical}"
    return hashlib.sha256(material.encode()).hexdigest()[:32]

def dispatch(call: Dispatchable, ctx: TurnContext) -> ToolResult:
    key = idempotency_key(ctx.conversation_id, ctx.turn_index,
                          call.name, call.args, call.ordinal)
    existing = ledger.claim(key)          # INSERT ... ON CONFLICT DO NOTHING
    if existing is not None:
        return existing.result_or_raise() # completed, or still in flight
    try:
        result = tools[call.name].run(call.args, idempotency_key=key)
    except Exception as e:
        ledger.fail(key, e)
        raise
    ledger.complete(key, result)
    return result

The turn_index is what stops a legitimate second payment from being swallowed. A user who genuinely wants to pay the same invoice twice does so in a later turn, which has a different index and therefore a different key. The ordinal handles the narrower case of two identical calls inside one turn, distinguishing the first from the second by position so that a model that really did emit the same call twice is not silently collapsed — that situation is usually a model error, and the right response is to let it through and notice it in the logs rather than to hide it in the deduplication layer.

Each of the other three components earns its place by a similar test: remove it and describe what breaks. Without the conversation id, two different users paying the same supplier the same amount from the same turn position collide, and one of them silently receives the other's result. Without the tool name, a send_payment and a void_payment with coincidentally identical arguments in the same turn hash together. Without canonicalising the arguments, the key stops being stable across the very retry it exists for: json.dumps on two semantically identical dictionaries can emit the keys in different orders and different spacing, producing different digests for the same request, and a key that changes between attempts deduplicates nothing at all. Sorting the keys and removing the whitespace is what makes "the same arguments" a property of the values rather than of how they happened to be serialised.

The claim has to be made before the tool runs, and it is worth seeing why the obvious ordering fails. Recording the key after a successful call is simpler and handles the case where the first attempt finished; it handles nothing else. The attempt that matters is the one interrupted in flight, and an interrupted attempt never reaches the line that would have recorded it. So the retry finds no row, concludes nothing has happened, and does the thing again — which is the original bug with a ledger next to it. Claiming first converts the uncertain state into a recorded one: the row exists and is unfinished, which is a fact the retry can read and act on.

Two storage layers carry the guarantee. The local ledger row is claimed before the tool runs, so a process that dies mid-POST leaves an in-flight row that the retry finds, which turns "I do not know whether the payment went out" into "another attempt is outstanding, wait for it or fail the turn" instead of into a second POST. And the same key travels downstream as the payment API's Idempotency-Key header, so even if your ledger write is lost the payment provider collapses the duplicate on its side and returns the original payment id. The two layers protect against different failures — the ledger against your own concurrency, the header against your own data loss — and neither substitutes for the other.

Not every tool needs the full apparatus, and deciding which do is a per-tool judgement made once, at registration time, rather than a global setting. Pure reads need nothing, since a repeated search_orders costs a query and returns the same rows. Tools whose downstream API already accepts a caller-supplied key — most payment processors, most ticketing systems that take an external reference — need the key derived and forwarded and can skip the local ledger if you are willing to accept an extra downstream round trip on the duplicate path. The tools that need both layers are the ones that create something with a server-generated identifier and no deduplication of their own, which in practice is internal APIs written by teams who never expected to be called twice. Marking each tool as read, keyed_write or unkeyed_write in the same registry that holds its schema keeps the decision visible next to the thing it describes, and makes the audit question — which of our tools can double-fire — answerable by reading one file.

When nothing ran, say nothing ran#

With the ledger in place the retry policy needs no change at all. The automatic retry after a stream error re-issues the turn, the model emits the same call with a new id, the derived key matches the claimed row, and the dispatcher returns the first payment's result without touching the payments API. The model then sees a tool_result describing a payment that succeeded, which is true, and the user sees one payment, which is also true. Provider-level request idempotency keys, where they exist, do not help here; they deduplicate the inference request and protect your token bill, and the duplicate you care about is downstream of that.

The distinction there is one people lose, and it is worth stating in the terms of the two systems involved. A provider's request idempotency key says: if you send me this request twice, I will charge you once and return the same completion. It is a statement about the inference call. Your dispatcher's key says: if this logical tool call is performed twice, the effect happens once. It is a statement about the payment. The first says nothing about the second, and the reason is visible if you grant the provider's guarantee in full. Suppose the retried request is recognised as a repeat and the provider returns the identical assistant turn without re-running inference. That turn still contains a send_payment call, your handler still reaches the completion event, and your dispatcher still executes it. Perfect deduplication upstream leaves the downstream effect exactly where it was, because the effect is not performed by the thing being deduplicated. Two mechanisms with the same name, guarding two different resources.

The other half of the cut-off case is chapter 6's Abandoned state: a connection reset partway through the arguments of a create_invoice call, after the preparing indicator has already appeared. Here the ledger has no row, the dispatcher was never called, and the invoice does not exist. The correct handling is a failed turn — the call block moves to "not run (connection lost)", the partial arguments go to the log, and nothing is sent to any downstream system.

The temptation to avoid is fabricating a result to keep the conversation coherent. A synthetic tool_result saying "the invoice could not be created" reads as reasonable and is a statement about a call that does not exist from the provider's point of view, since no completed tool-use block was ever produced. A synthetic result saying the invoice was created is worse in the obvious way. Both belong to the same family of shortcuts, which is writing into the conversation record something the stream never established.

Two facts decide everything else here. Where the stream died determines what has already happened in the world, and the accumulator is the only component that knows. And a turn can be issued twice through a dozen routes, of which the retry policy is one, so the component that has to tell a repeat from a new request is the one closest to the effect. Derive the key from the turn, claim it before you act, and forward it to anything that will accept one.

That leaves the conversation record itself, which is the last place a half-finished call can do damage, because everything after it reads from there.