Retries and duplicate generations

Everything in this topic so far has been about building a request the provider will accept. This chapter is about the request being sent twice. A worker builds a valid array, posts it, and never sees the response — the socket times out, the pod is evicted, a deploy rolls — and something above it does the sensible thing and runs the job again. The second attempt sends a byte-identical array, and by everything established here that is exactly the array the provider is happy to serve: valid roles, intact tool pairs, correct alternation. Nothing rejects it, because nothing is wrong with it.

What follows from that is two distinct problems with two distinct fixes, and conflating them is why this failure keeps recurring after people believe they have handled it. One is that your conversation history ends up holding two assistant replies to one user question, which the next request sends back to the model as though it really had said both things. The other is that you are billed for two full generations. A claim record in your own store solves the first completely and the second not at all; only an idempotency key — an identifier attached to a request so that the provider recognises a repeat and returns the first result instead of doing the work again — addresses the money.

A timeout is not a cancellation#

The thing that makes this class of bug persistent is that the two sides of the connection have different ideas about what happened. Your client abandoned the request; the provider never learned that, kept generating, completed the message, and recorded the usage. Closing the socket does not roll back a generation that is already running, and there is no compensating action available afterwards — the tokens were produced, they are on the invoice, and the response body went nowhere.

The word "abandoned" is carrying the mechanism, so it is worth unpacking. All your client did was stop waiting. Every step that costs money had already happened by then: the request was accepted, the input was read, and the generation loop was running on a machine that has no reason to consult your socket between tokens. There is also nothing you could have called instead. The plain messages endpoint offers no operation for abandoning a generation in progress, and the id on the response is a receipt issued at the end rather than a handle you hold at the start — the first chapter of this topic established that, and this is the first place the absence costs money.

So a retry here is not a second attempt at something that failed. It is a second, complete, independently billed piece of work on an input for which the answer already exists somewhere you cannot reach. That is the sentence to hold against the belief this chapter exists to remove, which is that a retry is free because the first attempt "did not work". The first attempt worked. It succeeded in every respect except delivery, and delivery is the only part you were watching.

The cost is worse than the retry rate suggests, and the reason is a selection effect. Retries do not fall uniformly on your traffic — they fall on the requests that took longest, and requests take longest when the array is largest, which is late in long conversations. On the support-bot shape from the chapter on quadratic input growth, a retried turn twenty-six resends around 15,600 input tokens and pays for a full second generation, while a retried turn two resends 1,200. Those two figures are the same multiplication the earlier chapter used: turn n carries n pairs of roughly equal size, so the array retried at turn twenty-six is thirteen times the array retried at turn two, and the second generation it pays for is priced on all of it.

The consequence is that a retry rate is two different numbers depending on the unit. Counted in requests, it is whatever fraction of calls were redelivered. Counted in tokens — which is what the invoice counts — it is that fraction weighted by how large the retried arrays were, and they were large by construction, because size is what caused the timeout in the first place. A 0.6% retry rate measured in requests can be twice that measured in tokens. On a $73,800 monthly input bill, a retry population concentrated at the tail is a four-figure line item for producing answers that were already produced.

Retries come from more layers than you think#

Most services have four retry mechanisms and are aware of one. Knowing which layer produced a duplicate determines which fix applies, because a guard in your worker cannot see a retry that happened below it.

The one teams most often overlook is inside the client library. The official SDKs retry automatically — typically twice by default — on connection errors and on 408, 409, 429 and 5xx responses. A client-side timeout is a connection-level failure, so by the time your code catches an exception the library may already have issued three requests, and the provider may have completed and billed three generations while your worker was waiting. It also means wall-clock latency can reach roughly the timeout multiplied by the retry count plus one, which is how a 30-second timeout produces a 90-second stall that nobody can account for.

The user-resubmit row deserves its own note, because it is a product question wearing infrastructure clothes. A customer who presses send twice with the same text has made one request in their mind and two in yours; a customer who presses send twice with different text has made two. Deduplicating on a client-generated id attached to the submission gets the first case right without breaking the second, and it belongs at the edge where the user turn is created rather than in the worker.

Where the write goes, and what breaks#

There are two ways to persist the assistant turn, and both of them produce a duplicate under retry — differently.

Post-call persistence is the version most people write first: persist the user message, call the provider, and on a successful response persist the assistant message. Nothing enters the store that the model did not actually produce, which is the property that makes it the sensible default. Its failure window is between the provider finishing and your write committing — a timeout, a crash, an eviction — and in that window the work is done, paid for, and unrecorded, so any retry does it again.

Optimistic persistence writes an assistant row before the response is in hand, usually to drive a UI that shows a streaming reply as it arrives. It gives the user immediate feedback, and it moves the failure window rather than closing it: now a crash mid-stream leaves a half-written assistant row in the store, and unless that row is marked incomplete, the next request will send a truncated assistant turn as though the model had said exactly that much and stopped. The model then continues a conversation in which it apparently trailed off mid-sentence.

Neither column is safe on its own, and choosing between them is not the decision that fixes this. The decision that fixes it is what identifies a turn.

Give the turn an identity and make the write idempotent#

A duplicate exists because the two attempts had no way to recognise each other. Both were "generate a reply for conversation 8812", and the store accepted both because it was asked to insert a row, twice, and inserting rows is what it does. Give the unit of work an identity derived from its input and the second write collides with the first instead of joining it.

The obvious cheaper version — read the table, see whether a reply exists, and insert one if it does not — does not work, and understanding why is what keeps somebody from writing it. Two workers can both perform the read before either performs the write, and both then see an empty result, because a read reports the state at the moment it ran rather than the state at the moment you act on it. Nothing about adding a delay or a second read closes that gap; it only makes the window narrower. A constraint in the database closes it, because the decision is then made in one place, by the component that is serialising the writes anyway: the second insert is refused by the same machinery that guarantees the first one landed. The general shape is worth carrying beyond this chapter — where two processes must agree about a fact, put the agreement in whatever already orders their writes, rather than manufacturing it out of two reads.

The identity that works is the user turn being answered. There is exactly one reply to a given user message, so a unique constraint expresses the invariant directly:

CREATE TABLE turns (
  id               uuid PRIMARY KEY,
  conversation_id  uuid NOT NULL,
  role             text NOT NULL,
  content          jsonb NOT NULL,
  status           text NOT NULL,          -- 'in_flight' | 'complete' | 'abandoned'
  responds_to      uuid REFERENCES turns(id),
  created_at       timestamptz NOT NULL DEFAULT now()
);

-- At most one live assistant turn per user turn.
CREATE UNIQUE INDEX one_reply_per_user_turn
  ON turns (responds_to)
  WHERE role = 'assistant' AND status <> 'abandoned';

The worker then claims the turn before calling out, and completes or abandons it afterwards:

def handle_turn(conv_id, user_turn_id):
    claim = try_insert_in_flight(conv_id, responds_to=user_turn_id)
    if claim is None:                       # another worker owns it
        existing = load_reply(user_turn_id)
        if existing and existing.status == "complete":
            return existing                 # already answered; nothing to do
        raise Retry(after=seconds(5))       # in flight elsewhere; back off

    try:
        messages = build_array(conv_id)     # trimmed and validated
        resp = client.messages.create(model=MODEL, max_tokens=1024,
                                       messages=messages)
        return complete(claim.id, resp.content, resp.usage)
    except Exception:
        abandon(claim.id)                   # frees the slot for a real retry
        raise

The in_flight claim is what makes a redelivered job harmless: the second worker discovers the turn is owned, and either returns the finished reply or backs off rather than generating a competing one. The abandoned status matters just as much, because without it a worker that dies while holding a claim blocks that turn forever; a sweeper that abandons claims older than a few minutes restores the ability to retry deliberately.

Where the provider supports an idempotency key, send one derived from the same turn id. That does not replace the claim — it addresses a different half of the problem, preventing the provider from billing a duplicate rather than preventing your store from holding one — and the two together mean a redelivered job costs nothing and changes nothing.

Why one guard cannot do both jobs follows from where each of them sits, and it is worth a paragraph because teams routinely ship the claim, watch the duplicate replies disappear, and conclude the problem is solved. The claim lives in your database. The provider cannot see it, has never heard of it, and would not change its behaviour if it could. The money, meanwhile, is committed on the provider's side, at the moment a request is accepted and the generation starts — before any of your code gets a chance to notice that this is the second attempt. So the only thing that can prevent the charge is something the provider reads, which means something carried on the request itself. That is the whole of what an idempotency key is: a value the provider stores against the first result, so that a second request bearing it is answered from that result rather than by generating again. Two guards, two sides of the wire, and a duplicate that escapes either one escapes it completely.

Two adjustments make the whole arrangement less likely to be exercised. Set the client timeout above the realistic worst case for your longest arrays rather than at a round number chosen for short requests. And stream long generations, because a streamed response delivers its first bytes in a second or two and keeps the connection demonstrably alive, which removes the ambiguity that a silent socket creates.

Determinism, partial output, and what has to be true underneath#

None of this makes the model's replies deterministic, and the guard does not depend on that. Two generations of the same array produce different text, which is why "just retry, the answer will be the same" is wrong on its own terms as well as on cost — the customer above received two different shipping estimates for one question. If anything, non-determinism is what makes the duplicate visible: two identical replies would have looked like a UI bug.

The guard also does not address partial output. A stream that dies at 60% leaves you billed for what was produced and holding an incomplete answer; the claim record keeps that partial out of the conversation history, and what to do with it — discard and regenerate, or ask the model to continue from the tail — is a product decision. Discarding is simpler and costs a second generation. Continuing is cheaper and requires the array to say honestly that the assistant turn was cut off.

And it does not remove the need for the earlier invariants. The retried array in this chapter was valid only because everything before it was correct: history shared across workers rather than held in one process, a window cut at legal boundaries, tool pairs intact, the instruction in the channel that outranks user text. A duplicate-suppression guard on top of an array that a second worker would have rebuilt differently suppresses nothing useful, because the two attempts were never the same request to begin with.

The rule to carry out of this chapter is that a turn is a unit of work with an identity, not a function call that either returns or does not. Give it that identity in your own store, claim it before you spend money on it, and mark it abandoned when you stop; attach an idempotency key so the provider can recognise the repeat that your code never sees. Those two guards are perhaps forty lines together, and they convert the most common operational failure in this area — one that scales with conversation length, and therefore hits your most engaged users hardest — into a job that runs twice and changes nothing.

You now have the whole request: the endpoint and the body, the instruction channel and what it is weighted above, the shape of a tool exchange and the ids that bind it, a history you own and can rebuild anywhere, a window measured in the right unit and cut in the right places, and a turn that survives being delivered twice. That is a client you can write in an afternoon and run for a year. What it produces is still just text arriving in a content list — and the next thing worth learning is what the vendor library you will actually use puts between your code and that body.