Idempotency keys and replay safety

Cancelling a request is not a message to the provider. Your client stops waiting, closes a socket, and raises an exception, and none of those actions tell the server to stop decoding. An abandoned stream does not stop the meter, and the same holds for a non-streaming call, where the generation runs to completion, is billed, and the response is written into a connection nobody is reading.

That leaves the client in a genuinely ambiguous state, and this chapter is about what to do inside it. An idempotency key is the standard instrument: a caller-generated identifier attached to a request, which the server stores alongside the outcome for a retention window, so that a request arriving with a key it has already seen gets the stored response instead of a second execution. The instrument is exact about what it protects — the provider's operation, on the provider's side — and the interesting part of the chapter is the two cases it does not cover: a stream whose output has already been rendered, and a tool call whose effect has already left the building.

Three things that may have happened, and no way to tell#

When a call times out, the client knows only that no response arrived within the deadline it set. Three distinct things may be true on the other side.

Case (a) makes a retry free and correct. Case (b) means the retry runs a second generation concurrently with one you are still paying for. Case (c) means you pay twice for an answer that already existed. No local evidence distinguishes them, and a policy built on the hope that it is always (a) is a policy that orders more generations than it uses. The frequency of (b) is set by where the timeout sits in the endpoint's completion-time distribution: a 30-second timeout on a path whose p50 is 11 seconds, p95 is 24 and tail runs past 40 abandons roughly four requests in a thousand while the provider is still working on them.

That last figure is read straight off the distribution, and reading it is the only way to know whether case (b) is a curiosity or an invoice. A completion time of 30 seconds sits between the p95 of 24 and the tail past 40, and on this endpoint it lands at about the 99.6th percentile — which is a statement that 99.6% of calls finish inside it. The remainder is what a timeout at that point cuts off: a little under half a per cent, or roughly four requests in every thousand. Nothing about that number is a property of the timeout mechanism. It is a property of where one number was placed on one distribution, and moving the number moves the rate, which is why the same 30 seconds is harmless on a path whose tail ends at 5 seconds and expensive here.

The key removes the ambiguity by making the second attempt refer to the first. The server stores the key alongside the outcome, and a duplicate that arrives while the original is still in flight is either held until the original completes or rejected with a conflict status, depending on the provider.

Why the identifier has to come from the caller is the step this whole instrument rests on, and it follows from a question the server cannot answer. Two requests arrive carrying identical bytes. One of them is a retry of the other, and the correct response is the stored outcome. The other is a second, deliberate submission of the same work — a user asking the same question twice — and the correct response is a fresh generation. Nothing in the bytes separates those two situations, because the bytes are the same in both. The only party that knows which one this is, is the party that decided to send it. So the distinguishing fact has to be manufactured by the caller and carried in the request, and that is all an idempotency key is: the caller's own statement that this submission is, or is not, the same operation as an earlier one.

The in-flight case follows from the same reasoning. A duplicate arriving while the original is still running cannot be answered with the stored outcome because there is no stored outcome yet — only a record that something with this key started. Holding the duplicate until the first finishes and rejecting it with a conflict are two ways of saying the same thing, which is that the server will not run this operation twice and cannot yet tell you how the first run ended.

# Attempt 1 and every retry of it carry the SAME key.
curl https://api.provider.com/v1/messages \
  -H "authorization: Bearer $KEY" \
  -H "idempotency-key: 7f3b1c9e-2a5d-4e18-9c40-6b2a1f8e0d33" \
  -H "content-type: application/json" \
  -d '{"model": "...", "max_tokens": 1024, "messages": [...]}'

The implementation detail that decides whether any of this works is where the key is generated.

import uuid

def answer(prompt: str) -> str:
    # Generated ONCE, outside the retry loop: one logical operation,
    # one key, however many attempts it takes.
    key = str(uuid.uuid4())
    for attempt in range(3):
        try:
            return call_provider(prompt, idempotency_key=key)
        except (TimeoutError, ProviderOverloaded):
            time.sleep(backoff(attempt))
    raise ExhaustedRetries()

Moving that uuid4() call one line down, inside the loop, is the single most common way this is got wrong, and it produces a system that looks correct in code review, passes every test, and provides no deduplication whatsoever, because each attempt presents a key the provider has never seen. A related failure is deriving the key from the request body: two genuinely separate requests with identical content — a user asking the same question twice on purpose — then collide, and the second silently receives the first one's answer. The key identifies an operation in your application's terms, so a request ID, a job ID, or a (conversation_id, turn_index) pair are all better sources than a hash of the payload.

Both of those mistakes are the same requirement broken from opposite ends. A key has to be stable across every attempt at one operation, and distinct across different operations. Generating it inside the loop satisfies the second and destroys the first, so every attempt looks new. Deriving it from the body satisfies the first and destroys the second, so two different operations look identical. Reading the pair together tells you where a key can legitimately come from: it must be a name for something in your application that exists before the first attempt is made and does not change while the operation is being attempted. A request ID and a (conversation_id, turn_index) pair both fit that description. A hash of the payload does not, because the payload describes what is being asked rather than which asking this is.

The reason this defect survives review is that nothing observable goes wrong. The header is present on every request, the value is a well-formed identifier, the provider accepts it, and the responses are correct. Deduplication is a property that only shows itself when a duplicate arrives, and a test suite almost never sends one. The failure is silent by construction, which makes the placement of a single line the thing to look at rather than the presence of the feature.

Two limits are worth being blunt about. Retention windows are finite, typically around 24 hours, so a replay from a dead-letter queue processed three days later is a fresh operation as far as the provider is concerned. The window is finite for a reason that is not going to change: recognising a duplicate means remembering the original, remembering costs storage in proportion to traffic multiplied by how long each record is kept, and traffic is not something the provider controls. A window bounds that product. The consequence for a caller is worth stating in one line, because it is the shape of every dedup system ever built — protection lasts as long as the memory does, and a replay that outlives the memory is not a replay any more.

Support for the header is also uneven across LLM APIs — some document it, some do not — so check the provider's reference before designing around it. Where the provider does not offer one, deduplication has to live in something you control: a gateway, or a small table keyed on your operation ID that records "attempt started" before the call and the outcome after it, so a replay can consult it. That table is weaker than a provider-side key, since it cannot tell you whether a request that vanished was executed, and it is enough to stop your own layer from re-submitting work it already has an answer for.

The same trap with the stream half-delivered#

Streaming does not remove the ambiguity, it relocates it. A stream that fails at token 400 with a 529 has already delivered 400 tokens to your process, and probably to the user's screen, and those tokens are billed. A generic retry wrapper that catches the exception, waits, and re-issues the same request produces a second generation starting from token zero, and the consumer — never told that the first attempt was being abandoned — appends the new output to the old.

The row that breaks the pattern is the third, and the reason a key does not help is that the duplicate the provider would recognise is a duplicate of a request that already partially succeeded from your point of view. What the retry needs is a decision at the consumer, made explicitly and owned by one layer. Restarting means clearing what was rendered and beginning again, which costs the 400 tokens twice and gives a coherent answer. Resuming means feeding the received text back as a prefilled assistant turn and asking for a continuation, which pays for the received tokens once as output and again as input, and produces a seam where the two halves join. Either is defensible; what is not defensible is a wrapper that re-issues the call without telling the consumer anything happened, because the consumer's state is then the concatenation of two different generations.

What makes the streaming case structurally different from the timeout case is where the escaped work ended up. On a non-streaming timeout, everything the first attempt produced is on the provider's side of the connection, which is exactly where an idempotency key has authority. Mid-stream, part of the first attempt has already been handed to your process, and in all likelihood painted onto a screen. A key can make the provider decline to generate a second answer. It has no opinion at all about the four hundred tokens sitting in a browser, because it never saw them leave. The instrument stops at the boundary of the system that implements it, and this is the first place in the chapter where the boundary is crossed.

The practical form of this is that streaming retries cannot live in the same generic decorator as request-response retries. The retry has to be a callback the consumer participates in — on_retry(discard_partial=True) or an explicit resume path — and the exception has to carry how much was already delivered, so the decision has something to be made on.

Effects that no key can recall#

The hardest case is the one where the first attempt did something outside your system before it failed. An agent loop that calls a send_invoice_email tool, takes a 529 on the following model call, and is retried from the top will send the email twice, and there is no header you can add to the provider request that unsends the first one. The provider's idempotency key protects the provider's operation; the tool executed in your process, against your credentials, on someone else's inbox.

Safety here comes from making the effect itself replay-tolerant rather than from making the call replay-tolerant, and two mechanisms are worth having together. The first is to give every side-effecting tool its own idempotency key derived from the agent run — most payment and messaging APIs accept one — so that a second charge_card with the same key is recognised by the payment processor rather than by you. The second is an intent ledger: before executing a tool, write a row recording the run ID, the tool name, and a hash of its arguments; after it returns, write the outcome. A replay then consults the ledger before acting.

def execute_tool(run_id: str, name: str, args: dict):
    fingerprint = f"{run_id}:{name}:{stable_hash(args)}"
    prior = ledger.get(fingerprint)
    if prior and prior.status == "completed":
        return prior.result          # already done; do not repeat
    if prior and prior.status == "started":
        raise NeedsReconciliation(fingerprint)   # outcome unknown — ask
    ledger.start(fingerprint)
    result = TOOLS[name](**args, idempotency_key=fingerprint)
    ledger.complete(fingerprint, result)
    return result

The order of the two writes in that function is the part worth arguing about, because both orderings lose something and only one of them loses it visibly. Recording the intent first and the outcome second means a crash in between leaves a row that says "started" with no result — which is a known unknown, and the function raises on it. Recording only after the tool returns means a crash between the effect and the write leaves no row at all, so the replay sees a clean slate and sends the email again, confidently. The first ordering converts an invisible double-send into an explicit question. That is the whole of what a write-ahead record buys, in this system and in every other one: not fewer failures, but failures that leave a trace of themselves.

The started-without-completed case is the interesting one, and it is deliberately not automatic. A row in that state means the process died between issuing the effect and recording it, so the system genuinely does not know whether the email went out, and the only way to find out is to ask the system that would know — query the mail provider for a message with that fingerprint in its metadata, query the payment processor for a charge with that key. Reconciliation is a query against the downstream system, and designing it in means making sure every side-effecting call carries an identifier you can search on afterwards.

None of this delivers exactly-once execution, which does not exist across a network boundary. What it delivers is at-least-once delivery — every operation is attempted until you know it happened, accepting that it may happen more than once — paired with idempotent effects, meaning effects that leave the same state behind however many times they are applied. From the outside that is indistinguishable from exactly-once, and the distinction matters because it says where the work goes: into making each effect safe to repeat, rather than into trying to guarantee it is never repeated.

A retry, then, is never simply another attempt. It is a second execution of an operation whose first execution may have completed, may still be running, and may already have spent money or sent mail on your behalf. An idempotency key generated once per logical operation resolves that for the provider's own work and costs a single header. Everything downstream of the provider — rendered tokens, dispatched emails, charged cards — needs its own identifier and its own ledger, because the provider cannot recognise a duplicate of something it never did.

Assembled, the client this topic describes makes six decisions per call: classify the failure against the documented error type, check the retry budget before spending an attempt, choose the delay from the reported reset where there is one and a jittered exponential schedule where there is not, bound the call with the kind of deadline its path needs, consult the breaker before attempting anything at all, and carry a stable key so a repeat is recognised as a repeat. Wire them in that order, then instrument attempts rather than calls — a counter of provider HTTP attempts labelled with error type and outcome, divided by your own logical call counter — because that single ratio is what says the policy is behaving as designed before an invoice does.

The labels are what make that ratio legible, and the arithmetic in this chapter is the reason. A defect affecting four requests in a thousand, each of them tripling its attempt count, moves the overall ratio by well under a per cent — a change no dashboard threshold would ever be set to catch, and one that a week's variation in traffic mix would hide completely. The same ratio computed only over attempts that ended in a timeout would have read three-to-one from the first day. The aggregate answers whether the client is healthy on average, which is not a question anybody has. The per-outcome slice answers where the attempts are going, and that is the one that would have found this a month earlier than finance did.