Idempotency decides whether a retry is safe, not the error

A tool call that fails after the request has left your process leaves two questions on the table, and it is worth keeping them apart from the first sentence. One is did the effect happen, which you cannot answer. The other is what does it cost if I make it happen again, which you can answer, and could have answered a year ago.

An operation is idempotent when performing it a second time leaves the world in the state a single performance would have left it in. customer.update(account_id, status="closed") sets a field to an absolute value, so running it twice leaves the record exactly where one run would have left it. billing.issue_credit(account_id, amount_eur=612.40) adds to a balance, so running it twice produces €1,224.80 of credit. email.send delivers a message, so running it twice delivers two messages. Those are facts about the implementations behind the tool names. They are stable across every error the transport can produce, and they are knowable at the time the tool is written.

This chapter is about wiring that property into the plan format of the previous chapters, so that the retry decision is made once per tool by somebody with the implementation in front of them, rather than during an incident by whoever is on call.

The error tells you what you know; the operation tells you what repeating costs#

A timeout is a statement about your process, saying that no response arrived inside the budget you set. It says nothing about the server. The request may have been dropped by a load balancer before reaching anything, or accepted, queued, and delivered to a mailbox 900 milliseconds after your client gave up. Those two worlds are indistinguishable from where the executor sits, and no amount of error inspection distinguishes them, because the information required is on the other side of a connection that is gone.

Put the two questions together as a condition and the governing rule falls out. Repeating a call is safe when either the effect did not happen, or repeating it is harmless. The first disjunct is exactly what an ambiguous error cannot tell you. The second is a property of the operation you can look up. So on every ambiguous failure the decision has to rest entirely on the second: retry where the operation is idempotent, because there the ambiguity is harmless — you are choosing between one effect and one effect. Do not retry where it is not, because there the ambiguity is the whole problem, and repeating converts an unknown outcome into a known duplicate.

The reflex that this displaces is a policy built out of error classes, usually arriving as "treat timeouts specially and trust the rest". It is a natural instinct, since the timeout is the error that visibly announces its own uncertainty, and it is worth testing against a real distribution rather than dismissing.

Read the third column down and the table splits in two places rather than one. The rows above the first Unknown are the authoritative negatives: the failure resolved before any work began, so the effect provably did not happen, and the last two columns agree because there is nothing to duplicate. Below that line every row carries the same Unknown, and it is the same uncertainty wearing different clothing — which is why the last column has one answer rather than a taxonomy. The final row is the one that catches teams by surprise: a 200 whose body does not parse, or whose status field holds a value the client does not recognise, is as ambiguous as a timeout, and it arrives through the success path where nobody has thought about retries at all.

The distinction has a familiar shape, because HTTP made it long before agents did. GET, PUT and DELETE are specified as idempotent and POST is not, and every client library that retries GET automatically and refuses to retry POST is applying exactly this rule. It is worth knowing where the precedent stops, though, because it is a specification of what a method is supposed to mean rather than a guarantee about a particular server: an endpoint can advertise PUT and implement an append, and the method name will not stop it. What agent executors tend to do instead is attach one retry policy to the loop, so the policy that is right for a subscription read is also applied to a credit issuance. The reason is architectural rather than careless — in a fused loop there is one call site and therefore one obvious place to put the configuration, and a single policy is what a single call site invites.

Making the plan format carry it#

The decision belongs on the tool, written once when the tool is added, and reviewed the way any other interface property is reviewed.

@dataclass(frozen=True)
class Tool:
    fn: Callable
    effect: Literal["read", "write", "money", "external"]
    idempotent: bool
    retry: Retry
    unsafe_reason: str | None = None   # required when idempotent is False

TOOLS = {
  "billing.list_invoices": Tool(
      fn=billing.list_invoices, effect="read", idempotent=True,
      retry=Retry(max=3, base_ms=200)),

  "customer.update": Tool(
      fn=crm.update, effect="write", idempotent=True,
      retry=Retry(max=3, base_ms=200)),          # absolute set, not a delta

  "billing.issue_credit": Tool(
      fn=billing.issue_credit, effect="money", idempotent=False,
      retry=Retry(max=0),
      unsafe_reason="adds to the account balance; a repeat doubles the credit"),

  "email.send": Tool(
      fn=postmark.send, effect="external", idempotent=False,
      retry=Retry(max=0),
      unsafe_reason="delivers a message; a repeat is visible to the customer"),
}

Two fields describe two different things and both are needed. The effect class says how much damage a call can do, which is what the permission policy in chapter one reasons about. Idempotency says what a second identical call costs, which is what the retry decision reasons about.

The reason they cannot be collapsed is that they come apart in both directions, and one example each is enough to prove it. customer.update is a write — it changes state, it needs billing:write scope, it belongs in an approval conversation — and it is safely repeatable, because it sets a field to an absolute value. A read against a paginated endpoint with a cursor that expires on use is the opposite: it changes nothing anybody cares about, it needs only read scope, and calling it twice with the same cursor fails or returns a different page. Since neither field determines the other, one field cannot express both, and collapsing them means either treating every write as unrepeatable — which costs you the transient-failure recovery on the safest write in the registry — or treating every read as free, which is wrong for the cursor.

unsafe_reason is there for a reason that is about process rather than code. It forces whoever adds a tool to articulate the duplicate, in a sentence a reviewer can disagree with, at the moment they have the implementation in front of them. Left optional, it is omitted, and six months later the flag is a bare False that nobody trusts enough to change and nobody understands well enough to keep.

Some tools resist a static flag, and the honest response is to split them rather than to guess. A generic run_sql tool is idempotent when the statement is a select or an update ... set x = 'closed', and is not when it is an insert or an update ... set balance = balance + 612.40, so the property depends on an argument the planner writes. Two designs handle that. Either the registry entry carries a predicate over the arguments instead of a constant, which keeps one tool and moves the decision to dispatch time, or the tool is replaced by narrower ones — warehouse.query, warehouse.upsert, warehouse.append — each with a constant flag. The narrower tools are usually the better trade for two reasons that compound: a predicate over arbitrary SQL is a small parser, and a small parser that has to decide whether a statement mutates has exactly the failure mode of the thing it replaced, only now buried one level deeper; and a closed set of verbs is easier for the planner to select from anyway, which is the same argument the tool field made in the first chapter.

The planner never supplies this. It is a property of the implementation behind the tool name, and asking a model to assert it per step invites a confident "idempotent": true on the email step of the one plan where it matters. What the plan carries is the tool name; the validator resolves the policy from the registry and stamps it onto the step so the trace records which policy was in force.

def bind_policy(step):
    tool = TOOLS[step["tool"]]
    return {**step, "_policy": {"idempotent": tool.idempotent,
                                "retry_max": tool.retry.max,
                                "effect": tool.effect}}

Stamping rather than looking up at the moment of the call is a small decision with a specific payoff. The trace then records the policy that was actually applied, so when the registry is edited later, an old trace still says what the rules were on the day, which is the same discipline the verifier version served in the previous topic.

07:31:12  step 6  email.send(to="ar@northwind.example")
          policy idempotent=false retry_max=0 effect=external
          -> ReadTimeout after 10.0s
          ambiguous outcome, no retry permitted
          step 6 -> outcome=UNKNOWN, run -> recovery

What idempotency does not settle#

The property has a narrow licence and it is worth naming the four things it does not extend to, because each one is a place a team has assumed it did.

It is not safety in the HTTP sense of "does not modify state". DELETE is idempotent and permanently destructive, and customer.archive is idempotent in exactly the same way — archive it twice and it is archived, which is precisely as bad as archiving it once. The flag licenses a repeat; it does not license the call. Deciding whether an operation should happen at all is the permission and approval question of the first chapter, and that gate runs before this one and answers a different question.

It also does not promise the retry will work. Marking billing.list_invoices idempotent means a second call cannot make things worse, which is a statement about the downside and says nothing about the upside. If the warehouse is down, all three attempts fail and the step ends in the same place chapter 2 put it, with the remainder of the plan voided and a re-plan to write. What retry policy buys you is the transient failures — the 503 during a deploy, the 429 under a burst — and the numbers on this queue are ordinary and worth reading as a yield curve: across the close-out queue's 900 monthly runs, retries recover about 1.4% of steps on the first attempt and 0.2% on the second, and the third attempt has never once succeeded. Each attempt clears a fraction of what the previous one left, and the fraction is falling steeply. That is an argument for max=2 that nobody has got round to making, and the registry above still says max=3.

A flag is also a claim, and claims about somebody else's service can be wrong in ways your tests will not reveal. Payment and messaging providers frequently document idempotent behaviour with conditions attached — deduplication that holds for 24 hours and not beyond, or that applies only when the request body is byte-identical, or that covers the charge and not the webhook it triggers. A registry entry that reads idempotent=True because the vendor's documentation used the word, without recording the window and the matching rule, is the same unexamined True the unsafe_reason field exists to prevent, pointing the other way. The version worth writing down names the mechanism: what the provider matches on, for how long, and what happens after the window closes.

The property does not compose across a plan, either, and the reason is the bindings rather than the tools. Six individually idempotent steps do not make the plan idempotent, because step one binds amount_eur from a subscription read, and re-running the whole plan tomorrow reads a different balance. The second execution is therefore a different set of calls that happen to share a shape with the first — same tools, same order, different arguments — and idempotency is a property of a call with its arguments, not of a tool name. Plan-level re-entry is the recovery-point problem from topic 10.5, and it is answered by probing what already happened rather than by any per-tool flag.

The flag says nothing, finally, about what to do with the tools it marks unsafe, and that is where this chapter stops rather than concludes. The February replay ends with 213 runs whose final step has an outcome nobody knows. The executor was right to stop, and the customers still have not been told their accounts are closed. Handing that upward does not obviously help: the planner has the same view of the world the executor had, so it cannot determine whether the message went out either, and a second model call is not a source of facts about a provider's queue. What is missing is a way to find out, or a way to make asking twice harmless.