An unobservable effect needs a key, a status read or a compensation

The previous chapter left an executor that correctly refuses to retry a non-idempotent call whose outcome is ambiguous, and 213 runs whose final step is a customer notification nobody can account for. Refusing to retry stopped the harm and did not close the accounts, and this chapter is about the machinery that does.

Three mechanisms exist, and each answers the question in a different way: one makes the question harmless, one answers it, and one makes the wrong answer survivable. A tool has one of them, or it has none, and which of those is true has to be a declared property of the tool rather than something discovered at 14:20 on the day it matters.

Why escalating to the planner cannot help#

The obvious first proposal is to route every ambiguous failure to the planner, on the grounds that a planner is the component that decides things and this was a decision nobody had encoded. It is worth working through rather than dismissing, because it was tried against one of the stalled runs and the result is the failure mode of the whole idea in six lines.

{"steps": [
  {"n": 1, "tool": "email.check_delivery",
   "args": {"to": "ar@northwind.example", "since": "2026-02-19T14:20:00Z"},
   "post": {"probe": "result_count", "gte": 0}},
  {"n": 2, "tool": "email.send",
   "args": {"to": "ar@northwind.example", "template": "account_closed"},
   "post": {"probe": "field_present", "path": "$.message_id"}}
]}

The validator rejected it at step one, because email.check_delivery is not a tool. The planner invented it, which is a reasonable thing for a planner to do when it has been asked a question whose answer requires a capability the tool list does not contain — the shape of the answer is right, and the thing that would produce the answer does not exist. Strip the invented step and what remains is email.send again, which is the duplicate the executor had just refused to produce.

The reason is worth stating as a general property rather than as an observation about this attempt. The planner sits behind the same boundary the executor does, and its inputs are the run state and the tool list. The fact that would resolve the ambiguity — whether a message left a provider's queue — is not in the run state, was never in the run state, and is not derivable from anything in it. A component whose entire input is a set of facts cannot produce a fact that is not among them or implied by them, so whatever comes back is either an invented capability or a guess. The planner in fact has fewer facts than the executor did, since it never saw the socket time out.

That is the shape of the problem. The uncertainty is not a decision waiting for a decision-maker; it is missing information, and the only things that produce information are a call that observes the world or a protocol that makes the question unnecessary.

The proposal was tempting because escalation genuinely is the right response to the other kind of failure, and the two look alike from a distance. When the customer record turned out to be archived, the world differed from the plan's assumptions in a way that required a different route to the goal, and finding routes is what the planner is for — it had the goal, the completed work and a concrete obstacle, and it produced customer.restore out of a tool list that contained it. An ambiguous outcome is a different category of problem, because the obstacle is not in the world, it is in the run's knowledge of the world. So the distinguishing question the ambiguous-failure path has to ask first is not how bad the failure was, or which tool failed: it is whether the executor knows what state the system is in. If it does, escalate. If it does not, the escalation has nothing to work with.

Three mechanisms, and what each one demands#

The last row is the one to read first, because it is the one you cannot change by writing more code. The first two mechanisms require something of the far end and cost you a single integration each; the third requires nothing of the far end and costs you a design decision for every operation you own. That is why the third is the fallback rather than the default, and it is also why a team that cannot get either of the first two from a provider ends up doing per-operation work forever.

An idempotency key is the strongest of the three because it removes the question rather than answering it. The provider stores the key with the outcome of the first request and returns that same outcome for any later request carrying it, so the executor's retry produces the original message_id instead of a second message. The mechanics are unremarkable and the two details that matter are both about the key itself.

def idempotency_key(run_id: str, step_n: int, args: dict) -> str:
    payload = json.dumps({"run": run_id, "step": step_n, "args": args},
                         sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(payload.encode()).hexdigest()[:32]

def dispatch(step, args, state):
    key = idempotency_key(state.run_id, step["n"], args)
    state.record_intent(step["n"], step["tool"], args, key)   # durable, before the call
    return TOOLS[step["tool"]].fn(**args, idempotency_key=key)

A key has to satisfy three things at once, and it is worth checking the function against each. It must be stable across every retry of the same step, or the retry is a new request and gets a second message. It must be distinct for a genuinely different send, or two different notifications collapse into one and a customer is never told something. And it must be recoverable after a crash, because a process that dies between dispatching and recording comes back knowing only which step it was on.

Deriving the key rather than generating a random one is what satisfies all three simultaneously. The inputs — the run, the step number and the canonical arguments — are exactly the things that are the same on a retry and different on a different send, so stability and distinctness come from the same expression. Recoverability comes free: a restarted process regenerates the key from inputs it still has and gets the same value, because a hash of the same bytes is the same hash. The sort_keys and compact separators arguments are what make "the same bytes" true, since a dictionary's iteration order and a serialiser's spacing are not properties anybody intended to hash. Random keys held only in memory satisfy stability and distinctness and fail recoverability, which is the common version of this design and loses exactly the case the mechanism exists for.

POST /v1/email HTTP/1.1
Idempotency-Key: 9f2c41ab7e0d4c8ea5b31f7d0c9a2e56
Content-Type: application/json

{"to":"ar@northwind.example","template":"account_closed","account":"acct_3319f"}

HTTP/1.1 200 OK
Idempotent-Replay: true

{"message_id":"m_5518","submitted_at":"2026-02-19T14:20:03Z","status":"sent"}

The Idempotent-Replay header on the response is worth logging even though the executor does not branch on it, because it is the only evidence that the deduplication actually engaged. A provider that silently ignores an unknown header returns a perfectly ordinary 200 with a new message_id, and the run looks identical to a successful replay right up until a customer complains.

A status read is what you use when the provider offers no key. It is an ordinary call — search the provider for a message tagged with the run and step you attached as metadata — and its trap is timing.

def observe(probe, deadline_s=45, interval_s=3):
    end = time.monotonic() + deadline_s
    while time.monotonic() < end:
        hits = probe()                    # idempotent read, retryable on its own errors
        if hits:
            return Observed(hits[0])
        time.sleep(interval_s)
    return Unknown("no record within deadline; index lag not excluded")

The provider's search index lags submission by 2–5 seconds at p95 — the figure that 95% of observations come in under — and by up to 30 seconds while the backend is degraded, which is precisely when you are asking. That sets the deadline rather than leaving it to taste: a deadline shorter than the worst lag guarantees a false negative in exactly the conditions that produced the ambiguity, so 45 seconds is chosen to sit above the degraded figure with margin. The polling interval then fixes the call count, since a 45-second deadline polled every 3 seconds makes at most fifteen probes. On a successful observation the median wait is about 4 seconds, because the p95 lag is a tail figure and most messages index quickly.

A read issued 200 milliseconds after the timeout returns nothing, and nothing is indistinguishable from "never sent" unless you treat it as provisional — which is what the Unknown return value is for, and why it names index lag in its own message rather than reporting a negative.

The read is itself subject to timeouts and gateway errors, and that is survivable in a way the write was not, because a read is idempotent, so a failed observation is retried without consequence. The asymmetry between the two is the entire reason this mechanism works at all: you are trading one call you cannot repeat for a different call you can repeat as often as you like.

The wait is the part to budget for honestly, because it holds the run open. On the close-out queue that is tolerable because the queue is a batch and nothing is waiting on it. In an interactive agent answering a user, a 45-second pause on the last step is the difference between a working feature and an abandoned session, and the design that fits there is to record the step as pending, return, and reconcile out of band — which is a different feature, with a different user-visible contract, rather than a tuning of this one.

Compensation is the fallback for effects that are neither dedupable nor observable, and it changes the question from "did it happen" to "can I make it not matter". A duplicate credit is compensated by a reversing entry of the same amount: the €612.40 credit is followed by a €612.40 debit annotated with the original credit_id, which leaves the balance correct and two rows in the ledger that an auditor can read as a single corrected event. That is a better outcome than a silent duplicate and it is not invisible — the account statement shows both lines, and somebody in accounts receivable will eventually ask about them. A subscription cancelled in error is compensated by reactivation.

Email has no compensation in the strict sense, and the reason generalises to a rule for deciding whether an operation has one at all. The message is in a mailbox; nothing you control reaches into a mailbox; so there is no operation available to you whose effect is the absence of the first one. What exists instead is a correction message, which is a second visible artefact and a worse customer experience than either of the other two mechanisms. The pattern here is the saga from distributed transactions, where each step in a long-running sequence carries a semantic undo, and it inherits that pattern's honest limitation: the undo is a new event in the world rather than a rollback, so it only works where a new event can cancel out an old one in whatever ledger the world keeps.

When there is none of the three#

Some tools have no key, no queryable status and no reversal. A webhook fired to a customer's endpoint is the usual example — you cannot dedupe at their end, you cannot query what they received, and you cannot retract it. For those, the correct outcome to record is that nobody knows.

UNKNOWN has to be a third value alongside succeeded and failed, and keeping it distinct is what makes the previous chapter's recovery sound. That re-planner builds its new plan on the assumption that the completed record is accurate, so consider what each collapse does to it. Fold unknown into failed and the record says the email never went, so the new plan sends it — which is the duplicate the executor declined to produce, arriving one component later. Fold unknown into succeeded and the record says the customer was notified, so the new plan does not send it, and the goal is reported complete on the strength of something nobody checked. Both collapses produce a confident plan; only the honest record produces a plan that addresses the problem.

{"completed": [
   {"n": 5, "tool": "billing.issue_credit", "outcome": "succeeded",
    "result": {"credit_id": "cr_4a90"}},
   {"n": 6, "tool": "email.send", "outcome": "unknown",
    "reason": "read timeout after request left process; no key, no status probe",
    "args": {"to": "ar@northwind.example", "template": "account_closed"}}],
 "remaining": [],
 "goal": "close out acct_3319f and notify the billing contact"}

The plan that comes back for that record does not send the email again. It queues the account for a human to check the provider's dashboard, or it sends a message whose text is correct whether or not the first one arrived. Both are decisions a person should have made in advance, and both become available only because the state said "unknown" rather than picking a side.

For that to be reliable, the registry has to require the declaration rather than hope for it.

Tool(fn=postmark.send, effect="external", idempotent=False,
     retry=Retry(max=0),
     unsafe_reason="delivers a message; a repeat is visible to the customer",
     recovery=Key(header="Idempotency-Key", window_h=24, matches="key only"))

Tool(fn=partner.webhook, effect="external", idempotent=False,
     retry=Retry(max=0),
     unsafe_reason="delivers an event to a third party endpoint",
     recovery=NoneAvailable(owner="platform-integrations",
                            on_ambiguous="escalate_human"))

A startup assertion refuses to load any non-idempotent tool whose recovery is unset, and NoneAvailable requires an owner and an explicit escalation route, so the absence of a mechanism is a reviewed position with a name attached rather than a field somebody forgot. Note what the Key entry records beyond the fact that a key exists: the header name, the window, and the matching rule. Those are the three things the previous chapter said a bare idempotent=True leaves out, and they are here because the mechanism's failure modes are entirely about them.

The limits of all three#

None of this makes the plan transactional. An idempotency key is scoped to one provider, because it is that provider's storage that holds the key and the outcome, so a run that issues a credit in the billing system and emails the customer can still end with the credit applied and the email in an unknown state. No key reconciles across two systems that share nothing. What the mechanisms buy is that each individual effect becomes either repeatable, observable or reversible, which is exactly enough to build the forward-only recovery of chapter two on — a route from wherever the world is to the goal — and is not enough to give you all-or-nothing.

The window on a key is a real expiry rather than a formality, and the arithmetic is worth doing before trusting one. A 24-hour dedupe window is comfortable for a retry seconds later. It is useless for a run that was parked in a human queue on Friday and resumed on Monday, because by then the provider has forgotten the key, the request is new, and the customer gets a second notice from a mechanism installed to prevent exactly that. So the rule is not "use keys" but a conditional one: any recovery path that can pause for longer than the window has to treat resumption as a fresh decision and check status rather than replay the key. A pause that a human controls is unbounded by definition, which means every human-approval step in a plan sits on the wrong side of that condition.

The industry's answer to a lot of this now lives below the agent rather than inside it. Durable execution runtimes persist each step's outcome to a log and replay deterministically after a crash, which removes a real class of unknowns — the ones caused by your own process dying between the call and the record. Match that against the taxonomy above and it covers the recoverability requirement of an idempotency key rather beautifully, and it covers nothing else, because the ambiguity this chapter is about is created by a provider that accepted a request and never answered. No amount of durability on your side observes what happened on theirs. That distinction is worth holding onto when a platform is proposed as the fix, since the platform is genuinely useful and the thing it is being proposed for is on the far side of a network boundary it does not cross.

Everything from the first chapter to here is machinery: a schema, a validator, a registry with two extra fields per tool, a recovery path with a budget, and a re-planning call that reads state. It is a substantial amount of apparatus to carry, and none of the preceding chapters has established what it returns. The comparison that settles that is against the design this one replaces, on the same tasks, and it is not decided by the average run.