No ordering of call and checkpoint survives a crash between them

Forty-one buyers were charged twice on 8 August, for a median of €4,300 and a total of €181,000. The refunds went out within a day, the card network fees on the reversals cost €287, and two buyers had their accounts overdrawn by the duplicate and charged their own bank's fee, which the finance team paid back with an apology. The engineering cause is two lines of code in the order of execution: the run called the payment API, the pod died, and the checkpoint that would have recorded the charge was never written.

The chapter before this one named that situation rather than papering over it: the step is in doubt, not done and not untouched. What the boundary cannot do is decide, because deciding requires information from the payment provider that the run has to arrange for in advance. This chapter is about arranging it.

The reordering that hides a worse bug#

The first fix proposed in an incident review was to move the checkpoint above the call. It is the natural instinct and it makes the incident disappear from the dashboard, because a run that records its intent and then dies will not charge twice on resume.

It also makes a worse incident invisible. The run now records that it charged, dies before the call goes out, resumes, skips the payment step because the checkpoint says it is done, books the shipment, and sends the buyer a confirmation. The pallet leaves the warehouse and no money arrives. There is no alert for that, because nothing failed; the discrepancy surfaces in month-end reconciliation, six weeks later, in a spreadsheet.

The two outcomes are not equally visible, and that asymmetry is why the reordering is tempting. Duplicate charges are counted — by the card network, by the refund queue, by the customers who ring up — so a change that drives that count to zero looks like a fix on every instrument anyone is watching. Silent omissions are counted by nothing until a reconciliation runs. The measured quantity goes to zero while the unmeasured one rises, and nobody sees the trade because only one side has a meter on it.

Both orderings have the same structure: a durable write in your database and a state change in somebody else's, with no transaction spanning them. Two writes to two independent stores have exactly one interval between them, whichever order they are performed in, so the number of crash windows is one for any ordering — there is no third permutation of two operations to look for. Whichever you put first, the crash between them leaves your record and their record disagreeing, and the only difference is the direction of the disagreement. Charging twice is loud and expensive and gets fixed the same week; skipping a charge is silent and cheap to fix per incident and expensive to detect.

What that identity licenses is a decision to stop searching for a better order and to move correctness somewhere else. Its residual is that the two orderings are not therefore equivalent, since they differ in what landing in the window costs, and a team that has not yet built the machinery below still has to choose which failure it would rather have in the meantime.

Two things the call needs#

The first is an idempotency key that a resumed run can reproduce exactly, and "exactly" is the constraint that decides its ingredients. A key works by letting the provider recognise a second request as a repeat of the first, which requires the second request to carry the identical string. A resumed run has access to two things and no others: the run's recorded input and its checkpoint. Any ingredient outside that set cannot be reproduced, and a key that differs by one character is a new charge rather than a repeat. So derive it from facts that survive a restart — the run id and the step name, plus a sequence number if a step legitimately performs more than one effect — and never from a timestamp, a random value or an attempt counter, because those change on resume.

def effect_key(run_id: str, step: str, seq: int = 0) -> str:
    return f"{run_id}:{step}:{seq}"        # run_8f3c1a:charge:0

POST /v1/payment_intents HTTP/1.1
Host: api.payments.example
Idempotency-Key: run_8f3c1a:charge:0
Content-Type: application/x-www-form-urlencoded

amount=1420000&currency=eur&customer=acct_5512&confirm=true
&metadata[run_id]=run_8f3c1a&metadata[step]=charge&metadata[order_ref]=PO-2026-0041

The provider stores the key with the response it produced. A second request carrying the same key returns the original response instead of creating a second charge, byte-identical in the body, and some gateways flag the replay with a response header such as Idempotent-Replayed: true.

Sending the same key with different parameters is an error rather than a silent overwrite, which matters more than it sounds, and the reason connects directly to the previous chapter. A run that pinned no price and recomputes the amount from a supplier feed that moved overnight will resume with the same key and a different amount. Without the check, the provider would have to choose between the two amounts silently. With it, the mismatch is refused, and a wrong charge becomes a 4xx that a human reads.

The seq argument exists because some steps genuinely perform more than one external write, and a single key per step forces those writes to share a guarantee they should not share. Splitting an order across two suppliers means two payment intents from one step, and they need :charge:0 and :charge:1; a partial capture — taking part of an authorised amount now and the remainder when the rest of the order ships — followed by a second capture needs the same.

The sequence has to be derived from the step's recorded inputs rather than from a counter that increments as the loop runs, and the failure of the counter version is worth following. A resumed step does not start its loop where the original did; it starts at whatever boundary was recorded. A resumed loop must arrive at the same numbering it used the first time, and a counter that begins at zero on entry instead assigns :charge:0 to a supplier that got :charge:1, and the provider sees two keys it has already honoured attached to two amounts it has not. Ordering the suppliers by their id and using the index is the version that survives replay, because a sort of recorded data is a pure function of recorded data; using the order in which the API returned them does not, because that order is a fact about a response nobody wrote down.

The metadata block is the second half of the design and is often left out. It is what makes the effect findable when the key alone is not enough, because a lookup by metadata[run_id] answers the in-doubt question directly: does a payment intent exist for this run and step, and what is its status.

The second thing the call needs is a local record of the effect that the resume path consults before acting. The provider's guarantee covers duplicate requests it receives; it does not tell your run what happened, and it expires. The ledger is a table with a unique constraint on the key, written before the call and updated after it.

CREATE TABLE effect (
  key          text PRIMARY KEY,            -- run_8f3c1a:charge:0
  run_id       text NOT NULL,
  step         text NOT NULL,
  provider     text NOT NULL,
  status       text NOT NULL,               -- attempted | confirmed | abandoned
  external_id  text,                        -- pi_3PqR...
  request      jsonb NOT NULL,
  response     jsonb,
  created_at   timestamptz NOT NULL DEFAULT now(),
  confirmed_at timestamptz
);

The ledger insert and chapter 3's intent commit are the same write and should be one transaction. Keeping them apart would reintroduce the chapter's own subject one level down: two durable writes before the call, with a crash window between them, and a resume that finds an intent with no effect row or an effect row with no intent. In practice the step's intent row and its effect row go in together, guarded by the epoch condition from chapter 1, so a worker that has lost its lease cannot open an effect it is no longer entitled to perform.

Three columns of that table carry more weight than their names suggest. key is the primary key, which is what makes the ledger enforce anything at all: a second insert under the same key fails at the database rather than being caught by application logic that a crash can skip. request holds what was sent, so a resumed run can compare the parameters it is about to use against the parameters the first attempt used, and detect chapter 2's moved-price problem before the provider does. And abandoned is the third status because some effects are neither confirmed nor retryable — a key whose window has expired with no external id and no lookup result is a row that has to be closed by a decision rather than left attempted forever, blocking the run and the reconciliation both.

The resume path reads three possible states and behaves differently in each. No row means the call never left, so the step is genuinely not started. A row marked confirmed means the effect happened and its identifier is known, so the step is done. A row marked attempted is the in-doubt case, and the run resolves it by querying the provider under the recorded key or metadata before it considers issuing anything.

The email is the same bug with no money attached#

The buyer confirmation went out twice on 8 August as well, to 41 people, and nobody filed an incident because a duplicate email costs nothing. It is the identical defect. What was missing is exactly what was missing on the payment: a record of the completed effect that the resume path consults, and a stable key under which the send was made.

Email transports make the first half harder because most of them accept a message and return an identifier of their own choosing, with no way to say "only if you have not already sent this one". The correctness therefore has to sit entirely on your side, and a unique constraint does the work. Insert the row keyed on run_8f3c1a:notify_buyer:0 before calling the transport; if the insert raises a unique violation, another attempt has already been made and this one stops.

What that buys is narrower than the payment case and worth stating precisely rather than being disappointed by later. Amazon SES will still send twice if the crash lands between the insert and the API call, so the residual window is real. What has changed is its size and its shape: it is now bounded by a single request rather than by an entire step, and the failure mode is a duplicate email rather than an unknown state — a resumed run reading its own ledger knows an attempt was made, even though it cannot know whether the attempt landed.

Some transports offer more. A provider that accepts a client-supplied message id and rejects duplicates gives you the payment-style guarantee, and it is worth checking rather than assuming, because the field often exists under a name like x-message-id or client_reference and is documented three pages deep.

When the far end offers nothing#

The third design is what remains when the gateway accepts anything, issues no usable request id, and offers no lookup — which is the normal state of affairs for older bank rails, meaning the batch transfer networks underneath a payment, and for some carrier APIs, and any partner whose integration is a form post. The run cannot prevent the duplicate and cannot resolve the doubt at the moment it matters, so it detects afterwards and repairs.

That means three standing pieces. The ledger stays, because it is the only place where your intent is recorded at all. A reconciliation job compares the ledger against whatever the provider does emit — a settlement file, meaning the periodic statement of the money it actually moved, a nightly report, a webhook stream — and flags effects that appear twice or not at all. And a compensating action exists for each such effect: a reversal, a void, a cancellation of the second booking.

The comparison has three outcomes and each needs a different response, which is why the job is more than a diff. An effect in the ledger that appears once in the provider's report is confirmed, and the row can be closed. An effect that appears twice is a duplicate, and the compensating action runs. An effect in the ledger that appears in no report at all is the silent-omission case from the top of this chapter arriving hours later, and its response is to perform the work rather than to undo it — which means the reconciliation job is also a delivery mechanism, and has to carry the same key discipline as the original call.

The compensating call is an external write with the same crash gap one level down, which looks at first like an infinite regress. It is not, and it terminates by construction rather than by cleverness, which is worth following because it is the only place in this topic where a recursion is closed rather than pushed further. The compensation gets its own key and its own ledger row, and that row is unique, so the compensation is applied at most once however many times it is attempted. If the compensation itself lands in doubt, the run stops and pages a human rather than compensating the compensation. The depth is therefore bounded at two, with a person at the bottom, and the bound comes from a deliberate decision to escalate rather than from any property of the machinery.

Reconciliation latency is the honest cost of the whole design.

The right-hand column of the first row deserves reading twice, because it names the case where the best design still leaves you stuck. Resolving an in-doubt effect requires calling the provider, and the resume is happening after a crash that a provider outage may have caused. If the lookup is down, the run cannot confirm and cannot safely retry, and the only correct move is to leave the ledger row attempted and try again later — which means a resume path needs a way to defer a step rather than only to run it or escalate it. Deferring is cheap to add and easy to forget, and a run that escalates every unreachable lookup will page somebody for every provider blip.

Read the detection-latency column as the thing you are actually buying, because it is the only column where the three rows are orders of magnitude apart. The first two designs resolve doubt inside the resume; the third resolves it on the provider's schedule, and no engineering on your side shortens that.

At-least-once delivery and at-most-once effect#

Idempotency does not give exactly-once effects, and the phrase is worth retiring in this context. Delivery of the attempt is at-least-once, meaning the request may reach the provider more than once and will not reach it zero times, because retries and resumes are how the system survives. What the key buys is at-most-once effect on the far side, meaning the charge is created once or not at all however many requests arrive.

Put those two together and the arithmetic is what produces the result people call exactly-once: at least one attempt, at most one effect, therefore exactly one charge. The two factors come from two different systems — your retry policy supplies the first, the provider's key handling supplies the second — so the guarantee is a property of two systems cooperating rather than of your code alone. Anybody claiming exactly-once from one side of that pair has claimed half of it.

The key also has a lifetime, which is the detail that breaks the design later in this topic. Stripe removes idempotency keys after 24 hours, and other gateways range from a few hours to a week. A run that pauses for a human decision on Monday and resumes on Thursday cannot rely on the key at all, because the provider has forgotten it — the second request is a first request as far as the provider is concerned, and the at-most-once half of the pair has silently gone away. So the resume path must consult the ledger and the provider's lookup instead and treat the key as a defence against fast retries only. That is a concrete reason the next chapter's waiting state has to carry more than a timer.

Finally, none of this removes the need for the boundary in chapter 3. The key makes the replay safe and the ledger makes the doubt resolvable, and both only run because the resume path stopped and asked instead of assuming the step had never happened. The interview version of this argument is short: moving the checkpoint before the call swaps a loud duplicate for a silent omission, and the fix is that the call carries a key the resumed run can reproduce and the run keeps its own record of what it attempted, so replay reaches the same charge instead of a second one.