Resuming replays the remainder, not the run from step one

The recovery boundary is that partition: given a checkpoint, which steps are done, which have not been started, and which were interrupted between the decision to act and the record of having acted. Computing it is the first thing a resumed run does and the only thing that makes the rest of the resume safe, because every later decision — skip this step, run that one, stop and ask — reads off the boundary.

Two things have to be true before the boundary can be computed at all, and the previous chapters supplied both. The run's position has to be recorded atomically alongside its ownership, or the boundary is computed from a state that half-happened. The values the run acted on have to be recorded, or a step the boundary marks as done produces different outputs the second time anybody reads them. What this chapter adds is the decision procedure on top.

The resume path that asked the portal#

The second version of the resume path was clever and it created duplicate purchase orders anyway. Instead of trusting the checkpoint, it asked the world: before submitting an order to the supplier's portal it called GET /orders?ref=PO-2026-0041, and if the portal returned nothing it submitted. This felt safer than reading a local row, because the portal is the system that actually matters.

On 8 August it produced two orders for the same reference inside four seconds, and the portal's support engineer explained why without much sympathy: their read replicas, the copies of the database that serve read traffic, lag writes by up to five seconds under load, and a lookup issued three seconds after a submit returns an empty list roughly one time in twenty.

That failure rate is the general lesson rather than a fact about one vendor. A read served by a replica reports the state of the replica, which is the primary's state as of some moment in the past. An empty result therefore means "this did not exist as of some earlier moment", and the question being asked is "does this exist now". Those are different questions, and the gap between them is the replication lag. So a negative answer from an eventually-consistent read is not evidence of absence — it is evidence of absence-or-lateness, with no way to tell which from the response itself.

Two further problems come with probing, and they are independent of consistency. The check is a different operation from the write, so it is served by different infrastructure with a different failure profile, and it can be wrong in ways the write is not. And it is unavailable exactly when you need it: the resume happens after a crash, and the crash was often caused by the same outage that now makes the probe time out. Recovery has to start from the record the run wrote about itself.

None of that makes asking the far system useless, and the distinction matters later in this chapter. A search for something matching a description is the operation that fails above. A lookup by an identifier your own run generated and recorded before acting is a different operation, usually served by the primary and usually strongly consistent, and it is the only version of "ask the world" the boundary will accept.

Three states, not two#

Most implementations have two buckets, and the missing third is the whole problem. The two-bucket partition is defined by a single question — is there a completed record for this step — and its negative branch merges two situations that need opposite treatment: a step that never began, and a step that may have finished in somebody else's system while its record did not. A step that was marked pending and then had a credit vendor called on its behalf is not untouched, and a resume that treats it as untouched will call the vendor again.

The third bucket only exists if the run writes down its intention before acting, and that is not a stylistic choice — a bucket is a durable distinction, and a durable distinction requires a durable write. So the intent commit is one extra commit per step, transactional with the cursor, saying that this worker at this epoch is about to perform this step and here is the key it will act under — a stable string that names this particular effect to the outside system, so that a repeat attempt can be recognised as the same one rather than as a new request. After the step returns, a second commit records the result and moves the cursor. A crash between the two leaves the durable trace of an intention with no outcome, which is precisely the signal the resume path needs.

The arrow that reads in_doubt --> not_started is safe only when absence is provable, which is a stronger statement than a probe returning nothing — and the section above is why. Chapter 4 is about the machinery that makes it provable for external writes; here the job is to make sure the run reaches that decision point at all, rather than sailing past it with a cheerful if step not in completed.

CREATE TABLE run_step (
  run_id      text    NOT NULL,
  step        text    NOT NULL,
  status      text    NOT NULL CHECK (status IN ('started','completed','failed')),
  attempt     int     NOT NULL DEFAULT 1,
  effect_key  text,
  started_at  timestamptz NOT NULL DEFAULT now(),
  result      jsonb,
  PRIMARY KEY (run_id, step, attempt)
);

Two details of that schema are load-bearing. attempt sits in the primary key, so a step that is started twice produces two rows rather than one row overwritten; the record of what attempt 1 did survives attempt 2, which is what lets an operator see that a step was tried before and what key it used. And the status check admits failed alongside started and completed, which is a fourth situation rather than a synonym for one of the three above: a call that returned an error is observed, so the run knows the far system rejected it. Observed failure and in-doubt are different information states, and the boundary keeps them apart for the same reason chapter 4 will keep a loud duplicate apart from a silent omission.

That table sits beside the single run_state row from chapter 1 rather than replacing it, and the division of labour is worth stating. The row holds one fact per run — where it is and who owns it — and is rewritten constantly, which is why it must stay small. The step table holds one row per attempt at each step and is only ever appended to, which is why it can hold the intent, the key and the result without slowing the lease down. The boundary is computed from the step table; the lease and the cursor stay in the row.

def recovery_boundary(rows, plan):
    done, in_doubt = set(), []
    for r in rows:
        if r.status == "completed":
            done.add(r.step)
        elif r.status == "started":
            in_doubt.append(r)          # intent recorded, outcome unknown
    remainder = [s for s in plan if s.name not in done]
    return done, in_doubt, remainder

Writing the intention doubles the number of commits, which is the honest price of the third bucket. Two conditional updates at 3 to 6 ms each means every step pays 6 to 12 ms of database time, so the nine-step run spends about 90 ms on bookkeeping against six minutes of work. The intent commit is worth skipping only for steps that are provably free of external effect, and the temptation to skip it more widely than that is strong precisely because those are the steps where nothing ever goes wrong in testing. A useful default is that the intent commit is mandatory for any step whose policy is not re_executable, which makes the cost proportional to the risk.

Steps that loop need the same treatment one level down. The quote step walks 240 line items, and if the run dies at item 173 the boundary can only see that quote is in doubt, which sends the resumed run back to item 1 for 47 seconds of lookups it already did. Recording the loop cursor inside the step's own progress record — a single integer, committed every 25 items rather than every item — brings the worst case down to 25 items of rework at a cost of ten extra commits per step.

That batch size is the balance point of two costs that move in opposite directions, and the shape is worth seeing because it recurs wherever a checkpoint has a granularity. Committing every item makes the extra commits proportional to the number of items and the rework zero; committing once makes the extra commits zero and the rework the whole step. Committing every b items puts the number of extra commits at the item count divided by b, which falls as b grows, and the worst-case rework at b items, which rises as b grows. Choosing b is choosing where those two meet, and on this step both measurements came out comfortable at 25: the commit overhead was under 2% of step time and the expected rework was under 5 seconds. Neither number was guessed, and the same two measurements are what you take on any loop before picking a figure.

plan is the ordered list of nine steps, and remainder is what the resumed run will execute. The boundary is computed once at the top of the resume, before any work happens, so the run knows its whole shape before it touches anything. That ordering matters for two reasons. A run that discovers a step is in doubt after it has already executed three later steps has done work it may have to undo. And it matters for reporting: an operator asking why order PO-2026-0041 is stuck gets an answer of the form "six steps complete, charge in doubt since 14:07, three steps remaining", which is a sentence you can act on, rather than a stack trace from a step that failed for the fourth time.

This is not a retry#

Topic 10.6 dealt with retrying a failed step, and the two mechanisms get conflated because both end up calling the same function twice. A retry happens inside a live process with the stack intact: the local variables are still there, the loop counter is real, the backoff sleeps in the same coroutine, and the failure being handled is an error value returned by a call. A resume rebuilds a run in a process that has never seen it, from a durable record, after a failure that returned nothing at all because the process ceased to exist mid-call.

The practical consequences differ in three ways, and the third is the one that costs money. A retry knows its attempt count from a variable; a resume knows it only if chapter 2's rule was applied and the counter was recorded, which is why the run_step table carries attempt in its primary key. A retry can hold a resource across attempts; a resume cannot hold anything.

And a retry's failure is observed, so the code knows the call failed, whereas a resume's failure is unobserved, so the code knows only that a call was started. Those are different information states. "The call returned an error" rules out the far system having accepted the work; "the process disappeared while a call was open" rules out nothing at all. Treating a resume as a retry means substituting the first for the second — treating "unknown" as "failed" — and every duplicate charge in this topic is that substitution.

Replay is not free in itself. Every step lookup is a row read, the boundary computation is one query returning at most a few dozen rows, and the run has to rebuild whatever it holds in memory. That last part is the cost people forget, because it is not measured in milliseconds. A resumed run starts with an empty process: no open HTTP session to the supplier's portal, no partially written temp file, no in-memory cache of the 240 price lookups, no authenticated websocket. Anything the original run held only in a local variable is gone, and the code has to be able to reconstruct it from input and completed or fetch it again.

That has a design consequence beyond performance. Steps that acquired something rather than computed something — a lock at the ERP, a reserved stock allocation, a temporary upload URL — hold a claim that exists in another system and whose handle lived in the dead process. If the acquisition is not modelled as a step with a recorded result, the resumed run will proceed as though it still holds a thing it does not, and the far system will keep holding the claim until it times out.

The boundary also does not make the skip logic automatically correct. if step in done: skip is a decision your own code makes, and it is only sound if the completed marker means what the reader thinks it means. A step marked complete because it wrote a row, whose row was later deleted by a cleanup job, is complete in the checkpoint and absent in the world. A step marked complete after the first of two writes it was supposed to make is a coding bug that the boundary faithfully preserves forever. The record is only as truthful as the commit that wrote it, and the useful habit is to make the commit the last statement of the step and to make the step do one externally meaningful thing.

The one the boundary always exposes#

Every real run eventually produces the in-doubt case: charge was started at 14:07:12, the pod died at 14:07:14, and the checkpoint records an intention with no outcome. The boundary refuses to guess, which is its contribution, and it has to hand that step to something that can decide. Three routes exist and the resume path chooses per step, declared in the plan rather than improvised at recovery time.

The step can be safely re-executable, in which case in-doubt collapses to not-started and the step simply runs. Quoting is like this: it reads prices and writes nothing outside the run, so a second execution leaves the world exactly as a first one did.

The step can be resolvable, meaning there is a way to ask the far system whether the effect exists using a key the run recorded before acting. That is the only version of "ask the world" that works, and the reason is the distinction drawn at the top of this chapter: the key is the run's own, so the query is a lookup by primary key rather than a search, and a lookup by primary key against the system of record is the operation whose negative answer can be trusted.

Or the step can be neither, in which case the run stops and asks a person, and the only thing the boundary owes you is that it stopped rather than proceeding. That third route is a decision rather than a fallback, and it costs something real: a run that escalates holds whatever it was holding — a stock allocation, an authorisation, a quote with a validity window — for as long as the person takes to answer. Choosing it means accepting that cost in exchange for never guessing, and a plan in which every step is escalate is a plan that pages somebody after every deploy.

POLICY = {
    "parse_request":  "re_executable",
    "quote":          "re_executable",
    "credit_check":   "resolvable",     # vendor exposes lookup by our request id
    "charge":         "resolvable",     # idempotency key + effect ledger
    "book_shipment":  "resolvable",
    "notify_buyer":   "escalate",       # no lookup, no key, human decides
}

That table of policies is a design artefact with real teeth, because writing it forces someone to answer, for each external system, whether an effect performed under a known key can be looked up afterwards. Answering it means checking two independent features rather than one, and vendors ship them separately. A provider may accept a client-supplied key without offering any way to search by it, in which case a repeat request is safe and an in-doubt step is still undecidable. Another may offer a rich search with no key support, in which case you can find out what happened and cannot stop a second attempt creating something new. resolvable needs both, and a plan that assumes it from one of them is a plan that fails on the day a worker dies mid-call. On that service the answer was yes for the payment provider and the carrier, no for the email sender, and "sort of" for the credit vendor, whose lookup endpoint existed and lagged by up to a minute — which is the replica problem from the top of this chapter wearing a different hat, and it means the credit vendor's lookup can be trusted after the lag has passed and not before.

Three different answers means three different designs, and building them is what turns resolvable from an aspiration into a guarantee. The boundary has taken the resume from "run everything again" to "run the remainder, and stop at the one step whose outcome is unknown", and it has done so entirely from the run's own record. What it cannot do is decide that one step, because deciding requires something the run has to arrange for before it acts rather than after it crashes.