Durable execution

Read these first

• Error recovery
• Concurrency and throughput

What you will know

• Checkpoint an agent so it survives a process restart

  Persist the workflow state and required inputs atomically enough that a restarted worker does not observe a half-written checkpoint or stale execution lease.

• Resume a run from the middle rather than the start

  The recovery boundary must identify completed work and replay only the remainder, including handling a crash after an external call but before its checkpoint.

• Pause for a human and resume days later

  The waiting state needs durable timers, authorization context, and an expiration or cancellation policy rather than relying on a live process or in-memory callback.

• Build a step that remains correct when its execution is delivered twice

  Retries and crash recovery can duplicate delivery, so external writes need idempotency keys, deduplication, or a compensating design.

• Choose between a durable engine and a queue, with reasons

  A queue alone transports work but leaves workflow state, timers, replay, and coordination to application code; an engine adds those guarantees and their operational cost.

Chapters

1. 1

   A checkpoint records execution position, not just data

   Restarting a worker is only survivable if something on disk says which step completed and what it produced. This chapter builds that record and then attacks it: a checkpoint written separately from the execution lease lets a restarted worker read half a state, or lets two workers believe they own the same run. Atomicity between state and lease is the property that makes everything later in this topic possible.

   Read chapter
2. 2

   Recomputing a generated id on resume invents a second reality

   A run that minted an order number, stamped a timestamp, or sampled a model response has already told the outside world about those values, so regenerating them after a restart produces a workflow that disagrees with the systems it talked to. The rule is that anything nondeterministic the run acted on is captured in the checkpoint and replayed from it. Deciding what counts is most of the work.

   Read chapter
3. 3

   Resuming replays the remainder, not the run from step one

   Recovery starts by asking which work is already done, and the answer has to come from the checkpoint rather than from re-running steps to see what happens. You will define the recovery boundary for a multi-step agent, mark completed steps so replay skips them, and handle the awkward case the boundary always exposes: a step that finished externally while its record did not.

   Read chapter
4. 4

   No ordering of call and checkpoint survives a crash between them

   Write the checkpoint after the payment and a crash duplicates the charge; write it before and the same crash skips a charge nobody notices. The gap is not closable by reordering, so correctness moves into the call itself: an idempotency key derived from the run and step, plus a record of the completed effect that the resume path consults before acting. The re-sent customer email is the same bug without the money, and it is also the case where neither route is available — a gateway that accepts anything and issues no request id — which leaves the third design: detect the duplicate afterwards, issue the reversal, reconcile your effect ledger against the external system on a schedule, and accept that the compensating call is itself an external write with this same problem one level down.

   Read chapter
5. 5

   A three-day approval cannot live in a process or a callback

   An in-memory promise waiting on a human dies with the next deploy, and so does the context that made the pending action meaningful. A durable pause persists the proposed action, a timer that fires without anyone holding it open, and the authorisation under which the run will continue days later. It also needs an answer for the approval that never comes: expiry, escalation, or cancellation, decided up front.

   Read chapter
6. 6

   A queue transports work; it never remembers where you were

   Queues deliver messages and retry them, and that is the whole of what they promise. Execution position, workflow state, durable timers, replay semantics and coordination between steps are yours to build on top, which is precisely the list the previous five chapters wrote by hand. Seeing the boundary this way tells you what a durable engine is actually selling, and which of your own modules it would replace.

   Read chapter
7. 7

   A durable engine replays your code from an event history

   The engine is not your checkpoint table with a nicer API. It appends every completed activity to a run history and, on recovery, re-executes the workflow function from the top, feeding the recorded results back where the calls previously blocked — which is how execution position survives a worker that no longer exists, and it is the thing the queue never does. The same mechanism is the bill: your workflow code has to be deterministic, not merely your recorded values, and changing the sequence of steps breaks runs already in flight, which is where versioning rules come from. Read the next chapter's cost list against this machine rather than against your own.

   Read chapter
8. 8

   Durability guarantees you never exercise still cost you daily

   An agent whose longest run is forty seconds, with no human pauses and no external writes worth deduplicating, buys deployment topology, versioning rules, determinism constraints on its own code and a new failure surface in exchange for guarantees it never uses. This chapter argues that case properly and then states the conditions that reverse it, so the decision rests on a requirement rather than on anticipation.

   Read chapter