A durable engine replays your code from an event history

The demo that convinces people takes about ninety seconds. Start a worker, start a run, wait until the log shows the credit check has been dispatched, and kill -9 the worker process. Start a second worker on a different machine, one that has never seen this run and shares no memory, disk or network state with the first. Within a few seconds it prints the credit score and moves on to the charge, and the local variables in the workflow function — the supplier record fetched two steps earlier, the loop index over the line items, the order reference minted at the top — are all present and correct. No code in the workflow read a checkpoint or asked where it was.

That is a different guarantee from the one chapters 1 through 5 built, and it is worth understanding the mechanism before deciding whether to buy it, because the mechanism is also the entire cost. Nothing about the demo is a trick, and nothing about it is free either; every constraint in the second half of this chapter is a consequence of what makes the first half work.

Re-execution is how the local variables come back#

The engine stores a run as an append-only history of events. An activity, in this vocabulary, is one unit of real work — a single API call, a model request, a database write — that runs outside the replayed function and whose result is recorded when it returns. When the workflow function calls an activity, the SDK does not perform the call; it emits a command to the service, which appends ActivityTaskScheduled to the history and dispatches the activity to a worker. When the activity returns, the service appends ActivityTaskCompleted with the result and schedules a workflow task, meaning an instruction to some worker to advance the function as far as the history now allows, and a worker resumes the function with that result.

 1 WorkflowExecutionStarted        {order_ref: null, buyer: acct_5512}
 2 WorkflowTaskScheduled
 3 WorkflowTaskStarted
 4 WorkflowTaskCompleted
 5 ActivityTaskScheduled           resolve_supplier
 6 ActivityTaskStarted
 7 ActivityTaskCompleted           {supplier_id: sup_881, terms: net30}
 8 WorkflowTaskCompleted
 9 ActivityTaskScheduled           credit_check   (attempt 1)
10 ActivityTaskStarted
-- worker killed here; nothing below existed yet --

One feature of that listing explains a cost that arrives later. Every activity contributes more than one event: the schedule, the start, the completion, and the workflow task triple around the point where the function is advanced. So the event count grows several times faster than the number of real calls, which is why a loop of a few hundred tool calls reaches thousands of events, and why the event ceiling arrives sooner than counting API calls would suggest.

Recovery does not resume a paused function, because there is no paused function anywhere; the process that held it is gone. A new worker fetches the history and re-executes the workflow function from its first line. Every activity call it reaches is matched against the history in order, and instead of dispatching anything, the SDK returns the recorded result immediately. resolve_supplier returns sup_881 in microseconds without touching the supplier's API. The code between the calls runs again — the loop, the string formatting, the branch on order value — and rebuilds every local variable on the way. When execution reaches a call with no corresponding event in the history, replay is over, and from there the run proceeds for real.

The step that makes the demo unmagical is worth stating as a property of the function rather than as a description of the machine. A local variable's value is determined by the code that produced it and the inputs that code consumed. The code is byte-identical on the second worker. The only inputs the function takes from outside itself are its starting argument and the results of activities, and the history holds both. So a workflow function is a pure function of its input and the sequence of activity results: give it the same input and the same results in the same order, and every intermediate value it computes is the same value. The variables are not restored, transferred or serialised. They are recomputed, and recomputation is exact because everything nondeterministic was routed out into activities where the engine could write it down.

That identity is what licenses the position claim. Position is encoded as "the prefix of this function whose calls appear in the history", which requires no cursor, no step names and no data structure of your own — the history's length is the position. The hand-built system in chapter 1 stored a cursor and re-entered at a named step, which requires every step to be a named, top-level unit; the engine's approach places no such constraint on the shape of the code, so a run can be paused halfway through the third iteration of a nested loop inside a conditional and come back exactly there. The residual is that purity is an obligation on your code rather than a property the engine supplies, and the second half of this chapter is the invoice for it.

Replay does not happen on every step, which is worth knowing before the cost estimate goes wrong in the other direction. Workers keep a cache of running workflow instances and the service routes subsequent tasks for a run back to the worker holding it, so the normal path resumes an in-memory coroutine and costs nothing. Full replay happens when that cache misses: a worker restart, an eviction under memory pressure, a run whose worker died, or a run woken after a long timer. Its cost scales with the length of the history and the size of the payloads in it, because replay is a walk over the events with a deserialisation at each one. On a history of 60 events the replay took 4 ms; on a 9,000-event agent loop with large payloads it took 2.1 seconds, most of it deserialising results, and that latency lands on the first task after every deploy.

It is also worth being clear about where your code runs, because the topology surprises people who expect a hosted service to execute their functions. The service stores histories and dispatches tasks; your workers hold all the code and poll the service for work over long-polling gRPC, a request each worker holds open until there is something to hand back, so nothing inbound is exposed and the service never sees your business logic. Scaling means running more workers, and a task queue — a name that a workflow is started under and that a pool of workers polls, so the service knows which fleet holds the code for this run — is the routing key between them. That topology has one property worth valuing explicitly, because it is what makes a hosted engine acceptable to teams who would not send their business logic anywhere. The workers reach out; the service never reaches in. Your code, your prompts and your credentials stay inside your network, and what crosses the boundary is a history of activity names and results. It is also what makes the operational burden real, since a fleet that polls is a fleet you must keep polling: a worker pool that is too small does not error, it simply falls behind, and the symptom is a growing backlog on a queue nobody is watching. That is a second deployment to build, monitor and roll — separate from your API, with its own alarms on task-queue backlog and poller counts — and it is a line item in chapter 8's cost list.

@workflow.defn
class PurchaseOrder:
    def __init__(self) -> None:
        self._decision: Decision | None = None

    @workflow.run
    async def run(self, req: OrderRequest) -> OrderResult:
        supplier = await workflow.execute_activity(
            resolve_supplier, req.buyer_id,
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=RetryPolicy(maximum_attempts=5),
        )
        order_ref = f"PO-{workflow.now().year}-{workflow.uuid4().hex[:6]}"
        quote = await workflow.execute_activity(
            build_quote, args=[req, supplier],
            start_to_close_timeout=timedelta(minutes=5),
        )
        if quote.total_cents > 1_000_000:
            await workflow.wait_condition(
                lambda: self._decision is not None,
                timeout=timedelta(days=3),
            )
            if self._decision is None or not self._decision.approved:
                return OrderResult(status="declined", order_ref=order_ref)
        await workflow.execute_activity(
            charge, args=[order_ref, quote.total_cents],
            start_to_close_timeout=timedelta(seconds=45),
        )
        return OrderResult(status="placed", order_ref=order_ref)

    @workflow.signal
    def approve(self, decision: Decision) -> None:
        self._decision = decision

Three of the previous chapters collapse into that listing, and each collapse is the purity identity doing the work. workflow.now() and workflow.uuid4() are the interception approach from chapter 2: they read from the history on replay, which is how the clock and the generator stop being sources of nondeterminism, so the order reference minted on Monday is the same string on Thursday. The wait_condition with a three-day timeout is chapter 5's pause, stored as a timer in the history rather than a row in a table you sweep; the approval arrives as a signal, meaning a message sent from outside to a named running workflow, which the service durably appends to the history before waking the run. The recovery boundary from chapter 3 has no code at all, because "which work is done" is answered by the history's ordering — a completed activity is an event, and the prefix is the boundary.

What replay demands in return#

The bill is determinism of the code, not merely of the recorded values, and this is the distinction chapter 2 flagged and deferred. It is worth being blunt about why it catches people: a team that has diligently applied chapter 2's rule has every value it acted on written down, and can check that every recorded value matches on replay, and will still stall a run. Matching values is not the condition. The condition is that the function issues the same sequence of commands, and a function can reach a different command sequence with every recorded value identical — one iteration order over a set, one inline feature-flag read, and the second replay schedules a different activity next.

So anything that can vary between the original execution and a replay six days later is a defect. Reading the system clock directly, calling random, iterating a Python set, reading an environment variable that changed, generating a uuid outside the SDK, making an HTTP call from workflow code, or racing two coroutines on wall-clock time will all produce a divergence.

The engine detects the divergence rather than silently corrupting the run, and the check is cheap because it is a comparison of commands rather than of data. As the function replays, each command it emits is matched against the next event of that kind in the history; a mismatch in type or identity is exactly the negation of the purity property, and the workflow task fails with an error naming it:

temporalio.worker.workflow_sandbox: workflow task failed
NondeterminismError: Activity type of scheduled event 'book_shipment' does not
  match activity type of activity command 'notify_buyer' (event sequence 41)
  workflow_id=order-8f3c1a run_id=019a0b3c-… task_queue=orders

That failure blocks the run rather than failing it, which is the behaviour you want and which surprises people the first time. The engine has two artefacts that disagree — a durable history and the code in front of it — and no way to know which is wrong, so discarding the run would throw away work that is recorded and real. Blocking keeps both: the run sits there, retrying the workflow task, until somebody deploys a worker whose code replays the history correctly. A single bad deploy can therefore stall every in-flight run at once, and rolling back the workers unblocks them, so the operational habit is to treat determinism errors as a deploy-level incident rather than a per-run bug.

Versioning follows directly from the same identity, and this is where a table that looks like a list of arbitrary rules becomes a single rule with consequences. Position is a prefix of the command sequence. A history recorded by version 4 of the code is a prefix of version 4's command sequence. For a version 5 worker to replay it, that same history has to be a prefix of version 5's command sequence too — so any edit that changes which commands are issued before the current point breaks replay, and any edit that does not, does not.

Every row is that rule applied. Editing an activity's body changes what the activity does and not which command is issued, so the sequence is untouched. Adding at the end changes only what comes after every existing prefix. Inserting, removing or reordering changes the sequence at a point some histories have already passed. Renaming changes a command's identity, which is what the mismatch above compares. Changing an argument shape changes the command's payload, which the same comparison reads.

The engines provide a branch marker for the unsafe cases — workflow.patched("add-fraud-check") in the Python SDK, which returns true for new runs and false for runs whose history predates the patch — and the branch has to stay in the code until no run in flight is old enough to need it. With a three-day human pause that is a minimum of three days of dead code; with a thirty-day contract workflow it is a month, and the patches accumulate faster than anybody removes them.

The row that saves the most trouble is the first one. Activities are executed once and their results recorded, so the body of an activity is ordinary code you can change freely — which means the design pressure is to keep workflow functions thin and put anything that changes weekly, including all the model calls and prompt edits, inside activities.

A different mechanism for the same word#

Not every system described as durable execution replays code, and the differences are exactly the trade between how much of your code is constrained and how precisely a position can be restored.

LangGraph's checkpointer, documented under LangGraph persistence and checkpointing, saves the graph's state after each node into a thread, and resumption loads the state and re-enters at the next node. That is chapters 1 through 3 as a library: it imposes no determinism requirement on your code, because nothing is re-executed, and in exchange the position it can restore is a node boundary rather than an arbitrary point in a function, and the state must be serialisable in full. The two halves of that sentence are the same trade seen from both ends — restoring by re-execution needs pure code and no serialisation, restoring by loading needs serialisable state and no purity.

DBOS takes a third position, recording each step's output in Postgres tables inside your own database and replaying the remainder from those records, which its write-up Durable Execution for Building Crashproof AI Agents argues is the useful part of the guarantee without a separate cluster to operate. Temporal's own AI cookbook: agentic loop with tool calls shows the replay model applied to exactly the loop that stresses the history limits.

The engine also does not do two things people assume it does, and both assumptions come from the demo being about the wrong subject. It does not make external writes safe: an activity that times out is retried by the engine, the retry is a second delivery to the payment API, and everything in chapter 4 still applies inside the activity body, keys and ledger included. The engine's guarantee is about your execution; the payment provider's is about its own state; nothing has changed about the gap between them. And it does not give exactly-once effects, for the same reason nothing else does.

What it gives is that the run's position, its local state and its pending timers outlive any particular process, which is a guarantee your queue never offered and your checkpoint table offered only at the boundaries you named by hand.

The demo remains persuasive and it is not an argument. A machine with this shape has a running cost — a cluster or a per-action bill, a deployment topology, a determinism rule over your fastest-changing code, and a versioning discipline with in-flight runs on the line — and whether that cost is worth paying depends entirely on whether the runs it protects are long enough, paused often enough, or expensive enough to lose.