Silence is not completion while an obligation is open

run_9271 was declared complete at 14:02:11 and the customer received a summary of their contract bundle four minutes later. The summary has no indemnity section, and the bundle has one, on pages 44 through 51 of the third document. Nothing in the run failed. The ledger for run_9271 has fourteen items and all fourteen say done. The completion check that fired at 14:02:11 was correct about the state it read.

Here is what happened between 13:58 and 14:02.

13:58:39.120  extractor-1  finished clause extraction for item 9 (bundle_b3, pp.40-88)
13:58:39.204  extractor-1  UPDATE ledger SET status='done', owner=NULL WHERE item=9  -> committed
13:58:39.310  extractor-1  PUT s3://artifacts/run_9271/item-9.json  -> in flight
13:58:40.887  ec2          spot reclamation, instance i-0a3f terminated
                           (upload never completed; no object at that key)
14:01:52.640  reconciler   read items 1..14, all status='done'
14:01:52.700  reconciler   GET s3://artifacts/run_9271/item-9.json -> 404, skipped
14:02:11.005  writer       run marked complete, 13 of 14 inputs merged

The item was marked complete 106 milliseconds before the work it describes became durable, and the machine died inside that window. The ledger is not wrong about anything it was asked; it was asked the wrong question.

The check that looks like it needs no coordination#

The completion rule in this system is one line, and its appeal is that any agent or observer can evaluate it at any time without talking to anybody.

def is_done(run_id, ledger):
    return not ledger.exists(run_id, status__in=("pending", "in_progress"))

That property — no coordination required — is real, and it is why the rule survives design review. There is no leader to elect, no barrier to synchronise on, no two-phase anything. Each peer agent polls the ledger, sees no outstanding items, and concludes the run is finished. In a system with no single controller, which is the situation this topic is about, a check that any participant can perform alone is worth a great deal.

The problem is that the rule reads completion off the state that failing agents are least able to keep accurate. Every status field in the ledger was written by an agent, at some moment of that agent's choosing, describing work happening outside the ledger. A crash puts the agent's story and the world out of step, and which way they fall out of step depends entirely on where in the sequence the crash landed.

Three of the four rows are wrong, and they are wrong in two different directions. The rows that hang are annoying and visible: run_9288 from the same week sat with item 12 stuck at in_progress under a dead owner until the thirty-minute run deadline killed it, which cost thirty minutes of a worker slot and produced a support ticket. The row that reports done early is the one that produced a customer-facing defect nobody noticed for four days, because a run that finishes and returns something looks exactly like a run that worked.

Given the choice, the failure worth preferring is the one that hangs. A run that never reports complete is caught by a budget, appears in a dashboard as an incomplete run, and can be resumed from the ledger. A run that reports complete with a missing artifact leaves no trace anywhere in the system, propagates into whatever consumes the output, and is discovered by a human reading the result. Biasing the design toward "refuse to declare done" costs latency and worker slots; biasing it toward "declare done optimistically" costs correctness, and correctness failures in this class are silent by construction.

Making the transitions carry their own evidence#

The two crash windows close differently, and both fixes are ordinary systems engineering rather than anything specific to agents.

The premature done closes by making the transition durable in the same sense the work is. The artifact has to exist before the status changes, and the status change has to record where the artifact is so that any later reader can check rather than trust — the pointer-and-verification pattern from 11.12, applied here to a completion decision rather than a duplicate-work decision.

def finish_item(store, ledger, run_id, item, payload, fence):
    body = json.dumps(payload).encode()
    put = store.put_object(                      # durable before any status change
        key=f"artifacts/{run_id}/item-{item}.json",
        body=body,
    )
    ledger.transition(                           # single durable commit
        run_id=run_id, item=item,
        expect_status="in_progress", expect_owner=fence.agent_id,
        expect_lease_epoch=fence.epoch,          # rejects a resurrected zombie
        set_status="done",
        set_output={"uri": put.uri, "bytes": len(body),
                    "etag": put.etag, "at": put.written_at},
    )

If the process dies after put_object and before transition, the item stays in_progress, which is the row that hangs rather than the row that lies. If it dies after transition, the artifact is already durable and the pointer resolves. The window where the ledger claims something the world cannot back up has been removed, at the cost of turning some crashes into redundant work. Re-extracting one bundle costs roughly 40 seconds and 9,000 tokens here, which is a price worth paying a few times a day to eliminate a class of silent wrong answers.

The in_progress orphan closes with claim expiry. A claim is not a status, it is a lease with an owner, an epoch and an expiry, and a claim whose expiry has passed is not evidence that anyone is working.

-- runs every 30s; reclaims items whose holder stopped renewing
UPDATE ledger_items
   SET status = 'pending',
       owner = NULL,
       lease_epoch = lease_epoch + 1,
       attempts = attempts + 1,
       last_error = 'lease_expired'
 WHERE run_id = $1
   AND status = 'in_progress'
   AND lease_expires_at < now()
   AND attempts < 3;

The lease_epoch bump is what makes the fencing check in finish_item meaningful — fencing being the practice of stamping every claim with a number that increases each time the claim changes hands, so a write can be rejected by comparing the number the writer carries against the one the ledger currently holds. An agent that was merely paused — a long garbage-collection stall, a network partition that healed — can come back at 14:07 believing it still owns item 9 and try to commit done. Its epoch is stale, the conditional transition rejects the write, and the work done by the replacement holder stands. Without the epoch, reclaiming a lease creates a second writer for the same item and the ledger acquires exactly the duplicate-completion problem the lease was introduced to prevent.

attempts < 3 matters too, because an item that fails deterministically — a corrupt PDF that crashes the OCR client — would otherwise be reclaimed forever, and the run would never reach a state with no in_progress items. After three attempts the item goes to failed_permanent, which is a terminal status. A completion rule that only accepts done as terminal cannot make progress past a genuinely unprocessable input, so the set of terminal statuses has to include failure, and the run's outcome has to be able to say "complete with 13 of 14 items, item 9 failed permanently, reason: OCR client abort".

Why none of this is a transaction#

The pairing of an object store write and a ledger commit is two independent durable systems being updated one after the other, and no ordering of those two writes makes them atomic. What the ordering buys is a choice about which inconsistency you get: write the artifact first and a crash leaves an orphaned object with no row, which is garbage and costs storage; write the row first and a crash leaves a row with no object, which is a lie and costs correctness. Choosing the orphan is choosing the cheaper failure, and the periodic sweep that deletes objects under a run_id with no matching done row costs a few cents a month.

The alternative that people reach for is a distributed transaction across the ledger and the store — a protocol, usually two-phase commit, in which a coordinator asks every participant to prepare a write, and then, once all of them have agreed, tells them all to commit or all to abort — and it is the wrong tool twice over. It needs a coordinator, which is the single controller this topic has assumed away, and a coordinator failure between prepare and commit blocks every participant holding a prepared write — the run stops in a state that requires operator intervention rather than one that expires on its own. More decisively, the interesting side effects in an agent run are not rollbackable at all. send_contract_summary puts an email in someone's inbox, create_jira_ticket puts a ticket in a project, and the OCR service has already been billed for the pages. A protocol that can undo a row and cannot undo an email is not giving you the property its name implies.

What the design gives instead is at-least-once execution — every unit of work runs one or more times and never zero times, so a crash costs repetition rather than a hole — with a deterministic key, which is enough because the artifact path artifacts/{run_id}/item-{item}.json is a pure function of the identifiers. Two agents that both extract item 9 write byte-equivalent content to the same key, and the second write is harmless. Tools with external effects need the same treatment explicitly: the claim's (item, lease_epoch) pair becomes an idempotency key — an identifier the receiving service records, so that a request arriving twice under the same key is performed once — carried into the tool call, so the second attempt at sending the summary is deduplicated by the mail service rather than by hope. Exactly-once delivery is not on offer from any message system, and exactly-once effect per item is achievable, one tool at a time, by whoever is willing to plumb the key through.

What this still does not establish#

With durable transitions and expiring claims, the ledger check has become sound about one thing: every unit of work that somebody claimed has reached a terminal state, and each done has an artifact behind it that a reader can verify. That is a real guarantee and it is narrower than "the run is finished".

The gap is that the ledger only knows about work someone already wrote down. Consider run_9310 at 15:22:03. No agent is executing. Every ledger item is terminal. Two seconds earlier, the reconciler sent a message to the writer asking for a revised summary because the merged clause table changed, and that message is sitting in the broker, not yet delivered. The writer is idle because it has not received anything. The reconciler is idle because it is waiting for a reply. Nobody has a claim, nothing is in_progress, and is_done returns true. The message is an obligation the run has not discharged, and it is invisible to a check that looks only at rows.

The same shape shows up without any messaging at all. An agent that dispatched a tool call taking four minutes may be represented in the ledger as holding an item, or may have already released it and be waiting on a callback that will arrive at 15:24. A partial result — extractor-2 wrote three of five sections and then hit its own budget — is a done row pointing at a real artifact that is incomplete in a way the pointer cannot express. Each of these is a live commitment inside the run that the ledger does not count.

So the conditions worth requiring before declaring a run complete are more than one predicate, and they have to be stated together: every item terminal with a verified output for each done, no agent currently executing, no claim outstanding, and nothing dispatched that has not been acknowledged. The first three are what this chapter built. The fourth one is a different kind of thing, since it is a statement about messages rather than about rows, and it cannot be evaluated by any single observer looking at shared state — the moment you sample it, the thing you are trying to observe may be on the wire between two agents you already looked at. Turning that into something countable is the next chapter.