Two whole-document writes in the same second lose one

The Thursday morning question was why the March compliance report had five sections instead of six. The run it came from, run_9d4a, had finished on Tuesday at 14:23, reported success, emitted no errors, and been signed off. Six agents had been assigned six sections. Five sections appeared. The sixth agent's logs showed it had completed normally, written 4,100 words, and updated the shared artifact at 14:22:07.

The artifact's own history, such as it was, showed something else:

version 61  14:22:07.412  agent-risk      sections: [legal, tax, ops, hr, vendor]
version 62  14:22:07.488  agent-vendor    sections: [legal, tax, ops, hr, risk]

Both writes succeeded. Neither was rejected, neither errored, and both documents are well-formed JSON that validates against the schema. Version 62 is the one the report stage read, and it contains agent-risk's section and not agent-vendor's — or rather, it contains a sections list assembled by agent-vendor from a document it read before agent-risk's write landed, so agent-risk's entry is absent and agent-vendor's own is present. The two agents did not corrupt anything. One of them simply overwrote a change it had never seen.

The mechanism, without the word "race"#

Every agent in this system performs the same three steps, in this order: fetch the whole document, modify its own entry in memory, put the whole document back. That sequence has a gap in the middle, and the length of the gap is the length of the agent's turn.

Both agents read version 60. Both computed a new document from version 60. Agent-vendor's put was written last, so the store's final content is version 60 plus vendor's edit, and agent-risk's edit is gone as completely as if it had never been attempted. The store did exactly what it was asked: it accepted two full-document replacements in the order they arrived.

This anomaly is old and has a name outside agent systems. Relational databases call it a lost update, and it is one of the phenomena isolation levels are defined against; the optimistic solution below dates to Kung and Robinson's 1981 paper on concurrency control without locking, which proposed exactly the pattern of doing the work first and validating at commit time. The reason it feels unfamiliar here is that agent runs make the read-modify-write window enormous. In a database transaction the window is milliseconds. In this pipeline the window is however long the agent's model calls take, which for a section-drafting turn is between two and six minutes. Two agents whose turns overlap at all will read the same version, and in a six-agent run where every turn takes minutes, overlap is the normal case rather than the unlucky one.

The 76-millisecond gap between the two writes is not the point, and chasing it is a common wrong turn. The collision was created at 14:16, when both agents read version 60. Making the writes faster, or adding a random jitter before writing, changes nothing about the fact that both documents were computed from the same stale base.

Making the write conditional on what you read#

The fix addresses the base rather than the timing: an agent tells the store which version it read, and the store refuses the write if the document has moved since. That primitive is compare-and-swap, meaning the store compares the version the writer claims to have read against the version it actually holds and swaps in the new document only when the two match, so an update computed from a stale base is rejected instead of applied. Every durable store worth using for this has it, under a different name.

def apply_update(store, key, mutate, max_attempts=6):
    for attempt in range(max_attempts):
        doc, etag = store.get_with_etag(key)
        new_doc = mutate(doc)                 # pure: no model calls in here
        ok = store.put_if_match(key, new_doc, if_match=etag)
        if ok:
            return new_doc
        time.sleep(0.05 * (2 ** attempt) + random.uniform(0, 0.05))
    raise ConflictExhausted(key)

On S3 that is PUT with an If-Match header carrying the ETag, the opaque version tag the store handed back with the read, which returns 412 when the object has changed since. On Postgres it is one statement whose row count tells you whether you won:

UPDATE runs
   SET doc = $1, version = version + 1, updated_at = now()
 WHERE run_id = $2
   AND version = $3;
-- rowcount 0 means someone else committed between your read and this write

Replaying Tuesday with this in place, agent-vendor's put carries the ETag of version 60, the store is at version 61, the write is rejected, and agent-vendor re-reads and reapplies:

Version 62 now holds six sections, and the whole retry cost about eighty milliseconds.

That number holds only because of the discipline in the comment on the mutate line. The function passed to apply_update must be a pure transformation of the document — insert this section pointer, set this status, clear this owner — and must not contain the four minutes of model work. If an agent's retry re-runs its turn, a conflict costs four minutes and $0.60 of tokens instead of eighty milliseconds and nothing, and under load the system spends most of its budget losing races. Structuring an agent so that the expensive work happens first, produces a value, and is then applied to the document by a small deterministic function is what makes optimistic concurrency affordable here — optimistic in the technical sense that a writer proceeds without asking permission and finds out at commit time whether it was entitled to. It also has a pleasant consequence for chapter 4: that small function is the only thing that ever needs to be retried.

Not every store people reach for offers the primitive, and that is worth checking before the design is settled rather than after. A plain POSIX filesystem gives you an atomic rename — a swap of one file for another that either happens completely or not at all, with no state in between that a reader can observe — but no compare-and-swap on content, so the version check has to be built by hand out of a separate marker file, with all the ways that can itself go wrong. A shared Google Doc or Notion page, which teams occasionally choose because a human can read it, has no conditional write at all from the API's point of view; the last edit wins and the revision history is the only evidence anything was lost. Redis has WATCH/MULTI, DynamoDB has condition expressions, Postgres has the row version above, and object stores have grown If-Match support relatively recently — S3 only gained conditional writes on existing objects in late 2024, which is why older designs on top of it invented their own locking layers. If the store you have cannot reject a write based on what the writer read, the concurrency design is constrained before you start.

Conflict rates in this pipeline, measured over a week at six agents and roughly one write per agent per turn, sit at 4.1% of writes rejected once and 0.2% rejected twice. Nothing reached six attempts. Those numbers rise with the number of writers and with how often each writes, and they are worth putting on a dashboard, because a rising rejection rate is the early signal that the document has become a contention point.

Why the obvious lock makes it worse#

The proposal that comes up first, and which is not wrong in principle, is to add a lock: an agent acquires it before reading, releases it after writing, and only one agent is inside the read-modify-write section at a time. That is a legitimate way to prevent lost updates, and this particular version of it is the one that hurts, because of where the boundaries have been drawn. The critical section here begins at the read and ends at the write, and everything between them is the agent's entire turn — its model calls, its tool calls, its retries against a flaky OCR service — optical character recognition, the step that turns a scanned page into text. Holding the lock across that means a four-minute stage holds the lock for four minutes.

Work through what that does to run_9d4a. Six agents, each taking about four minutes, previously running concurrently and finishing the parallel phase in roughly five minutes. Serialised behind a turn-length lock, the same six take twenty-four minutes, and five agents spend most of that blocked on a mutex — the lock object itself, so named because it enforces mutual exclusion — while holding an idle model connection. The concurrency the blackboard was supposed to enable is exactly what the lock removes.

The second problem is worse than the first because it does not degrade, it stops. An agent that crashes mid-turn — the OOM kill from chapter 1 — holds the lock forever. There is no owner left to release it, no timeout in the naive design, and the remaining five agents block indefinitely on a run that will never make progress. The proposal almost always leaves out expiry, because expiry is where the difficulty starts: once the lock can expire, you have to decide what happens when the original holder wakes up after expiry and writes anyway, believing it still holds the lock. A lock with an expiry is a lease, and a lease has its own failure mode, which chapter 5 sets against the alternatives properly.

Compare-and-swap avoids both problems by not holding anything. There is no state to leak when a process dies, because the only guarantee is per-write, and a crashed agent leaves nothing behind but a task whose ownership fields have gone stale in the sense chapter 2 described. What it gives up is progress guarantees under heavy contention, since a slow writer can lose repeatedly to faster ones.

Making Tuesday visible on Tuesday#

The incident's real cost was not the lost section. It was two days of a signed-off report being wrong, and that part was preventable with checks that have nothing to do with concurrency control.

The first is a completion invariant at the end of the run: the run knows it dispatched six sections, so before it reports success it compares the number of tasks with status: done against the number of output.uri entries present in the final document, and refuses to report success when they disagree. On Tuesday that comparison would have been six against five, and the run would have failed at 14:23 instead of succeeding.

The second is a monotonicity check on the document itself. A section pointer that exists in version 61 and is absent in version 62 is a deletion, and no agent in this system is supposed to delete another agent's output. A validator that rejects writes which remove entries another agent added would have refused agent-vendor's put outright, with a message naming exactly what it was about to drop.

The third is instrumentation on the write path: count rejected conditional writes, and count writes accepted whose base version was not the current head, which is the store's name for the latest version it holds. In a design with no conditional writes at all, that second counter is the one that would have been ticking every day since the system was built, silently, with nobody watching.

All three of these are cheap, and all three assume the write either happened completely or not at all. That assumption is doing more work than it looks like. Compare-and-swap orders writes that completed; it has nothing to say about a process that died with the file half-written, or about a retry whose first attempt may or may not have landed, and those are separate failures with separate fixes.