Leases, compare-and-swap and one designated writer fail differently

The pipeline that had six writers now has fourteen, because someone parallelised clause extraction across documents instead of running it once per run. The conditional-write rejection rate went from 4.1% to 19%, and the ninety-ninth percentile write now takes 1.4 seconds instead of 80 milliseconds because it is retrying three and four times. Nothing is broken. The document is correct, no updates have been lost since March, and the design review that has been called is about whether compare-and-swap is still the right policy at this scale or whether something else should hold the write path.

Four candidates are on the table, and the reason the discussion is hard is that they do not fail in comparable ways. Each one prevents a different subset of the problems from the last two chapters, and each one converts the remainder into a different kind of cost.

Detection scales badly, exclusion scales worse#

Compare-and-swap is pure detection. It never blocks a writer, holds no state between operations, and leaves nothing behind when a process dies, which is exactly why it survived the argument in chapter 3. Its weakness is that a loser has to redo something, and how much depends on what the update was computed from. Setting a field the agent owns costs a re-read and a few milliseconds. Choosing which task to claim is different: the loser's decision was derived from a document that has moved, so it has to re-run the selection, and under contention several agents can repeatedly select the same newly-freed task and all but one lose. Throughput does not collapse, it erodes, and the erosion is superlinear in the number of writers, meaning doubling the writers costs more than twice as much, because each retry is itself a write that can cause another agent to lose.

Those are the numbers from replaying a day of traffic at synthetic writer counts against the same Postgres row. The interesting column is the last one, which stops rising around twenty-four writers and then falls, so past that point adding agents makes the run slower.

A lease attacks a different problem. Rather than validating a write after the fact, it grants an agent an exclusive claim on one item for a bounded period, recorded in the document itself: owner, owned_since and lease_expires from chapter 2, doing the job they were designed for. The critical difference from the lock in chapter 3 is scope. The lease covers one task, not the whole document, so fourteen agents holding fourteen leases on fourteen tasks contend with each other not at all, and the only compare-and-swap contention left is on the moment of claiming.

def claim(store, key, task_id, agent, lease_secs=600):
    def take(doc):
        t = find(doc, task_id)
        held = t["owner"] and t["lease_expires"] > now()
        if held:
            raise AlreadyHeld(t["owner"])
        t["owner"] = agent
        t["owned_since"] = now().isoformat()
        t["lease_expires"] = (now() + timedelta(seconds=lease_secs)).isoformat()
        t["fence"] = doc["fence_counter"] = doc["fence_counter"] + 1
        return doc
    return apply_update(store, key, take)

The expiry is what makes a crashed holder recoverable: at 600 seconds, an agent whose container is evicted at minute three of a four-minute turn has its task returned to the pool ten minutes later, and a replacement picks it up. The cost is that a stolen item can finish twice. The original holder may not be dead — it may be blocked on a slow model call — and when it wakes at minute twelve it will happily write results for a task another agent now owns. fence is the defence, a fencing token, which is a number that only ever increases and is handed out with each grant of the same item: every write an agent makes carries the fence number it was granted at claim time, and the store rejects a write whose fence is lower than the current one for that task. The late writer's work is discarded rather than merged, which costs four minutes of duplicated compute and keeps the document correct.

Choosing the lease duration is the part with no clean answer. Too short and healthy agents lose tasks mid-turn, producing duplicate work as routine behaviour. Too long and a crashed holder blocks the task for that duration, which is the coarse-lock problem back at task granularity. Heartbeating — the holder extends its lease every thirty seconds while it works — decouples the two, letting the expiry be short while a live agent keeps it alive, at the price of every agent now needing a background renewal path that keeps running while its main loop is inside a model call.

Appending instead of overwriting#

The third option removes write conflicts by removing overwriting. Each agent appends an immutable record of what it did, nobody edits anybody's record, and current state is whatever you get by folding the log from the beginning — a fold being the operation of walking the records in order and applying each one to a running result, so that the final result is the state the log implies.

{"seq": 412, "at": "14:22:07.412Z", "agent": "agent-risk",   "type": "section_written", "task": "risk",   "uri": "s3://…/risk.md"}
{"seq": 413, "at": "14:22:07.488Z", "agent": "agent-vendor", "type": "section_written", "task": "vendor", "uri": "s3://…/vendor.md"}

Tuesday's lost update cannot happen here, because the two agents are not modifying a shared structure; they are adding two rows, and both survive. It also gives back the history that chapter 1 identified as the blackboard's weakest property, and it does so as a by-product rather than as a deliberate extra. Reconstructing that the risk section was written, retracted at 14:40, and rewritten at 14:52 is a matter of reading three consecutive records.

What it charges is paid by every reader. Current state is no longer something you open; it is something you compute, and the fold has to be identical in every agent or two agents will believe different things about the same log. A run producing around 4,000 events makes the fold cheap in wall-clock terms — under fifty milliseconds — and expensive in coupling, because the fold function becomes shared logic that every agent must run at the same version. Long runs need periodic snapshots so readers do not replay from zero, and a snapshot is a materialised current state, which means the design ends up carrying both representations. There is also an ordering question that appending does not answer by itself: the seq numbers above have to come from somewhere, and if they come from the store then the store is serialising appends, and if they come from the agents then two agents can produce the same sequence number and the fold order is ambiguous.

The designated writer is a single point of failure with a queue attached#

The fourth proposal is the one that arrives in almost every review: stop letting agents write at all, and route every change through one writer agent that owns the document. Agents send it intents; it applies them one at a time.

This is not the lock from chapter 3, and the distinction is worth granting explicitly, because the critique that killed the lock does not land here. Under a lock, an agent holds the document across its entire turn, so a four-minute stage blocks everyone for four minutes. Under a designated writer, an agent does its four minutes of work holding nothing, then submits a small intent that the writer applies in milliseconds. Nobody idles waiting for a model call to finish. Serialisation happens at the point of application rather than across the whole read-modify-write span, which is the property that makes it workable.

Three things follow from it that the proposal usually does not mention.

The writer is a single point of failure. Every run in the system now depends on one process being up, and when it is not, no agent can record anything, including that it has crashed. It also becomes a queue, so it has a depth, a latency distribution, a backpressure story — what the system does when intents arrive faster than the writer can apply them — and a poison-message problem, where one malformed intent fails repeatedly and stalls everything behind it, and somebody has to own all four. The failure modes that come with a queue are well understood, which is a genuine argument in its favour, but they are failure modes the previous three designs simply do not have.

It rebuilds the coordinator the blackboard was chosen to avoid. The reason to put state on a board is that no component needs to know the plan or the participants; agents self-select work, and the topic's reference design, Exploring Advanced LLM Multi-Agent Systems Based on Blackboard Architecture, makes that the defining property. A component that receives every intent and decides how to apply it is holding the plan again, and the pressure to give it judgement — reordering intents, rejecting ones that look wrong, filling in gaps — is constant, because it is the only component with a complete view.

And it does nothing about stale reads. Serialising writes says nothing about when the agents computed them.

The writer applied the intent in order and produced a document nobody would call corrupt. The problem from chapter 4 is untouched, because it was never about write ordering. The only version of this proposal that helps is one where intents are conditional rather than absolute — set report.uri if tasks[tax].status == "done" — at which point the writer is evaluating compare-and-swap predicates on behalf of its callers and the architectural difference has narrowed to where the predicate is evaluated.

Setting them against the same criteria#

Every column has a "no" in the stale-read row, which is the most useful thing the table says. Write policy is not a solution to reading a value that has since changed, and expecting it to be one is how teams end up with three concurrency mechanisms and the same class of wrong output.

What the pipeline in this chapter actually adopted was a combination: compare-and-swap on the document for correctness, leases on tasks that take longer than a minute so a crashed holder is reclaimed, and a bounded appended event list inside the same document for the transitions that matter to an operator. The frontier is elsewhere — CodeCRDT has agents coordinating by observing a shared CRDT document — a conflict-free replicated data type, a structure whose operations are defined so that any two edits combine to the same result regardless of the order they arrive in — where concurrent edits merge by construction and no write is ever rejected, which removes the contention argument entirely at the cost of a data model most teams are not ready to reason about. It is worth watching and it is not what most systems need this quarter.

All four mechanisms in the table are about keeping the document correct. None of them makes it legible, and correctness is not what the on-call engineer is short of at 2am.