Serialising buys correctness and pays in contention and deadlocks

Serialising means arranging that two runs which would otherwise overlap on the same data take their turns one after the other. It is the correct answer to the lost update of the previous chapters, and it is never free: whatever mechanism enforces the turn-taking also decides how long the second run waits, what happens when it waits too long, and what happens when two runs are each waiting for something the other holds. This chapter is about pricing that exchange, because the price is paid in a different currency from the bug — the wrong number was a correctness fault and everything it turns into is a liveness fault, meaning a system that is not producing wrong answers so much as failing to produce answers at all.

On the fulfilment agent from the previous chapter, where two entry points were both reading an order row and writing it back, the lost update went away on the Thursday. Four lines changed, the barrier test went green, the reconciliation query stopped finding disagreements, and the fix was two characters short of trivial:

async def reserve_items(order_id: str) -> dict:
    async with db.transaction():
        order = await orders.get_for_update(order_id)     # SELECT ... FOR UPDATE
        plan = await llm.decide_substitutions(order)      # 2.8 s median
        order.items_reserved += plan.count
        await orders.save(order)
    return {"reserved": plan.count}

The lock was correct. That is the whole subject of this chapter: the fix worked exactly as designed, and what it did was move the cost from a rare wrong number into a Monday-morning throughput collapse.

The transaction is now three seconds long#

A row lock in Postgres is held until the transaction commits or rolls back. That single sentence, plus the position of the llm call inside the async with block, determines everything that follows, so it is worth walking rather than asserting. The lock is taken by get_for_update on the first line inside the transaction. It is released when the transaction ends, which is at the closing of the block. The inference call sits between those two points. Therefore the lock is held for the duration of the inference call, and there is no arrangement of the code that avoids it while leaving the read and the write in the same transaction.

The critical section — the stretch of a run between taking the lock and giving it back, during which no other run may touch the thing it protects — measured over the previous week's traces, ran to 3.1 seconds at the median and 6.4 at p95, essentially all of it waiting on the model.

That number is the entire capacity story, and the capacity follows from it by one division. While one run holds ord_1, every other run touching ord_1 is asleep, so operations on a single key cannot overlap at all and the key's throughput is one operation per critical section. One divided by 3.1 seconds is about 0.32 operations per second, and that ceiling is a property of the key rather than of the machine — adding workers, cores or connections moves none of it, because the constraint is that the row admits one holder at a time.

The queueing follows from the same figure. Runs contending on one key form a line, and the k-th run in that line waits for the k minus one runs ahead of it, each holding for a critical section. Ten runs contending on the same key means the tenth waits roughly 28 seconds before it starts, which is longer than the 20-second tool timeout the agent framework was configured with, so it never got to run at all. The run at the back of the queue is not slow; it is cancelled before its turn arrives, and what the model sees is a tool error.

Ordinary web backends rarely notice this class of problem, because a lock held across a database write is measured in single-digit milliseconds and a thousand-fold safety margin hides a lot. An agent's critical section contains a network call to a model, which is three orders of magnitude slower and has a long tail — the p99 on that endpoint was 14 seconds, and a run that hits it holds the row for 14 seconds. That tail is the part worth staring at, because it is not yours. Anything that starts locking around an inference call has built a system whose per-key throughput is set by a third party's tail latency, and the p99 of somebody else's endpoint is a number you can measure and cannot change.

The rule that follows is short and worth applying mechanically: no lock is held across a model call. Rearranging the handler to obey it means the read happens outside any transaction, the model decides on data that may already be stale, and the write is conditional on nothing having changed since:

async def reserve_items(order_id: str, attempt: int = 0) -> dict:
    order = await orders.get(order_id)                    # no lock, no transaction
    plan = await llm.decide_substitutions(order)          # slow, holds nothing
    updated = await db.execute(
        """UPDATE orders SET items_reserved = $1, version = version + 1
           WHERE id = $2 AND version = $3""",
        order.items_reserved + plan.count, order_id, order.version,
    )
    if updated.rowcount == 0:
        if attempt >= 3:
            raise ConflictExhausted(order_id)
        return await reserve_items(order_id, attempt + 1)  # re-read, re-decide
    return {"reserved": plan.count}

The version = $3 clause is what makes the write safe without a lock, and it is worth being exact about the guarantee it gives. The statement changes the row only if the row still holds the version the run read, so a run whose data went stale affects zero rows and finds out by reading rowcount. What that licenses is the claim that no write is ever applied on top of a value the writer did not see. What it does not license is that the write happens: a run that loses the comparison has done its reading and its inference for nothing, and the retry does both again.

The critical section is now the UPDATE statement itself, about 1.2 milliseconds. Divide one by that and the per-key ceiling goes from 0.32 operations per second to something in the high hundreds, which is the same division as before applied to a critical section three orders of magnitude shorter. What replaces the waiting is failing: when two runs overlap, one write finds version already moved and affects zero rows, and that run pays for another read and another inference. The bet this makes is the one the name optimistic concurrency describes: collisions are assumed to be rare, nothing is held to prevent them, and the cost is paid only on the runs where one actually happens.

Pricing that bet needs the collision rate and the cost of a collision, and both were measured. On this workload the conflict rate at Monday-morning concurrency was 3.1%, and each conflict cost 2.8 seconds and roughly 4,000 input tokens, so the retries added about 0.9% to the inference bill. The bill moves by less than the conflict rate because a conflict repeats one inference call inside a run that makes several, so the extra tokens are a fraction of what a whole run costs rather than a whole run's worth. That is a bargain at 3%. At a 40% conflict rate the same design spends more time redoing work than doing it, and the answer stops being optimistic concurrency at all — the retries themselves lengthen the period during which other runs are reading, which raises the conflict rate further.

Four ways to stop the interleaving#

The four mechanisms differ in where the constraint lives. A row lock taken with SELECT ... FOR UPDATE makes every other transaction wanting that row wait in the database until the holder commits. An advisory lock is one the database hands out against any key you name, with no row behind it, so it can guard a business identifier that spans several tables or no table at all. A single-writer queue routes every operation on a key to one consumer, which removes the overlap rather than arbitrating it. The compare-and-set is the conditional write from the previous section.

Read the bottom two rows as gaps rather than as scores, because they are the two places where one column has something the others simply do not have. A queue gives ordering, which matters when the operations are not commutative — applying a discount before a tax calculation is not the same as applying it after — and neither locks nor compare-and-set give you that; they only give you that the operations do not overlap. The row lock is the only mechanism that also constrains a migration script or an admin console writing the same row without knowing about your scheme, because the lock lives in the database and everything that talks to the database respects it. Advisory locks and compare-and-set are conventions, and a convention protects you against the participants who follow it.

Granularity is a separate decision from mechanism and it moves the numbers further than the choice of mechanism does. The same FOR UPDATE taken on the order row rather than on the customer row changes contention by whatever the fan-out is: the median customer in this system has 1.4 open orders, so per-order locking sees collisions almost only from genuine same-order overlap, while per-customer locking pulls in every unrelated order that customer has. The mechanism for that is worth spelling out, because it is the same division as before with a different denominator. Locking a key that covers f objects makes every run touching any of those f objects queue behind every other, so the contention a run experiences scales with f rather than with the number of runs genuinely competing for the same object. Someone had in fact proposed locking the customer "to be safe", which would have been correct and would have raised the collision rate on the top two hundred wholesale accounts, each carrying dozens of concurrent orders, by more than an order of magnitude. The key to lock is the smallest one that covers the invariant you are protecting, and the invariant here is about a count on one order.

The queue deserves more credit than it usually gets in agent systems. Routing every operation on an order to the single consumer that owns hash(order_id) % 32 removes the concurrency rather than managing it, so there is no lock, no conflict handling, and no retry logic. The cost is head-of-line blocking: one run that takes 40 seconds delays every other operation on the same partition, and a partition holding a hot key becomes a queue nobody drains. It is the right shape when the per-key rate is low and the ordering matters, and the wrong shape when one key carries a hundred times the traffic of the others.

The deadlock nobody wrote#

Two runs in the Monday batch both needed the order row and the customer's credit row. The chat-triggered run reads the order first because it is answering a question about the order; the reconciliation run reads the customer first because it iterates by customer. Neither developer knew about the other's ordering, and both were locking correctly.

Follow the arrows into a cycle, since the cycle is the whole mechanism and it needs no coordination to form. Run A holds the order and asks for the customer. Run B holds the customer and asks for the order. Neither will release what it holds until it finishes, and neither can finish until it gets what it asked for, so the wait is permanent by construction rather than merely long. What produced it is that two runs acquired the same two locks in opposite orders, and neither piece of code contains a bug on its own — the defect is a relation between two files that no reviewer of either file could see.

Postgres notices the cycle after deadlock_timeout, which defaults to one second, kills one of the two transactions with 40P01, and lets the other proceed. That is the good outcome, and it is only good because the database was doing the detecting. The same cycle across two different systems — a row lock in Postgres and an advisory lock in Redis, or a lock in your database and a lock inside a third-party API — has nobody watching it, and it resolves when something times out, if something times out.

Two settings turn this from an incident into a handled case. Locks are acquired in a canonical order, which in practice means sorting the identifiers before taking them rather than taking them in the order the business logic happens to need them, so no cycle can form. That works because a cycle requires two holders to disagree about which lock comes first, and a total order that every run follows makes such a disagreement impossible: if every run takes the lower identifier first, the run holding the lower one is never waiting on a run holding the higher. And lock_timeout is set below the tool timeout the agent framework enforces — 5 seconds against a 20-second tool budget — so a run that cannot get the lock returns a typed, retryable error to the model with time left to do something about it, rather than the framework cancelling a coroutine that is still holding a transaction open on a connection the pool believes it has taken back.

What to measure before deciding you serialised the right amount#

"We added a lock" is not a state anybody can reason about later. Four numbers make it one, and all four are cheap to emit from the same place the lock is taken: how long runs waited to acquire it, how long they held it, how often acquisition failed, and how often a conflicting write was rejected.

lock.wait_ms{key=order}     p50 3    p95 8210   p99 19400
lock.hold_ms{key=order}     p50 3100 p95 6400   p99 14020
lock.timeout_rate{key=order}       0.061
cas.conflict_rate{key=order}       0.031

Hold time at p50 of 3.1 seconds is the diagnosis on its own, and it points at the model call rather than at the database, because no database operation in this handler takes seconds and the inference does. Wait time at p95 of 8.2 seconds says the contention is real and concentrated, and it is worth reading against the hold time rather than on its own: a p95 wait of roughly three hold times means the run at that percentile is queueing behind two or three other runs, which is a statement about how many runs are landing on one key. Comparing the same metric per key would show whether that is a few hot orders or the whole workload. A timeout rate of 6.1% means one run in sixteen is failing on a lock and retrying, which is work being done twice and inference being paid for twice. Without these, the argument about whether to keep the lock is conducted entirely in opinions, and the usual resolution — widen the lock to be safe — is the one that makes every number worse, since widening the key raises the fan-out that the wait time scales with.

Where serialising is the wrong answer#

A lock is scoped to the system that implements it, so it does nothing about effects that have already left. If a run sends the customer an email and then loses the compare-and-set, the email is still sent; the retry sends a second one. Ordering the writes so that irreversible effects happen last, after the conditional write has succeeded, is most of the remedy, and the rest belongs to idempotency keys and to the invariants of chapter 6. Distributed locks over a cache are a common way to get this wrong, because a lock held in Redis does not stop a process that paused for a garbage collection from waking up after its lease expired and completing a write it no longer has the right to make. The lock's guarantee is about the lock service's own state, and a process that has stopped reading the clock is outside that guarantee entirely.

The blunter failure is the global lock: one mutex around the whole agent handler, which passes every barrier test from chapter 3 and turns a system with thirty concurrent runs into one with a single active run. It looks like caution and it is a capacity decision made by accident, discoverable only by measuring the wait time nobody is measuring. It is also the version of the fan-out arithmetic taken to its limit, since the key being locked now covers every object in the system.

Serialising, then, is a purchase rather than a repair. What is bought is that two runs cannot both be inside the same data at once, and what is paid is that the second run waits, or fails, or deadlocks, and that the length of the wait is whatever the protected code takes — which in an agent is a model call belonging to someone else. That price is worth paying on the keys where the invariant is real, and the way to know whether it was worth paying is the four numbers above, emitted from the same line that takes the lock.

The deeper limit is that all four mechanisms answer the same question — how should concurrent access to a genuinely shared thing be ordered — and a large share of agent concurrency bugs are not about genuinely shared things at all. Ten runs collide over a package cache, a temporary file path, a config file in the home directory and a port number, none of which any of them wanted to share; they inherited them from defaults nobody set. Adding a lock to a shared temp path is possible and slow and beside the point, because the correct fix is for the two runs never to have been looking at the same path.