A race becomes a test when you hold one run between read and write

A race condition shows up as a defect that appears at a low rate in production and refuses to appear anywhere else, and the usual response — run the same code harder — cannot settle it, because a pass is consistent both with the code being correct and with the test being unable to reach the failure. This chapter is about the alternative, which is to stop generating schedules at random and start choosing one. The mechanism is a barrier: a point in the code where one run is parked and held until another has got to where you want it. What that buys is a test whose failure is a statement about the program rather than about the machine it ran on.

The fulfilment agent this chapter works from had two entry points. One ran when a customer asked about their order in chat, and one ran on a schedule to reconcile reservations against the warehouse feed. Both read the order row, decided which items to reserve, and wrote the row back. Over six weeks, 41 orders out of roughly 12,000 came out with a reservation count one lower than the number of reservation events in the audit table, a rate of about one in three hundred. Every one of them had both entry points active within the same few seconds.

The engineer who picked it up did the obvious thing first. She wrote a test that spawned twelve threads, each running the reserve path against one order, looped it ten thousand times, and let it run for four minutes on her laptop. It passed. She raised it to fifty threads and a hundred thousand iterations, gave it twenty minutes, and it passed again. At that point there are two possible conclusions and only one of them is true: either the bug is not in this code path, or the test cannot produce the conditions that trigger it. Deciding between those by running the loop longer is not a strategy, because a pass is consistent with both.

The 41 orders were not reported by anyone. They were found by a query that joined the order rows against the event table and looked for disagreement, which is worth saying because it is the shape of the only detector that works for this class of defect: nothing errored, no alert fired, and the customers whose orders were short by one item mostly re-ordered. The rate is the number that decides how the investigation goes. One in three hundred means roughly forty a week at current volume, and it also means that any experiment which runs a few hundred orders and sees nothing has learned nothing at all.

Why the laptop could not lose the update#

The reserve path is not a tight read-modify-write. It reads the order, asks the model which of the available substitutions are acceptable, and writes the result back:

async def reserve_items(order_id: str) -> dict:
    order = await orders.get(order_id)                    # read
    plan = await llm.decide_substitutions(order)          # 2.8 s median in prod
    order.items_reserved = order.items_reserved + plan.count
    await orders.save(order)                              # write
    return {"reserved": plan.count}

The quantity that decides whether two runs can collide is the width of the gap between the read and the write, because that is the stretch during which a second run can read a value the first run is about to invalidate. Call that width w. In production, w is dominated by the inference call: 2.8 seconds at the median, 6.1 at p95. Add a database round trip through the connection pooler and across an availability zone at 4.1 milliseconds, and the window in which another run can read the same stale value is close to three seconds wide, with the database contributing a rounding error next to the model.

In the test suite, llm was a fake that returned a canned plan with no await point of consequence, and the database was a local Postgres over a Unix socket at 0.35 milliseconds. The window on the laptop was somewhere around a millisecond, and the two threads had to be scheduled into that millisecond by the operating system, on a machine with an event loop that was not going to context-switch there anyway.

Now put those two widths together with the thing a stress loop actually varies. For a fixed arrival rate, the chance that a second run starts inside a given run's window rises roughly in proportion to w, so a window three orders of magnitude narrower makes each attempt three orders of magnitude less likely to collide. The number of expected collisions over a run of the loop is that per-attempt chance multiplied by the number of attempts, so the loop's detection power depends on the product of the two — and a stress test only ever increases the second factor. To recover what shrinking w threw away, the iteration count would have to rise by the same factor w fell, and twelve threads over ten thousand iterations is not that number by a wide margin.

The stress test had removed the very thing that made the race likely. This is the ordinary case rather than bad luck: the slow, external, awaitable parts of an agent are exactly what test doubles replace, and they are exactly what holds the transaction open. Adding threads makes collisions more likely per unit time and does nothing about a window that has been narrowed by three orders of magnitude, which is why the loop ran for twenty minutes and proved nothing about a defect that fires once every three hundred orders in a system where the window is three thousand times wider.

It is worth naming the belief that made the twenty-minute loop feel like evidence, because it is the same belief in every version of this investigation. The belief is that concurrency testing is a matter of applying more pressure, so a test that survives more threads for longer has ruled out more. What that reasoning holds constant is the window, and the window is the dangerous variable. A test double replaced a three-second await with a return statement and left every other quantity in the test looking more aggressive than production — more threads, more iterations, more wall-clock time — while the one quantity that governs whether the bug can happen at all was cut by a factor no amount of the others can repay.

Making the interleaving an argument#

The interleaving that produces the lost update — one run's write landing on top of another's when both read the same value first, so one of the two increments vanishes and nothing anywhere reports an error — is a specific one, and it can be written down. Run A reads the order and sees items_reserved = 0. Run B reads the same row and sees 0. B computes 1 and writes it. A, still holding its stale 0, computes 1 and writes it. The final value is 1 and two reservation events happened. Once you can name the sequence, you can drive it, and the mechanism is a place in the code where one run can be stopped and held while the other proceeds.

That place has to be a seam that exists in production as well as in tests, otherwise the test is exercising a different program. The cheapest version is a no-op hook on the repository:

class Orders:
    def __init__(self):
        self.after_read: Callable[[str], Awaitable[None]] = _noop

    async def get(self, order_id: str) -> Order:
        row = await db.fetch_one("SELECT * FROM orders WHERE id = $1", order_id)
        await self.after_read(order_id)
        return Order.from_row(row)

In production after_read is a coroutine that returns immediately, and its cost is one function call per read. In the test it becomes a barrier, which is a point in the code where one run is parked and held until the other has got to where you want it, so the schedule is chosen rather than waited for:

@pytest.mark.asyncio
async def test_two_runs_reserving_same_order_do_not_lose_an_update(orders, db):
    await db.execute("INSERT INTO orders (id, items_reserved) VALUES ('ord_1', 0)")
    a_has_read, b_has_committed = asyncio.Event(), asyncio.Event()

    async def hold_first_reader(order_id):
        if not a_has_read.is_set():
            a_has_read.set()                       # A announces its read
            await asyncio.wait_for(b_has_committed.wait(), timeout=5)  # and waits

    orders.after_read = hold_first_reader
    run_a = asyncio.create_task(reserve_items("ord_1"))
    await a_has_read.wait()                        # A is parked between read and write
    await reserve_items("ord_1")                   # B runs start to finish
    b_has_committed.set()                          # release A into its write
    await run_a

    row = await db.fetch_one("SELECT items_reserved FROM orders WHERE id = 'ord_1'")
    events = await db.fetch_val("SELECT count(*) FROM reservation_events WHERE order_id = 'ord_1'")
    assert row["items_reserved"] == 2, f"lost update: {events} events, balance {row['items_reserved']}"

Follow the two events through that body, because the whole method is in how they hand control back and forth. hold_first_reader runs inside run A's read, sets a_has_read, and then blocks on b_has_committed — so run A is now stopped at exactly the point in its life where it has a value in hand and has not written it. The test body, which was itself waiting on a_has_read, wakes up and runs B from start to finish; B reads the same row, sees the same value A saw, and commits its write. Only then does the test set b_has_committed, which unblocks A's hook and lets A carry its stale value into its own write. That is the lost-update sequence from the paragraph above, produced on purpose, with each step gated on the previous one rather than on a sleep.

The test now fails every single time, on every machine, in about 40 milliseconds, with a message that says what is wrong rather than that something is wrong:

FAILED test_reservations.py::test_two_runs_reserving_same_order_do_not_lose_an_update
AssertionError: lost update: 2 events, balance 1
assert 1 == 2

Two details about the seam decide whether this test survives the next refactor. It belongs on the repository, which is the boundary the rest of the system already goes through, rather than inside reserve_items, and it certainly does not belong in a mock.patch of a private method, because a patched internal is a test that fails when somebody renames a function and passes when somebody moves the read somewhere else. It also has to be per-instance rather than a module global, so that two tests running in the same session under pytest -n 4 cannot install each other's hooks. The name of the test carries the invariant, since the failure line in CI six months from now is the only documentation anybody will read at three in the afternoon.

The determinism here comes from the event loop having exactly one place to run at any moment, so await a_has_read.wait() in the test body cannot proceed until the hook has set it, and the hook cannot proceed until the test sets the second event. For code that uses threads rather than coroutines the pattern is identical with threading.Event in place of asyncio.Event, and the same rendezvous holds, because both primitives are doing the same job of turning "these might overlap" into "these overlap in this order". The whole test runs in 40 milliseconds against a real Postgres, which means it costs nothing to run on every commit, in contrast to the twenty-minute loop it replaces.

The timeout on b_has_committed.wait() is not decoration. Without it, a future change that makes B block on A — a lock, which is where chapter 4 goes — turns this test into a suite that hangs in CI rather than one that fails, and a hang is a much worse signal than an assertion.

What you assert decides what you have proved#

The temptation, having got two runs to overlap, is to assert that nothing threw. That assertion passes on the broken code, because the broken code does not throw, and it will keep passing through every variation of the bug that produces a wrong number quietly.

Read that table by the distance between its two columns rather than by working down the first one. The gap narrows as you descend: the top row catches a category the failing code never enters and misses the entire defect, while the fourth row catches a disagreement between two records that both exist. What is being bought with each step down is not strictness but proximity — each assertion sits closer to the fact the business cares about, and the closer it sits the fewer ways there are for the code to be wrong while the test is green.

The last row is worth noticing before the fix exists. Once the code uses a compare-and-set — a write that carries the value it read in its WHERE clause, so it affects zero rows if anything moved underneath it — the correct outcome of this interleaving is not "both succeed"; it is that one run wins, the other gets a typed conflict, and it retries from a fresh read. The assertion has to change with the design, so it is worth writing the test to assert the domain fact — after two reservation attempts the order reflects two reservations — rather than the mechanism, and to let the mechanism decide whether that is reached by both writes landing or by one retrying.

Impossible schedules, and races nobody thought of#

A barrier test encodes a hypothesis about an interleaving you already suspect. It will not find a race you have not thought of, and a suite full of them is not a concurrency test suite so much as a set of regression tests for concurrency bugs you have already paid for. Finding the unknown ones needs overlapping traffic and invariant checks, which is chapter 6's subject, and the two work together: the stress run surfaces a candidate, the barrier test pins it down so it fails identically forever.

The barrier can also lie, and this is the failure mode that costs the most, because its output looks exactly like a real finding. Because you are choosing the interleaving, you can construct one the real system cannot produce — holding a read across a boundary where production takes a lock, or interleaving two operations that are in fact routed to the same worker by a partition key. A test that fails on an impossible schedule sends somebody to fix a bug that does not exist, and the resulting lock is pure cost: contention, latency and a deadlock risk bought to prevent nothing.

The guard is to derive the barrier point from evidence rather than from the shape of the code. The query log timestamps that showed the two selects before the two updates are evidence. So are the trace spans from the incident, taken from a run where both entry points were active. What is not evidence is that a read and a write appear on consecutive lines, since plenty of such pairs are reached only by a single writer. If you cannot point at production data that shows the two operations overlapping, the first task is to find out whether they do.

None of this is a fix. What exists at the end of this chapter is a red build with a named failure and an unambiguous statement of the invariant that is being violated, and that is the whole of what a barrier is for: it converts a probability into a certainty, so the next step is measurable rather than hopeful. The schedule is no longer something the machine picks and the test hopes for; it is an input, and a test whose inputs are all specified fails for a reason somebody can read.

The fix is to stop the interleaving, and every mechanism for doing so — a row lock, a queue, a critical section, a compare-and-set — converts the correctness problem into a throughput problem. In an agent that holds its read open across a 2.8-second inference call, that conversion is not a minor tax, and the same test that fails today will pass with any of them while telling you nothing about what the p95 did to the fulfilment queue on a Monday morning.