A concurrency test asserts invariants after retries and timeouts

Every failure in the previous five chapters is silent. The lost credit, the duplicate folder, the reservation short by one, the workspace that a neighbouring run cleaned up — none of them raised an exception in the process that caused them, and most of them returned success. A test suite built around exceptions is therefore blind to almost the whole set, and the correction is to move the assertion from what the calls returned to what the data looks like afterwards. This chapter is about how to write that assertion, how to produce the overlaps and faults that make it worth writing, and where the resulting suite still cannot see.

There was a test called test_parallel_tool_calls. It had been written after the first duplicate refund, it ran in 1.2 seconds, and it looked like due diligence:

@pytest.mark.asyncio
async def test_parallel_tool_calls():
    results = await dispatch([
        tool_use("issue_refund", {"dispute_id": "dsp_1", "amount_eur": 40}),
        tool_use("issue_refund", {"dispute_id": "dsp_2", "amount_eur": 15}),
    ])
    assert all(not r.is_error for r in results)

In the three weeks after it went green, production issued two duplicate refunds totalling 340 EUR. Both came from the same sequence: the payment provider took longer than the 8-second tool timeout, the framework surfaced a timeout to the model, the model reissued the call, and the provider had in fact completed the first request. The test could not have found this, because the test asserted that two calls returned without an error, and in the failing production case both calls also returned without an error. Two successful refunds is the bug.

What a green run of that test rules out#

Very little, and it is worth being specific about which of the failures from this topic it would catch. The lost update from chapter 1 leaves both calls reporting success. The duplicate Q3 folder from chapter 2 leaves all four calls reporting success. The reservation race from chapter 3 raises nothing. The lock timeout from chapter 4 does raise, so a crash-only assertion catches that one, and the workspace collisions from chapter 5 mostly show up as a test failure in some unrelated repository twenty minutes later. A test that only detects thrown exceptions is aligned with the one failure mode in the set that was already loud.

That roll-call is the answer to a belief worth naming, because it is what made the 1.2-second test feel like coverage. The belief is that a system which completes its work without erroring has done its work. It holds for a single-threaded program, where the absence of an error and the correctness of the result are produced by the same run. It stops holding the moment two runs share data, because the thing that goes wrong is a relation between what two runs did, and neither run is in a position to notice it — each one finished its own steps in order and returned what its own steps produced. The quantity a crash-only assertion watches is therefore constant across the good case and the bad one, and the quantity that varies is in the database.

The assertion has to be about the state of the world after the calls, and the useful form of that is an invariant, meaning a property of the data that has to hold no matter how the schedule went — which run read first, which retried, which timed out. For the refunds system there are four:

• Every dispute has at most one settled refund, whatever the retry history.
• The sum of credit_log.amount_eur for an account equals accounts.credit_balance_eur.
• Every object in the receipts bucket is referenced by exactly one invoice row, and every invoice row's receipt_key resolves.
• Every job that was accepted reaches a terminal state; none is left in in_progress with no owner.

Each is a query rather than an opinion, which means each can be checked in a test, in CI, and against production on a schedule. That property is what makes an invariant worth more than a description: a sentence about how the system ought to behave has to be re-interpreted by whoever reads it, and a query returns rows or does not.

The last one needs its terminal states written down before it means anything, because "dropped" is only definable against a state machine somebody drew:

Follow why the abandoned transition has to exist before the invariant can be written. Without it, the only states a stalled job can be in are accepted and in_progress, and in_progress is also the state of every job that is merely taking a long time. A query cannot separate those two populations, because they are the same rows. Adding a transition out of in_progress on lease expiry gives the stalled case a name, and once it has a name the healthy population is defined by exclusion. A job sitting in in_progress with an expired lease and no reaper is the silently dropped work, and without the abandoned transition it is indistinguishable from a job that is merely slow. Adding the state is what turns "nothing was dropped" from a hope into SELECT count(*) FROM jobs WHERE state = 'in_progress' AND lease_until < now().

The harness that actually overlaps and actually fails#

Three ingredients make the test meaningful: enough overlap that interleavings occur, faults injected in the middle so the retry paths execute, and invariants checked once everything has settled.

class Chaos:
    """Wraps the tool layer; injects faults at the seams, deterministically per seed."""
    def __init__(self, seed: int, rate: float = 0.15):
        self.rng, self.rate = random.Random(seed), rate

    async def call(self, tool, **kwargs):
        roll = self.rng.random()
        if roll < self.rate * 0.4:                       # timeout after the effect lands
            asyncio.create_task(tool(**kwargs))
            raise asyncio.TimeoutError("tool timed out")
        if roll < self.rate * 0.7:                       # connection reset before it lands
            raise ConnectionResetError("upstream reset")
        result = await tool(**kwargs)
        if roll < self.rate:                             # duplicate delivery
            await tool(**kwargs)
        return result


@pytest.mark.parametrize("seed", range(200))
@pytest.mark.asyncio
async def test_refund_invariants_hold_under_overlap(db, seed):
    chaos = Chaos(seed)
    disputes = [f"dsp_{i}" for i in range(4)]
    await asyncio.gather(*[
        run_agent_turn(chaos, dispute_id=chaos.rng.choice(disputes))
        for _ in range(8)
    ], return_exceptions=True)                            # errors are expected; state is not

    await assert_invariants(db, disputes, seed=seed)

The single roll drawn at the top is what makes each call land in exactly one fault band, and the bands are cut out of the fault rate rather than out of the unit interval. The lowest four tenths of the rate produce a timeout raised to the caller while the effect is started anyway. The next band, up to seven tenths of the rate, produces a connection reset before the tool is called at all, so the caller sees a failure and nothing happened. The remaining part of the rate, above seven tenths, lets the call succeed and then delivers it a second time. Everything above the rate is an ordinary successful call. Those three bands are the three ways a network boundary lies to a caller: it says no when it meant yes, it says no and meant no, and it says yes twice for one request.

The first band is the one that found the production bug, because the effect is allowed to complete while the caller is told it timed out, which is exactly what the payment provider did. The return_exceptions=True is deliberate, since a test that fails on the first injected error is testing that faults do not happen rather than that the system survives them.

async def assert_invariants(db, disputes, seed):
    dupes = await db.fetch_all(
        """SELECT dispute_id, count(*) FROM refunds WHERE state = 'settled'
           GROUP BY dispute_id HAVING count(*) > 1""")
    assert not dupes, f"seed={seed} duplicate settled refunds: {dupes}"

    drift = await db.fetch_all(
        """SELECT a.id FROM accounts a
           JOIN (SELECT account_id, sum(amount_eur) s FROM credit_log GROUP BY 1) l
             ON l.account_id = a.id
           WHERE a.credit_balance_eur <> l.s""")
    assert not drift, f"seed={seed} balance/ledger drift on {drift}"

    stuck = await db.fetch_val(
        "SELECT count(*) FROM jobs WHERE state = 'in_progress' AND lease_until < now()")
    assert stuck == 0, f"seed={seed} {stuck} jobs abandoned without transition"

Determinism per seed is what the whole arrangement rests on, and it comes from a single generator drawn in a fixed order. Chaos holds one random.Random, seeded by the parameter, and every draw against it — the dispute each turn is given, and the roll inside each call — comes from that one stream. Replaying the seed replays the draws in the same order, so the same turn gets the same dispute and the same call gets the same fault. A second source of randomness anywhere in the path, including the ordering of a set or a clock read, breaks that and turns a reproducible failure into a story about a failure.

Two hundred seeds at eight concurrent turns runs in 4 minutes 40 seconds on a 4-vCPU CI runner against a real Postgres, which is affordable on a merge queue and too slow for every push, so it runs on merge and nightly at 2,000 seeds. The runtime scales with the seed count, since each seed is an independent iteration, so ten times the seeds is ten times the wall clock — which is the arithmetic that puts the larger sweep in a nightly window rather than on the merge queue.

The ratio of runs to resources is what decides whether anything overlaps at all, and it is the parameter people get wrong by setting it to something realistic. The arithmetic is a division. Spread t turns across r resources and the average number of runs per resource is t over r, and an overlap requires at least two runs to land on one resource. Eight concurrent turns spread across four dispute ids gives an average of two runs per dispute, so a collision is close to certain within every iteration. Eight turns across a hundred disputes gives an average well below one, so most disputes are touched once or not at all — that setting reproduces production's distribution and produces almost no overlap, which is the same mistake as the twelve-thread loop in chapter 3 wearing better clothes. The test's job is to make the rare thing common, so the resource pool is deliberately tiny and the concurrency deliberately higher than the system will ever see.

The seed in every assertion message is the part that makes this a tool rather than a source of noise. A failure comes with an integer, pytest -k test_refund_invariants -p no:randomly --seed 1477 reproduces it, and the reproduction is where a barrier test from chapter 3 gets written to pin that specific interleaving forever. The rule that keeps the suite honest is that a failing seed is never retried away; it becomes a ticket with the seed in the title, because a soak test — a long randomised run held at deliberately higher concurrency than production and repeated until something breaks — whose failures are re-run until green is a slower way of asserting that nothing crashed.

The gap to read in that table is between the first two columns on the "finds" row and the last column on the "fails when" row. The first two find things before they happen and are limited to schedules somebody's harness can produce; the third has no such limit and pays for it by only firing once the damage exists. That is a division of labour rather than a ranking, and the third column is not a testing strategy that failed to be a test. Production runs schedules your harness will never produce — a deploy mid-turn, a database failover, a partner API degrading for ninety seconds — and the same invariant queries, running as a scheduled job every fifteen minutes with an alert on a non-zero count, are what catch that. The drift query above cost 40 milliseconds against the production replica and would have found the two duplicate refunds within an hour rather than within a reconciliation cycle.

Sampling, proof, and the boundaries the harness stubs#

Randomised soaking explores a sample of the schedule space, and 2,000 seeds is a sample. A run that passes ten thousand seeds has evidence and not proof, and the difference matters most for the rarest interleavings, which is where the expensive bugs are. The state of the art for closing that gap is deterministic simulation testing, where the clock, the scheduler, the network and the disk are all controlled by the test framework so that a single seed reproduces an entire distributed execution exactly — FoundationDB's simulation harness is the canonical example, and Antithesis and madsim bring the approach to systems you did not write from scratch. Model checkers such as TLA+ attack it from the other end by proving properties of the protocol rather than testing an implementation. Both are more than most agent systems need and both are the honest answer to "how would you know for certain".

The harness also has a blind spot at every boundary it stubs, and the blind spot is exactly as wide as the stub. Fault injection at your tool layer exercises your retry paths; it says nothing about whether the payment provider's own idempotency handling is correct, and the deduplication that makes the duplicate-refund invariant hold is running on their side of a network. So the first invariant in the list above is not one your suite proves. It is one your suite checks given that the provider deduplicates, which is a different claim: any invariant that depends on somebody else's concurrency control is an invariant you are assuming. The most you can do is assert it against their sandbox and monitor it in production.

And a soak run on a shared machine will produce failures that have nothing to do with the code under test, because eight concurrent runs on one box hit exactly the shared caches and temp paths from chapter 5. Those failures are worth having, and they must be distinguishable from domain failures, otherwise the suite develops a reputation for flakiness and people stop reading it. Running each concurrent turn in its own workspace, with its own database schema or tenant id, keeps the test asserting the thing it names.

What holds up in practice is a cycle rather than a checklist. Every new tool gets four questions answered before it merges: what it mutates, what its conflict key is, whether it is safe to call twice with the same arguments, and what it shares by default. Anything that mutates gets a conflict key in the dispatcher's table and a decision about whether it is combined, handled optimistically, or serialised. Every shared resource that turns up gets an invariant written as a query, and that query goes into both the soak and the production schedule. When the production checker fires, the trace goes into a barrier test before anybody writes a fix, so the fix is verified against a failure rather than against an absence.

The four questions are the useful part to carry into the next system, because they are answerable at design time by anybody who knows what the tool does, and they front-load the work that concurrency otherwise defers into a reconciliation query somebody runs monthly. Each of them names a property of the data rather than of a schedule, which is the same move the invariants make: a schedule is something you cannot enumerate and a property is something you can check, and every technique in this topic is a way of converting a question about the first into a question about the second.