Fan-out cuts wall-clock time and buys rate limits and races

Fan-out is dispatching several branches of work at the same time and keeping whichever pays off. It is the lever engineers reach for first when an agent feels slow, and on a run whose steps are genuinely sequenced by data dependencies it can only recover the sum of the independent waits less the longest of them — the ceiling the second chapter of this topic derived.

What this chapter is about is the other side of that ledger, because the ceiling is not the constraint that bites. Three costs appear on the day fan-out goes to production and none of them is visible in a latency measurement taken in the week it ships: a per-minute ceiling shared with every other run in the fleet, a shared piece of state that two branches now write, and the arithmetic of executing work that a sequential run would have skipped. The first is loud, the second is silent, and the third is a straightforward increase in the bill that a latency-focused review will not look for.

Independence has four tests, not one#

"Independent" usually gets checked against the first of these and the others go unexamined.

The first test is the one a compiler could almost answer, which is why it is the one that gets asked. The other three are properties of the environment the branches run in rather than of the branches themselves, and none of them is visible in the function signatures.

The third and fourth are the ones that make parallelism cost money rather than merely risk correctness. The provider's share of that capacity is metered as input tokens per minute, ITPM for short, which is the number of prompt tokens an account may send in a rolling minute before further requests come back refused rather than answered. Shared capacity is not just the model provider: carrier_tracking sits behind a partner API allowing 60 requests per minute per account, and fanning out six shipment lookups for a wholesale customer consumed a tenth of that minute's allowance in one burst, which throttled the sequential runs happening alongside. Note where that cost landed — on runs that were not fanning out at all. A shared limit converts one run's burst into every run's latency, which is why this failure looks like a dependency outage rather than like a change anybody made.

Conditional necessity is the interaction with the previous chapter's stopping rule, and it is the reason token spend rose. Sequentially, the tracking-number strategy resolves on 71% of ambiguous tickets and the run proceeds; the other two strategies never execute. In parallel, all three execute every time, so the mean number of disambiguation model calls per ambiguous ticket went from 1.4 to 3.0. Latency improved because the work overlapped; cost rose because there was more work. The two effects are not in tension and they are not a trade in the usual sense — the same change produced both, and a review that measured one of them saw an unqualified improvement.

The diagram is the incident, and it shows all three failures at once. Three arrows converge on one row, which is the second test; one arrow comes back with a limit rejection that costs twelve seconds of the run it was supposed to accelerate, which is the third; and all three branches ran when one would have sufficed, which is the fourth. The value the loop reads is whichever write landed last rather than whichever branch was right, and nothing anywhere in the run records that a choice was made.

Bounding concurrency on purpose#

Left alone, the concurrency of an agent is whatever the model emits — a response with six parallel tool_use blocks produces six simultaneous calls, and a response with one produces one, which means the system's peak load is set by a sampling decision. That is worth stating plainly because it is the default: an agent with no explicit limit does not have a low limit, it has no limit, and the number that governs its worst moment is chosen at generation time by something that has never seen your rate limits.

The bound has to be explicit, and it has to exist at two levels: within a run, so one ticket cannot saturate a partner API, and across runs, so the fleet cannot saturate the model provider.

The cross-run number comes from the latency target and the arrival rate rather than from taste, and the derivation is four steps. Little's law — the standing population of a queueing system is its arrival rate multiplied by the time each item spends inside — puts the steady-state population at 34 arrivals per minute and a target time to success of 74 seconds, which is 34 ÷ 60 × 74 ≈ 42 runs in flight. Each run holds at most one model request at a time when sequential, so that is 42 concurrent requests. Each of those runs makes 7 requests over its 74 seconds, so the fleet issues about 238 requests a minute, and at roughly 14,000 input tokens each that is about 3.3 million input tokens per minute — above a 2-million-token-per-minute ceiling before any fan-out at all.

Two things follow from that arithmetic being possible before any code is written. The first is that the fleet was already over its ceiling at the target latency, so the headroom fan-out wanted did not exist to be spent. Prompt caching is what closes that gap, since cache reads bill and count differently and the rolling breakpoint from chapter three removed roughly two thirds of the fresh input tokens. Fan-out then spends whatever headroom is left, and there was not three times as much.

The ceilings themselves are worth reading carefully, because there are usually three of them and they bind under different conditions. Requests per minute binds an agent making many small calls, input tokens per minute binds one carrying a long transcript, and output tokens per minute binds one generating documents. Divide each ceiling by this agent's per-request figure and only one of them is close. This agent is squarely in the second case: at 14,000 input tokens per request it exhausts a 2-million-token allowance after 142 requests, well before a 4,000-request-per-minute cap is in sight. Which ceiling binds determines which optimisation helps, and it is the reason chapter three's caching work bought headroom here while chapter four's tiering, which mostly reduced price rather than token count, bought almost none.

class RunConcurrency:
    def __init__(self, per_run: int, fleet: int, tokens_per_min: int):
        self._branch = asyncio.Semaphore(per_run)     # branches inside one run
        self._fleet = asyncio.Semaphore(fleet)        # requests across all runs
        self._tokens = TokenBucket(tokens_per_min)    # provider ITPM headroom

    @asynccontextmanager
    async def slot(self, estimated_input_tokens: int):
        async with self._branch, self._fleet:
            await self._tokens.acquire(estimated_input_tokens)
            yield

async def disambiguate(run: RunState, conc: RunConcurrency) -> str | None:
    strategies = pick_strategies(run)                 # 1..3, cheapest first
    async def attempt(name: str) -> Candidate | None:
        async with conc.slot(run.estimated_input_tokens):
            return await STRATEGIES[name](run)

    tasks = [asyncio.create_task(attempt(s)) for s in strategies]
    try:
        for done in asyncio.as_completed(tasks):
            candidate = await done
            if candidate and candidate.confidence >= 0.9:
                return candidate.shipment_id      # first acceptable wins
    finally:
        for t in tasks:
            t.cancel()                            # stop paying for the losers
    return None

Two details in that function are the difference between a fan-out that saves money and one that spends it. Cancelling the outstanding tasks in the finally block stops the losing branches at whatever token they have reached rather than letting them run to completion for output nobody reads, which recovered about 40% of the extra spend on this agent. That it sits in finally rather than after the loop is what makes it hold on the early-return path, which is the path taken whenever the fan-out works.

The second detail is that the branches return candidates to the caller instead of writing to shared state, so the merge happens in one place, in one goroutine's worth of control flow, where a policy — highest confidence, or first acceptable, or require agreement between two — can be applied deliberately. That removes the second independence test rather than passing it: the run_scratch race disappears because nothing shared is written until the winner is chosen.

Where a branch genuinely must write, the write needs a key that makes a duplicate harmless: f"{run_id}:{tool}:{sha256(canonical_args)}" as an idempotency key on ledger_write turns two branches producing the same note into one record and two branches producing different notes into a visible conflict rather than a silent overwrite. The arguments are in the key on purpose — that is what makes the second case distinguishable from the first instead of collapsing both into one row.

What the bound is worth#

Sweeping the per-run branch limit over the same replayed workload at Monday-peak arrival rates gives a curve with an interior optimum, which is the shape to expect and the reason a fixed "parallelise everything" instinct goes wrong.

The three right-hand columns move differently and that is the whole reading. Cost rises at every step, monotonically, because more branches means more work executed. The 429 rate rises at every step and accelerates. p95 improves once and then gets worse than sequential from three branches onward, because 429 waits land on the tail. Only p50 has an interior optimum, at three, and it is the one column a fan-out change is usually reported against.

Two branches bought 5 seconds of p50 and 30 seconds of p95 for 13% more spend. Three bought 3 more seconds of p50 while p95 got worse than sequential. Beyond that, every additional branch made both latency and cost worse, since the limit is a fixed resource and contending for it harder does not produce more of it. The team shipped the limit at 2 with the token bucket sized to leave 25% headroom, which held through the following Monday.

The p95 column deserves more attention than the p50 column it is usually reported beside. Fan-out under a shared ceiling improves the median by moving work into a window that is not yet contended and degrades the tail by making contention worse when it happens, so the two move in opposite directions and a change that looks positive on the mean can be negative for the customers who notice.

Fan-out also changes what the previous chapter's budget means, and the policy has to be written down rather than inherited. Three branches each making two calls consume six steps of a twenty-step allowance in one turn, so a run that forks twice has materially less room than one that does not, and a wall-clock budget of six minutes is consumed at three times the rate when three branches are in flight. The arrangement that has behaved sensibly is to charge steps and tokens to the parent from a shared pool, since those are the dimensions measuring work done, while leaving wall clock on real elapsed time, since that is the dimension measuring the customer's wait. Per-tool write limits stay global to the run for the reason the race demonstrates.

Where parallelism is not the answer#

It is not a throughput mechanism. Running four branches at once inside one run does not let the fleet process more tickets per minute — it uses the same capacity in a shorter window, so under a saturated limit the fleet's throughput is unchanged and its latency distribution is worse. The reason is the ceiling arithmetic above: throughput under a token ceiling is the ceiling divided by tokens per task, and fan-out changes neither term. Throughput comes from needing fewer tokens per task, which is what caching and tiering did, or from having a higher ceiling, which is a procurement conversation.

It also does not fix a slow tool. The 8.2-second policy_retrieval from chapter two is 8.2 seconds whether one branch waits for it or three do; parallelism only compresses distinct waits that were previously sequential. Reaching for concurrency when a single dependency dominates the critical path produces a system that is harder to reason about and exactly as slow, and the breakdown from chapter two is what tells the two situations apart before the code is written.

Read-only fan-out over genuinely disjoint resources is the case where all four independence tests pass easily, and it is worth saying that this case is common and safe — six carrier_tracking reads for six different shipments share nothing but capacity, and capacity is handled by the bucket. Write fan-out rarely passes the second test without an idempotency key, and the correct default is that branches propose and a single merge step disposes.

Every lever in this topic now has a measured before and after: caching at 64% of input tokens, tiering at $0.152 to $0.105, the stopping rule at another 15%, a budget worth 2%, and concurrency worth 5 seconds at 13% more spend. Four of those five were taken and one was taken at a smaller setting than the one that first shipped, which is a reasonable outcome for a sequence of changes each of which looked free from a distance.

What none of those numbers settles is what to do when a change saves real money and costs real success. Every result so far has been either a clean win or a clean rejection, and the ablation table in chapter four already contains one row that is neither. That is the argument that actually happens, and it has an answer that does not depend on who is more senior.