One defect multiplied by a shared queue is an outage

The backlog tool did what it was asked. Support had 8,400 tickets open on the same shipping delay, so someone used the bulk-reply feature: the model read a sample of the tickets, drafted one reply, and the tool sent it to all 8,400 accounts overnight. The draft said the customer's order had been dispatched and included a tracking link built from a field that was empty for about a third of them. By 09:00 the support queue had 2,600 new inbound messages, several dozen customers had called their card issuer to dispute a charge for an order they now believed had shipped and hadn't, and the team spent two days sending a correction that made things worse for anyone who had already thrown the first email away.

The model was wrong once. It generated one draft, and by every measure the previous two chapters would apply, this was a single error inside a month of otherwise fine outputs — one bad generation against maybe 40,000 good ones, an error rate too small to register on any dashboard. The damage came from the tool that took one output and applied it 8,400 times.

Blast radius is about where the output lands#

The term comes from ordnance, where it means the distance at which a single detonation still causes damage, and it survives in operations for the same reason: the useful question about a failure is not how likely it is but how far it reaches. Two systems with identical error rates have completely different risk profiles if one writes its output into a field on a single request and the other writes it into a template, a cache, or a shared record. The rate is a property of the model. The radius is a property of the plumbing you attached it to, which means it is entirely under your control in a way that the model's accuracy is not.

The five paths differ in more than the multiplier. A cached wrong answer keeps producing harm for as long as the entry survives, so its radius has a time dimension: at a 24-hour TTL and 900 hits a day, one bad generation is 900 wrong answers and the fix requires an eviction rather than a redeploy. A wrong value written into a shared account record — a corrected billing address, a risk score, a summarised "customer context" note that every future agent reads — has effectively no expiry, and it also contaminates the inputs of everything downstream, which is the fifth path. That last one is worth separating because its radius is not a number you can count in advance: an agent step that consumes a wrong output can take an action, and the action can create another wrong input, so the radius is bounded by the chain's own stopping conditions rather than by any fan-out you designed.

Reading down the multiplier column suggests the cheapest intervention available on most features, which is to reduce the multiplier rather than the error rate. Generating a per-ticket reply instead of one template makes the incident above 8,400 independent errors at whatever the model's real per-item accuracy is — say 3% wrong, so about 250 bad emails, which is bad and is roughly two orders of magnitude better than 2,800. It costs more in tokens; at 8,400 calls against one, at roughly 1,800 input and 250 output tokens per call, that is real money rather than a rounding error, and it is a straightforward trade to put in front of whoever owns the budget.

The 1% that tells you something#

Where the fan-out cannot be removed — bulk actions exist because someone genuinely needs to answer 8,400 tickets — the cheapest available bound is to apply the output to a small sample first and look at what happened before releasing the rest. Send the generated reply to a randomly sampled 1% of the queue, which is 84 tickets, wait, and check the outcomes. That sample is a canary, a small population deliberately exposed first so that what happens to it answers the question for the remainder before the remainder is committed. If the reply is wrong in the way the real one was wrong, 84 customers are annoyed instead of 8,400, and the correction is a phone call rather than a two-day incident.

Three details decide whether that is a control or a ritual.

The sample has to be random over the queue, not the first 84 items. Queues are ordered by arrival, and arrival correlates with exactly the segmentation the previous chapter dealt with — the first 84 tickets are disproportionately one tenant, one region, or one time zone, so a canary drawn from the head of the queue can be entirely clean while the defect sits in a segment further down.

There has to be an outcome signal, and its latency has to be shorter than the time you are willing to hold the rollout. This is the part most implementations get wrong, because the canary is written as "send 1%, pause five minutes, send the rest" with nothing being measured during the pause. For a support reply, usable signals arrive within about thirty minutes: inbound replies to the canary batch, re-open rate on those tickets, negative-sentiment classification on the responses, hard bounces. If the only signal you have is next week's CSAT survey, then the pause is decoration and you should either find a faster proxy or accept that the canary is not a control.

The gate has to be automatic in the halting direction. A canary that requires an engineer to notice a dashboard has the same failure mode at 02:00 as no canary at all.

def bulk_apply(items, render, send, *, canary_frac=0.01, min_canary=50,
               observe, tolerance, wait_s=1800):
    """Send to a random sample, measure, then release the remainder or abort."""
    import random
    sample_n = max(min_canary, int(len(items) * canary_frac))
    shuffled = random.sample(items, len(items))
    canary, rest = shuffled[:sample_n], shuffled[sample_n:]

    body = render(canary)                      # one generation, reused across the batch
    batch_id = send(canary, body)
    signal = observe(batch_id, wait_s)         # e.g. {"replies": 9, "reopens": 4, "bounces": 0}

    adverse = (signal["replies"] + signal["reopens"]) / sample_n
    if adverse > tolerance:                    # tolerance from the baseline rate, not from zero
        raise CanaryFailed(f"{adverse:.1%} adverse on {sample_n} items; {len(rest)} held")
    return send(rest, body)

The tolerance argument is where the judgement sits. Sending 84 support emails produces some replies no matter how good the content is, so the threshold has to be set against the baseline reply rate for that queue rather than against zero. On this queue the baseline was about 6% inbound replies within thirty minutes; the bad template produced 31% on the canary batch, which is far outside anything noise explains at n=84. Setting the tolerance at 15% would have caught it and would not have fired on a normal batch. Deciding that number requires having measured the baseline first, which is a week of instrumentation that has to happen before the feature ships rather than after the incident.

Counting the radius before you ship#

The audit that would have caught the bulk-reply tool is mechanical, and it takes about an hour per feature. List every place the feature's output is written or transmitted, and for each one write down how many distinct people or systems will read that write, over what period, and what stops it mid-flight. The output is usually surprising in one specific way: features have more write paths than their designers remember, because the interesting one is the primary output and the dangerous ones are the incidental writes that were added later.

On the support tool the list came to four entries. The reply itself, radius equal to the batch size. A last_contacted timestamp on each account, radius 1 per account and harmless. A summary line appended to the account's activity feed, radius equal to every future agent who opens that account, which is where a wrong summary quietly poisons subsequent conversations. And a generated macro saved into the shared macro library, which was the sleeper: macros persist, and a macro that has been used 300 times has a radius of 300 and rising, with no event anywhere that says it was ever generated by a model.

Several reductions are available at design time and cost less than the mitigations you would otherwise need. Caches keyed per tenant instead of globally convert a cross-tenant radius into a per-tenant one, at the cost of a lower hit rate. A hard cap on batch size — bulk actions apply to at most 250 items, and a batch of 8,400 must be split into 34 explicit confirmations — reintroduces a human decision every 250 items without prohibiting the workflow. Writes to shared records can be routed to a separate, clearly-labelled AI-generated field rather than overwriting the human-maintained one, so the prior value survives and downstream readers can choose. Each of these narrows the radius without touching the model.

Where a change is rolled out rather than a single batch sent, the same logic extends into a staged release, and the stages should be sized from exposure rather than from habit. Going 1% → 10% → 50% → 100% on a feature serving 200,000 monthly requests exposes about 2,000, then 20,000, then 100,000 requests to a defect present from the start, so the first stage is the only one that is genuinely cheap and it is the one people cut short. The hold at each stage has to exceed the latency of the signal that would reveal the problem; if the fastest reliable indicator is a support contact and support contacts arrive a median of four hours after the interaction, then a 30-minute stage is measuring nothing, and the release is a big-bang deploy performed in four instalments.

Staged rollout as a discipline came from deployment engineering, where the canary release replaced the maintenance-window cutover for essentially this reason — the cost of finding a defect scales with how many users met it, so meeting it with a small population first is strictly better as long as you can detect and roll back. What is new in the model case is that the rollback is often incomplete: you can stop serving a bad prompt in seconds, and the outputs it already wrote into records, caches and inboxes stay where they are. The staged release bounds the population and leaves you with a cleanup, which makes knowing exactly what was written during the stage — and being able to enumerate it from a log — part of the rollout design rather than an afterthought.

What bounding the radius does not buy you#

A canary bounds how many people receive a wrong output; it does nothing about the ones who already did. The 84 customers in the sample got the email, and email does not come back. Every technique in this chapter reduces exposure by a factor and none of them takes the exposure to zero, which is why they sit alongside rather than instead of the controls that keep a single action recoverable.

Blast radius is also frequently confused with rate limiting, and the two solve adjacent problems. A rate limit on the send endpoint — say 200 emails a minute — would have turned a one-hour incident into a seven-hour one and given someone a chance to hit the stop button, which is worth having. It does not reduce the radius; it stretches the same radius over more time and buys a detection window. That is a useful property to design for deliberately, and a poor one to rely on accidentally, since the window is only useful if something is watching it.

The last limitation is the most awkward. A small radius is a weak defence when the harm per instance is large enough. One wrongly cleared KYC alert, one deleted production table, one contract sent to the wrong counterparty — the radius is 1 and it is still the worst thing that happens that quarter. For those, the multiplier is not the lever; whether the action can be taken back is. That is a property of the action rather than of the model, and it has three concrete requirements that an interface cannot fake.