The blast radius of a retry

A retry is the client re-sending a request after a failure, usually inside a wrapper that every call in a service passes through. Whether that is a good idea depends entirely on the class of failure underneath it. A failure caused by a malformed payload — an unknown field, a tool schema the provider's validator rejects — is deterministic: the provider parses the JSON, validates it, and returns 400 before a model is loaded, before a token is sampled, before any scheduling decision is made. The second attempt sends the same bytes to the same validator and gets the same answer, and so does the fifth.

Everyone agrees on that in the abstract, and wrappers retry these calls anyway, because the retry rule is usually written once as "anything that is not a 2xx" and nobody revisits it. This chapter is about what that costs, and specifically about blast radius — the set of things damaged by a fault beyond the thing that was broken. The token bill for a retried 400 is zero, which is the first number anybody reaches for and the least interesting one available. The two that matter are the provider's per-organisation request allowance and your own worker pool, and neither of them appears on a cost dashboard.

What a five-attempt wrapper does to one failing endpoint#

The loop that produces the damage is unremarkable, and versions of it are eighteen months old in most services:

RETRYABLE = lambda status: status < 200 or status >= 300

for attempt in range(MAX_ATTEMPTS):          # MAX_ATTEMPTS = 5
    response = session.post(url, json=payload, timeout=30)
    if not RETRYABLE(response.status_code):
        return response
    time.sleep(BASE_DELAY * (2 ** attempt))  # BASE_DELAY = 1.0

Nothing in it is unreasonable in isolation. It retries on failure, it backs off between attempts, it stops after five. Applied to a permanently failing request, it turns a call that failed correctly in 210 milliseconds into a call that fails five times and holds a worker for 1 + 2 + 4 + 8 seconds of sleeping while doing it.

That sum is worth looking at rather than reading past, because its shape explains why a schedule like this gets expensive so much faster than the attempt count suggests. Each delay is twice the one before it, so the last delay on its own is larger than every earlier delay added together: one plus two plus four is seven, and the next term is eight. Adding a sixth attempt would therefore cost more than the whole schedule so far, and the same arithmetic runs backwards — cutting the attempt count by one removes more than half of the waiting. The four sleeps here come to fifteen seconds. Each failing call adds about a fifth of a second more, and the table below carries the whole hold as 15.2 seconds, against the 210 milliseconds the failure genuinely took to establish. Everything after this point uses that figure as the service time, so it is worth being clear where it comes from: to within a rounding, it is the sleeping.

The middle row of that table is the one that looks like the finding and is not. Zero billed tokens both ways is correct — a request rejected by a validator never reaches a model, so there is nothing to meter and nothing to charge — and it is the row a cost dashboard can see. The two rows either side of it are the ones that hurt, and neither of them is denominated in money. One counts requests against an allowance, the other counts seconds against a worker, and both of those are resources the broken endpoint shares with code that has nothing wrong with it.

Take a service with one endpoint at 140 requests per minute and 1,450 per minute across everything else, against an organisation limit of 2,000 requests per minute. Healthy, the total is 1,590 and comfortably inside the allowance. With the broken endpoint retrying, it alone draws 700 requests per minute from the same shared pool and the total reaches 2,150 — above the limit. Other endpoints begin taking 429s. Rate limits are metered per organisation, not per endpoint, and rejected attempts count against the budget exactly as successful ones do: the broken code path is not spending money, it is spending the allowance that healthy traffic needs.

Each step in that is small and none of them is skippable. Healthy traffic is 140 from the broken endpoint plus 1,450 from everything else, which is 1,590 against a ceiling of 2,000 — a little over four hundred requests a minute of headroom, the kind of margin that looks generous on a capacity review. Retrying multiplies one of those two terms and leaves the other untouched: five attempts per request turns 140 into 700, an increase of five hundred and sixty requests a minute all by itself. That increase is larger than the headroom, so the new total of 2,150 crosses the ceiling, and it crosses it by a hundred and fifty requests a minute that somebody has to be refused. Which somebody is not chosen by whose fault it was.

"Rejected attempts count against the budget" deserves its condition, because it is not true of every kind of rejection. The attempts at issue here are ones the provider accepted, read and refused: they arrived, they were admitted, and a validator looked at them. The meter has to run at admission, before anything downstream can know whether the payload is valid, because validity is a fact that only exists once the request has been parsed. A request turned away by the rate limiter itself is the other case — that one is being refused by the meter rather than counted by it. The class that hurts here is the class that got far enough in to be judged, and a retry loop against a permanent error manufactures nothing else.

The worker arithmetic is the larger half. Say the service runs a pool of 24 request-handling workers shared across every endpoint. Concurrency is arrival rate multiplied by service time, so 140 requests per minute — 2.33 per second — with a 210 ms service time occupies about half a worker on a normal day. With the retry schedule, service time becomes 15.2 seconds and the same arrival rate demands 35 workers.

That multiplication is the one piece of arithmetic here worth deriving rather than accepting, because every other number in the chapter is downstream of it. Requests arrive at some steady rate, and each one holds a worker for some length of time. Ask how many are in progress at a given instant. It is exactly those that arrived recently enough not to have finished yet — which is to say, those that arrived within the last service time, since anything older than that has already left. So if requests arrive at a rate of r a second and each holds a worker for s seconds, the number in flight is the number that arrived during the last s seconds, and that is r × s. The relationship has a name, Little's law, and the only thing it assumes is that the system is keeping up with its arrivals.

Both figures above come out of that product. The arrival rate is 140 a minute, and a minute is sixty seconds, so it is 2.33 a second. Healthy, the service time is 210 milliseconds, near enough a fifth of a second, and 2.33 multiplied by a fifth is under half a worker occupied — which is why nobody in the history of this service has thought about the size of the pool. Broken, the service time is 15.2 seconds, and 2.33 multiplied by 15.2 is a little over 35. Notice which factor moved. The arrival rate is identical: the same callers are sending the same traffic they always sent. Service time rose by roughly a factor of seventy, so the demand for workers rose by roughly a factor of seventy, and the product has no term in it for whether the extra time was spent working or sleeping.

The pool has 24. There is no configuration in which 35 concurrent requests fit into 24 workers, so the excess queues, and the queue serves every endpoint. A request to an untouched endpoint is not slow because that endpoint is slow; it is slow because it sits behind requests that are sleeping through a backoff schedule for an answer that will never change. Once queue depth exceeds what an upstream timeout tolerates, those callers start giving up and retrying too, which adds arrivals to a system already over capacity.

The 35 is a demand rather than an outcome, and that distinction is where the failure actually lives. Little's law describes a system that is keeping up, and this one is not, so there is no steady state left for it to describe. What happens instead is that arrivals exceed departures every second and the difference has to go somewhere, which is the queue. A queue that grows by a fixed amount each second does not settle at some larger size and stay there. It grows for as long as the condition holds, which is why a service in this state does not degrade into a slower equilibrium and wait to be rescued.

The belief that none of this can reach a healthy endpoint is worth stating plainly, because most services hold it without ever writing it down. The endpoints are separate code paths. They call different models with different prompts, and nothing in the /search handler imports anything from the /draft handler. All of that is true, and none of it is protective. Sharing a worker pool means sharing one scalar — the count of workers not currently occupied — and that count is decremented by whoever is holding one, for whatever reason, including holding one in order to sleep. Isolation between two endpoints is a property of not drawing from the same pool. Separate code is not separate capacity.

The final step in the sequence is the one that turns a slow service into a failing one, and it is a loop rather than a chain. Once queueing delay exceeds what an upstream caller's timeout tolerates, that caller stops waiting and re-sends. The re-send is a new arrival, so the arrival rate rises while the queue is already draining too slowly. A higher arrival rate lengthens the queue further, which pushes more callers past their timeouts, which produces more re-sends. Every term feeds the term that caused it. Nothing in that loop requires the original fault to still be present, which is why services in this state do not always recover at the moment the bad deploy is rolled back.

Two shared resources, then, and a third that lengthens the outage rather than widening it. The error rate on a fully broken endpoint is 100% either way, so retries do not make the alert fire sooner or louder. What they do is multiply the log volume by the attempt count and bury the single distinguishing line — the validator's complaint about the exact field it rejected — under five times as many copies of itself, on a dashboard that now also shows a queue backing up across every endpoint. That combination looks exactly like a provider incident, and diagnosis goes where the dashboard points.

The case for giving a 400 one retry#

The reasonable-sounding counter-proposal is to cap permanent errors at a single retry rather than zero. The argument is that some 400s are transient — a request body truncated in transit, a race where a resource was created moments after the check, a provider-side validation service having a bad moment — and one extra attempt is cheap insurance against a class of failure nobody has fully enumerated.

Take the cost side first, since "one retry is cheap" is where the reasoning starts. One retry doubles the request-budget draw and doubles the worker hold time for every permanently failing call. On the numbers above that is 280 requests per minute instead of 140 against the shared allowance, and a service time of about 1.2 seconds instead of 210 milliseconds. It is a much smaller blast radius than five attempts produced, and it is not nothing: the same deploy on a service running closer to its limits would still have pushed healthy endpoints into 429s.

The service time in that sentence is the schedule with only its first term in it: one attempt, one sleep of a second, one more attempt, which is where the 1.2 seconds comes from. Put it back through the same product and the worker demand is 2.33 multiplied by 1.2, which is under three workers out of 24. That is the honest shape of the counter-proposal — it costs about six times the occupancy of failing fast and still fits inside the pool, so it does not produce the outage described above. What it does do is take a failure that resolves in a fifth of a second and hand the caller back a second and a bit later, on every single request, in exchange for a repair that the paragraphs below show is rarely the one available.

The benefit side is where it falls apart. The failures the proposal worries about are real, but they are not 400s from the provider's validator. A truncated request body does not usually reach the validator at all — the connection fails, and that surfaces as a connection error with no HTTP status, which is a separate branch that should indeed be retried. A response that never arrived is likewise a timeout, not a 400. What is left, once those are routed to the branch they belong in, is the set of provider 400s that are genuinely non-deterministic, and that set is small enough to enumerate rather than to cover with a blanket rule.

It is not empty, and this is the honest part of the answer. There are 400s whose cause is not in the payload at all: an organisation whose data-retention configuration makes every request to a particular model invalid returns 400 invalid_request_error on a perfectly well-formed body, and a change to that configuration makes the identical request start working. There are gateways and proxies in front of the provider that emit their own 400s with an HTML body when they dislike something about a header. Neither is repaired by an immediate retry, but both are cases where "400 means your payload is wrong" is the wrong diagnosis, and a policy that says so flatly will send someone hunting through a request builder that has no bug in it.

Everything above concerns the class of error where retrying is straightforwardly wrong. It is the easy case: the correct number of attempts is zero, and the only open question is how much damage the wrong answer does. Two shared resources answer it — a per-organisation request budget that rejected attempts consume just as readily as successful ones, and a worker pool whose occupancy is arrival rate times service time, where the retry schedule is service time. Both of them belong to every other endpoint in the service as much as to the broken one, and neither is visible on the dashboard anyone looks at first.

The retryable classes are harder, because there the correct number of attempts is greater than zero and the interesting question becomes when. The same incident had a second act that makes the point: after the rollback, the service was catching up on a backlog, its 24 workers all hit the provider's rate limit within the same second, all backed off on the same fixed schedule, and all returned together one second later to be rate-limited again. The policy was by then classifying every error correctly. It was still producing a failure loop, because a schedule that every client computes identically puts every client back on the wire at the same instant.