Retry, circuit breaker and fallback

A feature built on a large language model rests on an HTTP API that belongs to somebody else, and without machinery in between, a provider unavailable for forty minutes is a feature unavailable for forty minutes. That machinery lives in one place: the gateway or client wrapper every model call passes through, between your product code and the vendor SDKs.

Three mechanisms live in that gateway and this chapter is about the differences between them. A retry re-sends a request that failed, on the assumption that a second attempt will land better than the first; the pause between attempts is the backoff, and doubling it each time — one second, then two, then four — is the usual policy. A fallback chain is an ordered list of places a request can be served from: the primary vendor first, then a second vendor, then whatever sits below that, which is often a cached answer or a smaller model you host yourself. Serving from further down that list instead of returning an error is graceful degradation, the deliberate trade of answer quality for continued availability. A circuit breaker is a piece of shared state in front of a provider that counts recent failures and, once too many of them arrive in too short a window, stops sending that provider traffic at all for a cooldown period, so requests skip straight to the next entry in the chain. If you have used a resilience library — Polly in .NET, resilience4j in Java, or any of the wrappers built on them — you have met retry policies and breakers under exactly these names. What changes here is the dependency they point at, which fails in ways a database call does not and which has a second and a third address you can send the work to.

What a missing breaker costs, measured#

Take a primary returning 500s on about 40% of requests to one endpoint, and a client that retries three times with exponential backoff before falling through to a secondary vendor. Availability holds, because the fallback fires on every exhausted call and serves a good answer, while p99 latency — the figure below which 99 of every 100 requests complete — moves from 2.4 seconds to 10.2 seconds. Nothing has failed by the definition the gateway is measuring.

The arithmetic behind that second number is worth doing by hand, because it explains the whole chapter. Start with what one failed attempt costs in time. A 500 comes back quickly, around 150 ms, because the provider is shedding load — rejecting requests immediately to protect the capacity it still has — rather than hanging on to them. A rejection is cheap for the provider to produce, which is the entire point of shedding, and cheap responses are exactly the ones that make a retry loop look harmless when somebody reads it in review.

Four attempts means three sleeps rather than four, because the loop only pauses between attempts and the last failure falls straight through to the fallback. Doubling from one second gives sleeps of one, two and four seconds, which add to seven. The four rejections themselves cost 150 ms each, which comes to six tenths of a second in total. A request that fails every attempt has therefore spent seven and six tenths of a second before it has asked anybody who might actually answer it.

Whether that path is the one p99 measures depends on how often it is taken. Treating the attempts as independent, the chance that a single request burns all four is 0.4⁴ — four tenths multiplied by itself four times. Four tenths of four tenths is a bit under a fifth; another factor of four tenths takes that to a little over six in a hundred; the last one brings it to roughly 2.6%. Now recall what the percentile is: p99 is the latency below which 99 of every 100 requests complete, so it is set by the slowest one per cent of them. The comparison that decides the whole question is therefore between 2.6% and the 1% line. Because 2.6% sits above it, the group of requests taking the full path is more than large enough to fill the slowest one per cent on its own, so the slowest one per cent is drawn entirely from inside that group and p99 is that group's latency rather than a blend of anything. Had the full-path share come out under the 1% line, the same requests would still have been slow and p99 would have been set by something faster, which is the sense in which this arithmetic decides what the graph shows rather than what the system does.

The rest is addition. Four failed attempts and their sleeps are seven and six tenths of a second, and the secondary then takes 2.6 seconds to generate an answer, which totals about 10.2 seconds. Every part of that total except the last 2.6 seconds is time the request spent being refused or asleep.

The independence assumption is the weak step in that derivation, and it is worth naming rather than smuggling past. Attempts a second or two apart against a provider having a bad morning are not independent events: whatever is wrong at the first attempt is very likely to still be wrong at the fourth, so the real share of requests that burn the whole chain is higher than 2.6% rather than lower. Independence is therefore the conservative direction for this particular argument. It produces the smallest defensible estimate of how often the full path is taken, and even that estimate is more than enough to put the full path at p99.

The same arithmetic says which part of the latency distribution moves, and it is not the middle of it. Three requests in five succeed on the first attempt and are served in the primary's ordinary time, so the median request never enters the retry loop and the median latency does not move at all. Most of the rest succeed on a second or third attempt, having paid one or three seconds of sleep. Only the last few per cent pay the entire 10.2 seconds. A retry-and-fallback arrangement changes almost nothing in the middle of the distribution and piles everything it does change into the tail, which is why the graphs a team watches by habit — error rate, availability, mean latency — report an ordinary morning.

Availability survived and the latency budget did not, and the two measurements disagree because they count different things. Availability counts outcomes: a request that eventually returns a good answer is a success whether it took two seconds or ten. Latency counts time, and time inside an in-flight request is a resource that something else is not using. The reason the tail moved is that every request paid, on its own account, to rediscover a fact that had already been established several thousand times that morning.

The code that produces this is not exotic. It is what almost every LLM gateway looks like in its first month:

def complete(request):
    for attempt in range(4):
        try:
            return primary.complete(request)
        except ProviderError:
            if attempt < 3:
                time.sleep(2 ** attempt)   # 1s, 2s, 4s
    return secondary.complete(request)

There are two mechanisms in those nine lines and it is easy to call them one thing. There is a retry, which decides how hard to press the primary, and there is a fallback, which decides where to go when pressing stops. What the function does not have is memory. Every call enters it knowing nothing about the ten thousand calls that preceded it, so every call re-runs the same experiment and pays the same seven seconds to reach the same conclusion. The missing mechanism is the one that holds the verdict.

The bet each mechanism makes about the failure#

A retry is a bet about the shape of the failure. It assumes the failure is transient — it will be gone in a second or two — and that it is specific to this request, or at least to this moment on this connection. That bet is correct surprisingly often. A TCP reset mid-stream, a single 503 from a node that is being drained, a load balancer that closed a pooled connection between your health check and your POST: these are one-request failures, and a second attempt succeeds. On one production gateway, 0.41% of calls to the primary failed on the first attempt in a normal week, and 97% of those succeeded on the second. Retry converted a 0.41% error rate into a 0.012% one for the cost of a few milliseconds on a small fraction of requests.

That last figure is the product of the two before it, and multiplying it out is the whole case for keeping a retry. The calls a user sees fail are the ones that fail twice, which is the remaining three in a hundred of the 0.41% that failed once, and three in a hundred of forty-one hundredths of a per cent is about 0.012%. Nothing about the provider changed between those two numbers. A second attempt simply moved thirty-three of every thirty-four failures out of the column the user can see, and it did so by spending a few milliseconds on the 0.41% of requests that needed it and nothing at all on the rest.

A circuit breaker makes the opposite bet on the same evidence. It assumes the failure is shared across requests and durable — that whatever is wrong will still be wrong in ten seconds — and it concludes that the expensive thing is not the failure itself but the per-request cost of rediscovering it. So it keeps counts over a rolling window, and once those counts cross a threshold it stops sending traffic to the primary altogether for a cooldown period. New requests fail instantly at the breaker, in microseconds, and go wherever the fallback points. The name comes from household electrics, where the breaker's job is not to fix the short but to disconnect the circuit fast enough that the wiring behind it survives.

The step that makes that possible is the one the retry loop above does not have, and it is worth spelling out because it is the difference between microseconds and ten seconds. A failure is information, and information can be written down. Writing it down means keeping a count somewhere that outlives the request that produced it, so that the next request can read the conclusion instead of re-deriving it. Reading a count is a memory access; re-deriving it is four network attempts and seven seconds of sleeping. Once one request has paid to find out that the primary is refusing traffic, every request behind it can have that answer for free — but only if there is a place to put it. That is why the last row of the table below, the one about state, is the row that decides whether you have a breaker.

That distinction gets folded away when a team lists retry, breaker and fallback as three grades of the same idea. The breaker is not a retry with a limit. A retry with a limit is still a retry: it presses this request, on its own, without reference to any other. The breaker is a decision made once on behalf of every request in flight, and its whole value is that no individual request pays for it.

The fallback is neither of those. It answers where, and it has no opinion at all about when. A fallback chain declares an ordered set of destinations and nothing about the conditions under which traffic moves between them. That separation is the reason the three are worth naming separately: you can hold the same fallback chain constant while changing the retry policy and the breaker thresholds independently, and you will do exactly that during a real incident.

The last row is the practical tell. If your resilience layer keeps no state across requests, you do not have a breaker no matter what the wrapper class is called, and the latency collapse above is available to you at any time.

The belief that keeps a stateless gateway in production for years is worth stating plainly so that it can be checked. It is that a second vendor makes the feature reliable, and the availability graph appears to confirm it every morning. What the second vendor actually removes is one specific failure — the primary having nothing to offer — and it removes it by spending time. Whether the feature is reliable is a question about answers arriving inside the period somebody is prepared to wait, and a chain that always eventually answers is entirely compatible with a feature that has stopped being usable. The three internal teams timing out at eight seconds in the scenario below were not looking at an unavailable feature. They were looking at an available one that had become slower than their own deadline, which no availability metric is constructed to report.

The retry budget belongs in seconds, not in attempts#

A retry count is the wrong unit for the budget, because the quantity the caller cares about is a deadline. A gateway that allows four attempts while its callers time out at eight seconds, with nothing in the code connecting the two, will cheerfully begin a fourth attempt at second nine on behalf of somebody who gave up at second eight. Those attempts are pure waste in both directions: they hold a connection slot and a worker, they cost a generation that the provider bills for, and the result lands on a socket that closed a second earlier. Propagate the caller's deadline into the call — most HTTP clients take one, and where yours does not, a monotonic clock and one subtraction will do — and check before each attempt whether enough of it remains for that attempt to plausibly finish. The retry budget then becomes a number of seconds you can reason about against a latency SLO — the service level objective you have committed to, such as a p99 under three seconds — rather than a count that means something different at every failure rate.

Two details around the backoff are worth fixing at the same time. Exponential backoff without randomisation synchronises the fleet, because every client that failed in the same second sleeps and wakes together, producing a load pattern shaped like a comb aimed at a provider whose problem is very often load. Sleeping a uniform random amount between zero and the computed backoff — jitter, in the usual shorthand — costs one line and flattens that. And a retry after a read timeout is not free in the way a retry after a fast 500 is: the generation may have completed on the provider's side and been billed, so a request that times out at 30 seconds and is retried twice can cost three completions and produce one. Where the vendor supports an idempotency key — a value you attach to a request so that a repeat carrying the same key returns the original result instead of billing a second one — send one; where it does not, price the retry of a timeout differently from the retry of a rejection.

The reason those two retries are priced differently is that the two responses say different things about what the provider did with the work. A 500 arriving in 150 ms is a statement that no generation was attempted: nothing was computed, so nothing is billable. A read timeout is not a statement at all. It is silence, and silence is entirely compatible with the generation having completed on the provider's hardware a moment after you stopped listening. The tokens were produced, the accelerator was occupied for the duration, and the meter ran. An initial attempt plus two retries is therefore three generations commissioned, of which you can use one, and the two you discard cost the same as the one you keep.

Pricing each mechanism against its own absence#

Suppose there is only one provider, so the fallback chain degenerates into a degraded mode of some kind — a cache, a smaller self-hosted model, or an honest error. You can keep two of the three mechanisms. Working out which to drop forces you to price each one, and the prices are not comparable.

Dropping the retry raises the user-visible error rate to roughly the raw per-call failure rate: on the numbers above, from 0.012% to 0.41%, a factor of thirty-four. That is a real regression and it is also a bounded, measurable, continuously visible one. It shows up on the same graph you already watch, it degrades gracefully as the provider's baseline health changes, and it never converts a small problem into a large one.

Dropping the breaker prices differently, because its absence is free until the moment it is catastrophic. During normal operation a breaker on a healthy provider does nothing observable. During a sustained failure its absence multiplies per-request latency by the size of the retry budget, and that latency is spent holding resources: connection pool slots, worker threads, request contexts, the memory of every buffered prompt. A gateway sized for 400 concurrent in-flight requests at 2.4 seconds each will saturate at a far lower throughput once those requests take 10.2 seconds, and then it begins rejecting traffic it could have served, including traffic for features that were never touching the failing provider. On top of that, the retries you keep sending are amplification: four times the request volume aimed at a provider whose problem is very often capacity in the first place.

Two steps in that paragraph are usually taken on trust and are worth walking. The first is how slower requests become rejected ones. A server with a ceiling on concurrent in-flight requests can complete, each second, that ceiling divided by the time each request occupies a slot — which is not a rule about gateways but about anything with a fixed number of places and a queue in front of them. At 400 slots and 2.4 seconds a slot, some number of requests per second is the most the gateway can retire. At 400 slots and 10.2 seconds a slot, each request holds its place roughly four times as long, so the same gateway retires roughly a quarter as many. Nothing about the hardware changed and nothing about the arriving traffic changed. Arrivals are now above departures, the queue grows until it hits whatever bound it has, and requests are refused at the door. What gets refused is whatever arrives next, sorted by nothing, which is how a fault confined to one vendor's endpoint takes down endpoints that never called it.

The second step is why the retries make the provider's own problem worse rather than better. A provider shedding load is a provider with more work arriving than it can perform. Each user request that used to produce one call at that provider now produces four, so a fixed rate of arriving users becomes four times the rate of arriving calls. That quadrupling of load on the thing that is already over capacity is not a side effect of the retry policy; it is the retry policy working exactly as written, in every client that implemented the same sensible version of it, at the same moment, for the same reason. The loop is closed: shedding causes retries, retries cause more shedding. A breaker is the only mechanism in this chapter that opens that loop, because it is the only one that stops generating calls when the answers stop being useful.

Dropping the fallback converts a provider incident into a total feature outage for its full duration, with no partial credit. Nothing is amplified and nothing else breaks, but for forty minutes the feature is gone.

The defensible answer is to drop the retry, and to say why in those terms: its loss is a bounded increase in an error rate you already graph, while the loss of either of the others turns a vendor's bad hour into a structural failure of your own service — a latency collapse that spreads to unrelated endpoints, or a complete outage with no degraded path. It is also the one whose job is most nearly done by something else, because a breaker in the half-open state is already probing the primary on a schedule, and a fallback that returns quickly gives you a cheap second attempt at a different address. That reasoning matters more than the choice; an answer that ranks the three by "strength" has not priced anything.

An error is not evidence that the primary is failing#

Everything above assumed a fact the code cannot actually observe: that an error means the primary is failing. The breaker counts failures over a window, and the entire behaviour of the system rests on which responses get counted. A 429 counted as a failure will trip a breaker during your own traffic spike and shove that spike onto a secondary that is sized for a tenth of it. A 400 counted as a failure will trip a breaker because of a bug in your own request construction, and the secondary will reject the same malformed body. A content-filter refusal counted as a failure will trip a breaker on a single unusual document and burn the fallback's quota producing the identical refusal.

The failure all three mechanisms are actually built for is narrower than the set of errors a provider can return: a sustained rate of responses saying the provider cannot currently serve requests it would ordinarily serve, observed over a window long enough to distinguish it from noise. Everything else arriving as an error status is a different problem wearing the same clothes, and feeding it into a breaker produces a mechanism that fires on your bugs and your traffic rather than on the vendor's bad hour.

The threshold, the window, the cooldown and the probe fraction are the easy part of configuring a breaker. Retry decides how hard to press one request, the breaker decides once for all of them and holds the verdict where every request can read it, and the fallback only says where the traffic goes when the verdict is bad. Keeping the three separate is what makes "which could we afford to lose" answerable at all: you can price a mechanism only once you know what it is for.

Which responses get counted is the part that decides whether the breaker helps you or hurts you, and no amount of tuning the four numbers recovers a classification that is wrong. That makes classifying the responses the first piece of work rather than the last, and it is where the next chapter starts.