Circuit breakers and load shedding

A brownout is a dependency that is degraded rather than down: answering some calls slowly, rejecting the rest, and doing so for long enough that no single request can wait it out. Twenty minutes of it is an ordinary event on a hosted model API, and it is the case that every policy in this topic so far handles correctly at the level of one call and badly at the level of a fleet.

The reason is structural. A retry policy has no memory by construction. Call number 8,000 begins with no knowledge that calls 1 through 7,999 just failed, so it pays the full cost of discovering the outage for itself: a connect, a wait for a deadline or a rejection, a backoff, a second attempt, a second failure. Multiply that by the arrival rate and the discovery cost becomes the dominant load on the system, which is the mechanism by which a degraded dependency turns into a saturated caller. There is a second, quieter cost: every attempt that reaches the provider competes with the attempts of every other tenant doing the same arithmetic, so aggregate demand on a service already short of capacity goes up at exactly the moment it needs to go down. Retries are countercyclical in the wrong direction.

Discovery cost, and the state that removes it#

Work the arithmetic on an interactive path taking 120 requests per second with a per-call policy that is doing everything right: each failure classified against the documented error type, every reported reset honoured, every invented delay jittered, one retry inside a 15-second ceiling. At a 35% failure rate that is roughly 42 requests per second in retry state for up to 15 seconds each, which is 630 concurrent calls waiting on a service that has already told them, thousands of times, that it has no capacity. A bulk consumer does the same thing on its own scale: each item spends up to 10 minutes across five attempts before being dead-lettered, so over twenty-two minutes it processes a fraction of its normal throughput and dead-letters items that were merely early.

Those two numbers are the concurrency product from earlier in this topic, applied to the part of the traffic that is failing. Of 120 requests a second, a 35% failure rate means about 42 a second end up in retry state, since 120 multiplied by roughly a third is roughly 42. Each of those holds a worker for as long as its policy allows, which is up to 15 seconds, and 42 arrivals a second each held for 15 seconds is 630 in flight. The "up to" is doing real work in that sentence: 630 is the ceiling rather than the average, and the average depends on how many of the retries succeed early. What makes the ceiling the number to design against is that a brownout is exactly the condition in which retries do not succeed early.

The uncomfortable part of the arithmetic is that every one of those 630 waits is buying information that the process already has. Call number 8,000 is sleeping in order to learn something calls 1 through 7,999 established minutes ago. The cost of discovering an outage is being paid once per request when the fact discovered is the same fact every time, and that is the specific inefficiency a breaker removes — not the failures, which are the provider's, but the repeated purchase of one piece of knowledge.

The standard response is well enough understood to have a canonical reference: the circuit breaker, described by Michael Nygard in Release It! in 2007 and popularised in this form by Netflix's Hystrix a few years later, borrowed from electrical engineering for the property that matters here — a breaker that has tripped stays tripped without anyone deciding again, and it protects the wiring rather than the appliance.

The state that makes this work is shared across calls, which is the whole difference from everything in this topic so far. One process observes the failures of many requests, forms a view about the dependency's health, and imposes that view on requests that have not been attempted yet. A call arriving while the breaker is open does not fail after 15 seconds of trying; it fails in under a millisecond without leaving the process.

Three thresholds in that diagram carry all the tuning, and each fails in a specific way when set carelessly. The failure ratio needs a minimum call volume alongside it, because two failures out of three calls is 67% and means nothing; without a volume floor a low-traffic endpoint trips its breaker on statistical noise several times a day. The step under that is why a ratio computed from a handful of calls is not a measurement. The quantity the breaker wants is the dependency's underlying failure rate, and a ratio over three calls is an estimate of it drawn from three observations. An estimate from three observations can only take four values — none failed, one, two or all three — so it cannot express anything between them, and a dependency failing a small fraction of the time will still produce two failures out of three often enough to be unremarkable over a day of traffic. The volume floor is not a conservatism about tripping too eagerly. It is the condition under which the ratio means anything at all. The cool-down sets how long the dependency gets to itself before you look again, and doubling it on each failed probe cycle — capped, so recovery is never more than a couple of minutes away — is what keeps a long outage from becoming a repeated poll. The probe count decides how confidently you close.

Two details of how the ratio is computed matter as much as the number it is compared against. It is measured over a window — thirty seconds here — rather than over the incident, so what opens the breaker is the failure rate during some thirty-second stretch and not the average anybody quotes afterwards. A degradation that averages a third of calls failing across twenty minutes is not a steady third of calls failing every second of it; it is worse in some windows and better in others, and the breaker responds to the worst window it sees rather than to the summary. And the ratio counts only what the filter in the code below lets through, so it is a failure rate among calls that carry evidence about the dependency, not among all calls. Both details mean the number in the diagram cannot be compared directly against an error rate on a dashboard, which is measured over a different window and a different denominator.

Probing beats reopening on a timer#

The tempting simplification is to skip the half-open state: wait ten seconds, then resume normal traffic and see what happens. It fails for the same structural reason as a fixed retry schedule, which is that it moves the entire fleet at once. A breaker that reopens to full traffic sends 120 requests per second at a provider that has been given ten seconds of quiet, and if it was not ready, the fleet re-trips its breakers together and repeats the cycle indefinitely.

Half-open is an admission-control state rather than a delay. A small number of calls — five, or a percentage of arrivals, whichever is smaller — is allowed through while everything else continues to fail fast, and the decision to close is made on the outcome of those probes. The cost of being wrong is five failed requests instead of 1,200. That second figure is one cool-down's worth of arrivals at this path's rate: 120 requests a second across ten seconds is 1,200, which is the scale of what a fleet puts back on the wire every time it reopens on a timer into a dependency that was not ready. The probe version stakes five requests and then decides. Nothing about the dependency differs between the two cases. The difference is entirely in how much is committed before any evidence comes back. Requiring four of five successes rather than one of one also rejects a dependency that is flapping, which matters because a provider recovering from an overload frequently serves a few requests successfully before falling over again.

class Breaker:
    def __init__(self, threshold=0.5, min_calls=20, cooldown=10.0,
                 max_cooldown=120.0, probes=5):
        self.state, self.cooldown = "closed", cooldown
        self.base_cooldown, self.max_cooldown = cooldown, max_cooldown
        self.window = SlidingWindow(seconds=30)
        self.threshold, self.min_calls, self.probes = threshold, min_calls, probes
        self.opened_at, self.probe_results = 0.0, []

    def allow(self) -> bool:
        if self.state == "closed":
            return True
        if self.state == "open":
            if time.monotonic() - self.opened_at < self.cooldown:
                return False
            self.state, self.probe_results = "half_open", []
        return len(self.probe_results) < self.probes   # admit probes only

    def record(self, ok: bool, error_type: str | None = None) -> None:
        # Only provider-health signals count. A 400 is our bug, not theirs.
        if error_type in ("invalid_request_error", "authentication_error"):
            return
        if self.state == "half_open":
            self.probe_results.append(ok)
            if self.probe_results.count(False) >= 2:
                self.cooldown = min(self.cooldown * 2, self.max_cooldown)
                self._open()
            elif len(self.probe_results) == self.probes:
                self.state, self.cooldown = "closed", self.base_cooldown
                self.window.reset()
            return
        self.window.add(ok)
        if self.window.count() >= self.min_calls and \
           self.window.failure_ratio() > self.threshold:
            self._open()

The filter at the top of record is where this chapter depends on the classification the topic opened with. A breaker counts evidence that the dependency is unhealthy, and a malformed-request error is evidence that your last deploy is unhealthy, so counting it opens a breaker on a fault that failing fast already handles — and worse, takes down the healthy traffic sharing that breaker for as long as the bad deploy is live. Timeouts and connection errors count, 529 overload responses count, and 429 rate-limit rejections are a judgement call: they say your consumption is too high rather than that the provider is unwell, and a breaker keyed on them is really a crude client-side rate limiter. Keeping them on a separate breaker with a much higher threshold, or leaving them to the header-driven pacing from the previous chapter, keeps the two signals from contaminating each other.

Shedding and deferring are the same breaker, opposite responses#

Both paths need the breaker and they need opposite things from it, because the resource each one is protecting is different. An interactive path is protecting a person's next thirty seconds, and there is no version of a brownout in which that person's request gets served by the provider. A bulk path is protecting a delivery commitment measured in hours, and there is a version in which every one of its items gets served — later.

The row costing the brownout is where the difference between the two columns stops being a matter of degree. Twenty-two minutes is one thousand three hundred and twenty seconds, and at 120 requests a second that is a little over 158,000 requests arriving with nowhere good to go — every one of them turned into a degraded response, because on this path a request that is not answered now is not answered. The cell beside it is measured in minutes rather than in requests, and that is not an inconsistency in the table. The bulk path loses nothing but time, since its items are still in the queue when the breaker closes. The units differ because the scarce resource differs, which is the same observation the first row makes and the reason the two paths cannot share a policy.

Load shedding on the interactive path means the answer must exist without the provider. A cached response for a repeated question, a smaller model on a different provider, a deterministic template, or a clear message with a retry affordance are all acceptable; a spinner is not, because a spinner is a queue with a human in it. The measurable win is in the worker pool: at 120 requests per second with a 1-millisecond rejection, the chat path holds 0.12 concurrent workers instead of 630, which means every other endpoint in the service keeps working normally throughout. That is the same arrival-rate-times-service-time product as before, run with a service time three orders of magnitude smaller: 120 a second multiplied by a thousandth of a second is 0.12. Nothing was done about the arrival rate, and nothing needed to be. Occupancy is a product of two terms, and a breaker attacks the one that a retry policy inflates.

The honest objection to shedding is that most of what gets shed would have worked. At a 35% failure rate, roughly two calls in three were still being answered, and an open breaker refuses those along with the rest. Two things make that trade worth taking, and neither is obvious. The first is that a successful call during this episode took 9 seconds where it normally takes 700 milliseconds, so what is being given up is not the ordinary product — it is an answer that arrives after most people have stopped waiting for it. The second is that the cost of chasing those successes is not paid by the chat path alone. The 630 workers held come out of a pool that serves every endpoint, so the calls being protected are not only the ones being refused. A breaker converts a distributed, invisible cost into a local, visible one, and the visible one is easier to argue about because somebody has to choose what the degraded response says.

Deferring on the bulk path means the consumer stops rather than fails. Pausing the queue consumer preserves message visibility timeouts — the window a queue gives a consumer to acknowledge an item before deciding it failed and handing the item to someone else — keeps the dead-letter queue empty, and leaves the cursor where it is. The alternative, letting each item discover the outage, exhaust its attempts and dead-letter, spends the whole degradation producing items that need manual reprocessing and a report with holes in it, while adding load to a struggling provider. A pause is one decision made for 50,000 items instead of 50,000 decisions made once each. The condition for resuming is a probe too, and a cheaper one than the interactive path's: a batch job can afford a single small throwaway request every 30 seconds to test the water, rather than gambling real items on it, which costs a handful of tokens and answers the question definitively.

Where breakers misfire#

A breaker shared across every call in a process trips for the whole process, so an outage confined to one model or one endpoint takes down calls that would have succeeded. Keying breakers per route, per model, or per provider — the bulkhead idea, borrowed from ship compartments and the sibling pattern to the breaker in Nygard's book — restores the isolation, at the cost of each breaker seeing less traffic and therefore needing a lower volume floor.

The multi-instance case is the honest limitation. With 40 stateless instances each holding its own breaker, a brownout trips 40 breakers independently and recovery admits 40 × 5 probes rather than 5, which is 200 requests against a provider that may not be ready. The multiplication is the whole of the problem: each instance believes it is running the careful version, and the careful version was designed around a probe count that assumed one observer. Forty observers each staking five requests stake two hundred between them, and none of them can see the other thirty-nine. Sharing breaker state through Redis fixes the arithmetic and introduces a dependency that is itself in the request path, which is a poor trade for most services. That last point deserves its reason rather than an assertion: a shared breaker has to be consulted before every call, including the calls it is going to allow, so a store that is slow adds its latency to healthy traffic, and a store that is unavailable forces a choice between failing every request and ignoring the breaker entirely. The component installed to protect against a dependency failure has become a dependency. The practical compromise is per-instance breakers with a probe allowance scaled down as the fleet grows, accepting that fleet-wide probe volume is approximate.

The state of the art has moved somewhat past the binary breaker. Adaptive concurrency limiting — Netflix's concurrency-limits work and the TCP-congestion-style algorithms it borrows from — replaces open-or-closed with a continuously adjusted in-flight limit derived from observed latency, so a dependency that is 40% degraded receives 60% of the traffic rather than all or none. That is a better fit for a brownout, which is a partial failure by definition, and it is meaningfully harder to reason about during an incident. A breaker with well-chosen thresholds is the version most teams should have first.

The belief to guard against once one is installed is that a breaker which never opens is reporting good health. It may be, and it may equally be reporting that its threshold sits above anything the dependency has ever done, that its volume floor sits above the endpoint's traffic, or that the failures it is seeing are being filtered out at the top of record as somebody else's fault. All three produce the identical observation — no trips — and only one of them means what it appears to mean. The check is therefore not the trip count. It is whether the breaker has ever opened in a drill, and what the failure ratio and the call volume in its own window actually read during the last degradation the service lived through.

The move that a sustained degradation calls for is therefore different in kind from anything a retry schedule can express, because it is a decision made once about a dependency rather than once per call. What it costs is the willingness to serve something worse than the real answer on the interactive path, and the willingness to serve nothing at all for a while on the bulk path, in exchange for a service whose other endpoints stay up and a job whose items are still where it left them.

None of that makes the deferred work safe to replay. When a paused batch resumes at item 8,200, some of the items around the pause point were sent to the provider and timed out on the client side without anyone ever learning whether they were executed, and re-sending them is the last unexamined assumption in this topic.