Two retry layers multiply, they do not add

The provider's status page later called it a 30-second elevated error rate in one region. Between 14:02:10 and 14:02:40, most requests came back 503 with a retry-after of 2 seconds. The company's own incident lasted 4 minutes and 10 seconds, and the graph that explained why was the count of requests leaving the gateway toward the provider: a flat 400 per second before 14:02, a spike to 3,600 per second at 14:02:12, and a decay that did not return to baseline until 14:06.

The gateway had been deployed six weeks earlier in front of services that were written when calling the provider was their own job. Those services retry twice on 5xx with exponential backoff, which was correct behaviour before the gateway existed and was never removed. The gateway ships with retries on by default, also twice, also with backoff.

The arithmetic that turns three into nine#

Each caller request becomes up to three gateway requests. Each gateway request becomes up to three provider attempts. When the failure is total, every branch runs to exhaustion.

Nine attempts per user-visible request, from two policies that each look modest on their own. The general form is that stacked retry layers multiply their attempt counts, so three layers of the same policy — an SDK that retries, a service that retries, a gateway that retries — is 27. Load during a total failure is the product of the per-layer attempt counts, not the sum, and the intuition that "we retry twice, they retry twice, so worst case is a bit more than double" is wrong by a factor of three at two layers and a factor of nine at three.

The consequence that matters is not the wasted requests. It is that a provider having a bad thirty seconds is a provider under load, and the response of every one of its customers to elevated errors is to send more load. The brownout at 14:02 would plausibly have cleared on its own; what kept the error rate above 50% until 14:06 was 3,600 requests per second arriving at a capacity pool that had been sized for 400. Retry storms are a well-documented failure mode in RPC systems generally — it is why gRPC has retry throttling in the protocol layer and why AWS's guidance has said backoff and jitter for over a decade — and model providers are a particularly good place to hit one, because the requests are expensive to serve and slow to shed.

Settle it in one layer, and make the survivor smarter#

Chapter 1 put retries at the gateway in the ownership table. The reason it is the gateway rather than the caller is that the gateway can see something no individual caller can: the health of the provider across all traffic. A caller that gets one 503 has no idea whether it was unlucky or whether the vendor is down. The gateway has 400 requests per second of evidence, which is enough to open a circuit breaker — the switch that trips once the error rate crosses a threshold and refuses further attempts until a cool-off has passed — within a second or two and start failing fast for everybody, and enough to run a retry budget.

A retry budget is the part worth adding when you consolidate. Rather than "retry twice", the rule is that retries may consume at most a fixed fraction of successful request volume — 10% is the common default — enforced by a token bucket, meaning a counter that every retry draws one unit from and every successful request puts one back into. Under normal conditions the budget is never the binding constraint, and every retry that would help gets made. When the error rate crosses roughly 10%, the bucket empties and retries stop almost entirely, which turns the storm off precisely when it would do damage. The effect on the incident above, replayed against the same trace, is a peak of 440 requests per second instead of 3,600.

routes:
  - match: { model_prefix: "anthropic/" }
    retry:
      max_attempts: 3
      on: [502, 503, 504, connect_error]
      backoff: { base_ms: 250, multiplier: 2, jitter: full }
      budget: { ratio: 0.1, min_per_second: 5 }
      respect_retry_after: true
    circuit_breaker:
      error_ratio: 0.5
      window: 10s
      open_for: 5s

Turning the caller's retries off is the other half, and it is the half that gets skipped because it needs a change in nine repositories rather than one config file. Until it lands, the gateway's careful budget is being multiplied by whatever the callers do, and the budget only limits the layer it lives in.

Load shedding belongs in the same configuration, because a retry budget controls what the gateway sends and a concurrency limit controls what it accepts. During the incident the gateway held around 4,100 in-flight requests against a steady-state of 900, most of them waiting on attempts that would fail, and each one occupying a connection slot and a slice of memory. A hard cap on in-flight upstream requests per provider, with immediate rejection above it rather than queuing, keeps the gateway's own resource usage bounded and pushes the decision back to callers who at least know whether their request still matters. Rejecting fast is a better failure than accepting and holding, since a caller that gets a 503 in 2 ms can fall back, and a caller whose request sits in a queue for 30 seconds has already lost its user.

The one row where the caller keeps a retry is the last-resort case in which the gateway itself is unreachable, and even there it should be a single immediate attempt against a second replica rather than a backoff loop, because a gateway that is down does not recover faster for being asked repeatedly.

Timeouts have to travel with the request#

The second thing that goes wrong when a hop is introduced is that the caller's patience and the gateway's patience are set independently, and nobody notices until they disagree.

The caller in this system had an 8-second timeout, chosen because the product's loading spinner gives up at 8 seconds. The gateway had a 30-second upstream timeout and two retries with backoff. A slow provider therefore produces this sequence: the gateway dispatches attempt one at t=0, waits until t=30, backs off, dispatches attempt two at t=30.5. The caller gave up at t=8. Everything the gateway does after t=8 is work nobody will read, billed by the vendor at full rate, occupying a connection and a goroutine, and adding to the load on a provider that is already slow. This is the same orphaned-work problem that showed up as $610 of unrecorded retry spend in the previous chapter, seen from the other side.

The fix is to carry the deadline in the request and to treat it as a hard ceiling on everything downstream.

# Caller: send an absolute deadline, not a duration.
headers = {"x-deadline-ms": str(int((time.time() + 8.0) * 1000))}

# Gateway: never start an attempt that cannot finish inside it.
def attempt_timeout(deadline_ms: int, backoff_ms: int) -> float:
    remaining = (deadline_ms - now_ms()) - backoff_ms - 50   # 50ms return margin
    return remaining / 1000.0

if attempt_timeout(deadline_ms, backoff_ms) <= 0:
    raise DeadlineExceeded("no time for another attempt")

An absolute deadline rather than a duration is what makes this composable, since each hop subtracts its own elapsed time automatically and no layer has to guess how much of the budget the layers above already spent. The rule that follows is that a gateway never retries when the remaining deadline cannot accommodate another attempt, which in the 8-second case means at most one retry and usually none.

Streaming complicates the retry decision in a way worth stating plainly. Once the first content token has been forwarded to the caller, a retry is no longer transparent — the caller has already received partial output, and starting again produces either duplicated text or a stream that appears to restart. Retries on streaming routes should be restricted to failures before the first byte of content, and everything after that is an error the caller has to see. Any request whose generation triggered tool calls with side effects is in the same category, and an idempotency key generated by the caller and reused across the gateway's attempts is what keeps a retried request from being executed twice at the vendor.

What to measure, and what the numbers looked like#

Chapter 1 quoted 6 ms p50 and 41 ms p99 for the hop. Those figures are only meaningful if they are measured continuously and separated from the provider's own latency, which requires the gateway to record the upstream call's duration and subtract.

Four metrics carry the whole picture. Added latency, as total handler duration minus upstream duration, reported at p50, p99 and p99.9 — the p99.9 is where connection-pool exhaustion and garbage collection show up, and in this system it was 310 ms against a p99 of 41 ms, which is a factor of eight that no median would have revealed. Upstream attempts per caller request, as a histogram, because a mean of 1.02 is compatible with a small population of requests making nine attempts and the histogram's tail is the alarm. Remaining deadline at dispatch, which shows immediately whether callers are sending deadlines at all and whether the gateway is starting attempts it cannot finish. And the error rate when a dependency is unavailable, which cannot be observed from production traffic and has to be produced deliberately.

That last one is a fault-injection exercise rather than a dashboard: block the provider at the network layer for 60 seconds in staging and record what the callers do; do the same to the budget store from the previous chapter; do the same to the gateway itself. The first two produce a number. The third produces an argument, because the answer for most callers turns out to be that they have no behaviour at all for a gateway that is not there.

Where this stops being the explanation#

Retry amplification is one member of a family and it is the easiest to fix. Load also multiplies through queues: a gateway with a 500-deep request queue and a caller that times out at 8 seconds will happily dispatch requests whose callers left minutes ago, which is the same orphaned-work failure with no retries involved at all. It multiplies through humans, since a chat UI that fails visibly gets its users clicking retry, and no amount of server-side budget touches that. It multiplies through fallback routing, where a policy of "on failure, try the other provider" turns one vendor's incident into load on a second vendor that your capacity was never provisioned for.

Consolidating retries also does not make the gateway a source of reliability by itself. What it does is stop your own architecture from converting a small provider incident into a large one. The incident in this chapter was still, at its root, thirty seconds of vendor 503s, and the best possible retry configuration would have left the callers seeing thirty seconds of elevated errors.

The failure mode that has no upstream cause at all is the gateway going away on its own — during a deploy, a bad config push, a certificate expiry. Every caller is behind it, and until now the assumption throughout has been that they will handle that sensibly.