Synchronised backoff and jitter

Synchronised backoff is what happens when independent clients compute identical retry delays. Nothing in the code is wrong: each worker takes a 429, reads the retry-after header, sleeps for the number of seconds the platform asked for, and tries again. The behaviour is correct in isolation and wrong in aggregate, because the platform gave every caller the same number and nothing in the client perturbed it.

This chapter reads a trace from a saturation test — a run that drives a model endpoint's token bucket to zero on purpose, with the containment the previous chapter describes — and follows one specific effect: the endpoint sharing that account stays degraded for nine minutes after the load generator has returned to baseline, in a pattern with a visible sixty-second period. Nothing is still generating load. What is still generating load is the backlog of work rejected earlier, coming back in waves, and the waves are a property of how the retry code was written rather than of anything the platform did.

What arrives, and what the client does with it#

The rejection is an HTTP 429 with a platform-specific error code — ThrottlingException on one platform, RateLimitError mapped from the same status on another — and a retry-after header expressed in seconds. Under sustained saturation the value the platform returns is typically stable; in this trace it was 12 on every rejection.

The first thing a trace like this shows is that most rejections never reach the application. A provider SDK's built-in retry policy converts a 429 into a delay and another attempt before returning anything to your code, so an application-visible error rate of 4% can sit underneath an HTTP-level 429 rate of 31%. That gap is the first thing to reconcile when reading any throttling trace, and the way to see it is to count attempts at the HTTP layer rather than calls at the SDK layer.

Reconciling it means noticing that the two percentages are not measurements of the same population. The application-visible rate has one call in its denominator for each unit of work. The HTTP rate has up to three, because the SDK turns one call into as many as three attempts before returning anything. The numerators differ as well: the application counts a failure only once every attempt has failed, while the HTTP layer counts every throttled attempt, including the ones a later attempt recovered from. So one figure is small because the retries mostly worked, the other is large because the retries mostly happened, and 4% next to 31% is two correct measurements of two different things. This is where a throttling investigation usually goes wrong, and the tell is somebody in the channel trying to establish which of the two is the real error rate.

The second thing it shows is the multiplication. Each queued task went through the SDK's three attempts, failed, and was requeued by the job runner with its own sixty-second delay and no attempt cap. Three HTTP attempts per task per queue attempt, with the queue attempting indefinitely, meant that 1,180 backlogged tasks produced roughly 3,540 HTTP requests every sixty seconds, all arriving inside a window of about two seconds.

Three attempts each from 1,180 tasks is 3,540 requests, and that multiplication is the whole of the amplification. No new work was created — the same 1,180 documents needed summarising before and after — and the load against the endpoint nevertheless tripled. Amplification of this kind compounds across layers rather than adding, because each layer multiplies whatever the layer below it produced, so the table below counts layers rather than components. Two retrying layers at three attempts each would have made it nine.

Why the waves land on the same second#

Every element of that schedule is deterministic and shared. The 1,180 tasks were all rejected within the same two-second interval, because that is what saturating a token bucket does to everything in flight. They all received retry-after: 12. They all waited exactly twelve seconds, retried together, were rejected together, and after their third attempt were requeued together with an identical sixty-second delay. Independent workers running independent code converge on one clock because the platform handed them all the same estimate.

Honouring retry-after exactly is what makes this worse rather than better, which is an uncomfortable result, because obeying a server's instruction is normally correct. A server-supplied delay is a good estimate of when capacity will exist and a bad choice of when you specifically should return, since every other caller received the same estimate. The value is right and the synchronisation is the problem.

This is an old result. Ethernet's collision handling in the 1970s ran into the identical structure — stations detecting a collision, all waiting, all retransmitting together, colliding again — and the fix there was to randomise the wait over an exponentially growing window. The distributed-systems version was written up carefully by AWS a decade ago under the name backoff and jitter, with the finding that adding randomness mattered more than the shape of the backoff curve.

The sawtooth is what breaks the interactive traffic#

The degradation is not really about the batch. That backlog was going to take a long time to drain regardless: 1,180 tasks at 28,000 input tokens each against about 1,712,000 tokens a minute of spare capacity after interactive traffic is 61 tasks a minute, which is roughly nineteen minutes of honest work.

Every figure in that sentence comes from the quota table in the previous chapter, and it is worth walking rather than accepting. The input-token bucket refills at two million tokens a minute, the interactive traffic spends a little under three hundred thousand of that, and what remains is the 1,712,000 a minute available to anything else. A task carrying a 28,000-token document takes 28,000 of that remainder, so the spare budget admits 1,712,000 divided by 28,000, which is 61 tasks a minute. A backlog of 1,180 tasks at 61 a minute is a little over nineteen minutes. Nothing about jitter, attempt caps or retry policy moves any of those four figures. The work is the work and the bucket refills at the rate it refills, which is the sense in which the drain time was never the problem.

What the waves change is how that capacity is consumed. A burst of 1,180 requests arriving inside two seconds empties the bucket in about 200 milliseconds, and for the remaining 59.8 seconds of the minute the bucket is refilling from zero while every arriving interactive request finds it near empty. The interactive endpoint is not competing with a steady batch; it is competing with a front-runner that takes the entire minute's allocation in the first fraction of a second. Sixty-one of the 1,180 arrivals are enough to empty it, and sixty-one arrivals out of a burst delivered over two seconds land within the first tenth of it, which is why exhausting the bucket is effectively instantaneous while refilling it takes the rest of the minute. Each wave is 30% smaller than the last, because 61 tasks succeed each time and the job runner's own failures eventually retire some, so the batch decays geometrically and crosses the threshold where it stops starving the interactive path after roughly nine waves.

That 30% is worth deriving rather than asserting, because it explains why recovery is slow. Of the 1,180 tasks in a wave, 61 are admitted on token budget. A further 290 or so leave the job runner across the drain for unrelated reasons — documents already partially processed, tasks whose deadline passed, a handful that hit the concurrency ceiling and timed out at the client — and the rest come back. The surviving fraction per wave sits around 0.70, so wave sizes run 1,180, 826, 578, 405, 283, 198, 139, 97, 68, and it is around the eighth or ninth wave that the burst stops being large enough to drain the bucket faster than the interactive traffic can draw from it.

That fraction is an addition and a division. Sixty-one tasks leave on token budget and roughly 290 leave for the unrelated reasons above, which is about three hundred and fifty gone from a wave of 1,180, so a little over eight hundred come back. Eight hundred and odd out of 1,180 is the surviving fraction of about 0.70, and every subsequent wave size is the previous one multiplied by it.

A geometric decay with a ratio near 0.7 is a slow way to disappear, and the reason is what the multiplication removes. Each wave loses a little under a third of what is left rather than a fixed count of tasks, so the number of tasks retired per wave shrinks along with the backlog, and the later waves clear far fewer tasks than the early ones did. The sequence above shows exactly that: the first step sheds several hundred tasks and the eighth sheds a few dozen. Nothing slows down — the same fraction leaves every time — and the absolute progress falls away regardless, which is why the tail of the recovery is longer than anybody watching the first wave would predict.

The same sequence says how long the recovery takes. The ninth wave is 68 tasks against a first wave of 1,180, a fall of considerably more than an order of magnitude, and there are eight multiplications between the two. Each of those eight is a minute long, because a human chose sixty seconds as a requeue delay in a configuration file, which is what converts an arithmetic property into nine minutes on a clock.

That is the diagnostic worth carrying: when a degradation graph has a clean period, look for a fixed delay somewhere in a retry path. An oscillation with a regular period is not something congestion produces by itself; a queue under load produces noise, and noise has no period. A period means a clock, and a clock in this kind of system is always a constant somebody typed. Fixed delays are everywhere in ordinary code — cron expressions, setInterval, queue visibility timeouts, health-check intervals — and each is a candidate synchroniser the moment more than one process shares it.

Jitter and a cap on attempts#

Two changes address the two mechanisms. Randomising the delay stops the convergence, and bounding the attempts stops the amplification.

import random, time

BASE = 0.5      # seconds
CAP  = 20.0     # seconds
MAX_ATTEMPTS = 4

def backoff_delay(attempt: int, retry_after: float | None) -> float:
    # Full jitter over the whole window, not a jittered point near the target.
    window = min(CAP, BASE * (2 ** attempt))
    if retry_after is not None:
        # Respect the server's floor, then spread arrivals over a window
        # of the same magnitude so callers do not return together.
        return retry_after + random.uniform(0, min(CAP, retry_after))
    return random.uniform(0, window)

def call_with_backoff(fn, *args, **kwargs):
    for attempt in range(MAX_ATTEMPTS):
        try:
            return fn(*args, **kwargs)
        except Throttled as exc:
            if attempt == MAX_ATTEMPTS - 1:
                raise                      # give up; let the caller shed load
            time.sleep(backoff_delay(attempt, exc.retry_after))

Full jitter — a uniform draw across the entire window rather than the window's endpoint plus a small perturbation — is the variant that performs best in the published comparisons, and the reason is that a small perturbation around a shared centre is still a wave, just a slightly wider one. Adding 10% jitter to a twelve-second delay spreads 1,180 requests over 2.4 seconds, which empties the bucket in 2.4 seconds instead of 0.2 and changes almost nothing. Drawing uniformly from zero to twelve spreads them over twelve seconds, and the tail of that distribution keeps arriving while the bucket refills.

The 2.4 seconds is ten percent either side of twelve, which is a window 2.4 seconds wide, and twelve seconds is five times that. What makes the difference is not that five is a large multiplier. It is that the width of the arrival window only starts helping once it is comparable to the time the bucket needs to refill enough budget for the next request, and 2.4 seconds is not. Below that point the burst still arrives faster than the bucket can absorb it and every caller after the first few dozen is throttled exactly as before, so the perturbation buys a slightly wider wave and no change in outcome. This is the reason the published comparisons favour full jitter so decisively: the useful quantity is the width of the window relative to the refill, and a small percentage of a delay is small on that scale whatever the delay was.

The retry_after branch deserves a note. The server's value is a floor and should be respected, because returning before it is guaranteed to fail and wastes a request against the request-rate bucket; the randomness is added on top of it rather than inside it. That is the precise sense in which honouring the header is right and honouring it exactly is wrong — the instruction is a statement about when capacity exists, not a statement about when you personally should return, and the second is the part every caller has to decide for itself if they are not all to become synchronised again.

Capping attempts is the other half. MAX_ATTEMPTS = 4 combined with a job runner that gives up after three requeues bounds total HTTP attempts per task at twelve, which gives the retry traffic a finite integral. Four attempts per queue attempt across three queue attempts is twelve requests per task in the worst case, against an arrangement whose product was three times a number with no upper bound in it at all.

Both bounds show up directly in the attempt counts a rerun produces, and the comparison table below is arithmetic rather than measurement noise. Nine waves of 1,180 tasks at three attempts each is just under thirty-two thousand HTTP requests across the drain, which is where a figure like 31,900 comes from. The same 1,180 tasks under a four-attempt cap, arriving spread out enough that most succeed on an early try, come to something a little over 4,700. The billed count does not move between the two, because the number of inferences the platform actually performed is one per task either way. An uncapped retry loop turns a transient ceiling into a permanent load source, and the tasks it is retrying are ones your users stopped waiting for several minutes ago.

Backoff does not create capacity#

Backoff and jitter are a scheduling fix for a capacity problem. If sustained demand exceeds the ceiling, no arrangement of retries drains the queue, and the correct responses are pacing the producer, chunking the documents, moving the batch to a window where interactive traffic is idle, or buying more capacity. Retries smooth bursts; a client that retries politely into a permanently saturated endpoint has converted an error into a timeout.

There is a control that does address sustained overload, and it belongs alongside the backoff rather than instead of it. A concurrency limiter on the batch — a semaphore sized so it can never hold more than, say, twelve requests in flight — bounds the burst at source, so a wave is 12 requests rather than 1,180 and the interactive traffic is never starved even when the retry code is imperfect. Adaptive variants that shrink the limit when throttling is observed and grow it back slowly are better still, because they converge on whatever headroom actually exists rather than on a number somebody guessed. The client-side ordering to reach for is: bound concurrency first, jitter the retries second, cap the attempts third.

Jitter also does nothing for the non-retryable ceilings. An account-level service quota produces an error that will not succeed on a later attempt, and retrying it with beautiful exponential backoff is twelve wasted attempts and several minutes of user-visible delay before the same failure. Classify the error before deciding to retry it, and keep the retryable set narrow.

This is also why the argument for deleting retry code after a quota increase is backwards. A higher ceiling makes throttling rarer, which makes the retry path colder and less trustworthy, while leaving every mechanism in this chapter intact for the day the workload grows into the new limit or a neighbouring team's batch job arrives. Reserved capacity does not remove them either; it has its own ceiling and its own overflow behaviour rather than an absence of one.

Everything so far has treated the endpoint as a capacity resource. The other thing a cloud endpoint is, and the reason procurement chose it, is a location — a hostname in a region that a contract has something to say about. That property turns out to be looser than the hostname suggests.