Queueing a request the user has abandoned is not serving it

A retail client sends a written requirement three weeks before a flash promotion: roughly ten times normal traffic for about forty minutes, and no request may be dropped. Their shopping assistant normally handles 200 requests per minute against an account limited to 500,000 tokens per minute, at a mean of 1,200 tokens per settled request, which gives a serviceable rate of about 417 requests per minute. The promotion will offer 2,000. The obvious reading of the requirement is to put a queue in front of the provider and drain it at whatever rate the quota permits, and that design does satisfy the sentence the client wrote.

Work the arithmetic all the way through and the outcome is not a slow service. Over the forty-minute window the promotion offers 80,000 requests and the account can serve 16,680 of them, so 63,320 requests are still in the queue when the promotion ends. Normal traffic resumes at 200 requests per minute, leaving 217 requests per minute of drain capacity, so the backlog clears in 292 minutes — four hours and fifty-two minutes after the promotion is over. The last request enqueued waits about five hours for its answer.

Nobody waits five hours. The client's HTTP timeout is 25 seconds, so every request queued behind more than about 25 seconds of work — which, at 417 requests per minute, means everything past the first 174 in the queue — is eventually answered into a socket that closed hours earlier. That is 99.8 percent of the promotion's traffic. The system spends its entire token quota for five hours generating 76 million tokens of text that no human being reads, and the genuine post-promotion traffic queues behind the corpses. Meanwhile the queue itself is 63,320 request objects holding prompts that average 6 KB, or about 600 MB once framework overhead is counted, and if those prompts carry retrieved product context at 40 KB each it is 2.5 GB in a process that was sized for a few hundred in-flight requests. The failure the client will report is not "responses were slow"; it is "the assistant was broken all afternoon".

A queue is a promise measured in seconds of work#

The defect in that design is that the queue has a capacity expressed in requests, or in nothing at all, when the only unit that means anything is time. A queue holding 174 requests at 417 requests per minute holds 25 seconds of work; the same 174 requests when the mean drops to 200 per minute holds 52 seconds. What a caller cares about is the second number, so the queue's bound has to be derived from the drain rate rather than fixed at a request count.

Because everything in this topic is denominated in tokens, the drain rate is a token rate and the queue depth is a token count. A 500,000-token-per-minute budget drains 8,333 tokens per second, and a 25-second deadline means the queue may hold at most 208,000 tokens of reserved work — roughly 173 requests at 1,200 tokens each. Beyond that, a newly arriving request is provably going to miss its deadline before it is dispatched, and the honest response is to say so immediately rather than to accept it and disappoint it later. Saying so means returning HTTP 503, the status that reports a service temporarily unable to take the work, with a Retry-After header carrying the number of seconds the client should wait before trying again.

def admit(req: Request, ledger: TokenLedger, queue: Queue) -> Decision:
    hold = req.prompt_tokens + req.max_tokens

    granted = ledger.reserve(req.prompt_tokens, req.max_tokens)
    if granted is not None:
        return Decision.DISPATCH

    # No budget now: will this request still be wanted when there is?
    drain_per_sec = ledger.budget / 60
    wait_s = (queue.queued_tokens + hold) / drain_per_sec
    budget_s = (req.deadline - now()).total_seconds()

    if wait_s + req.expected_service_s > budget_s:
        return Decision.SHED          # 503 with Retry-After, immediately
    return Decision.ENQUEUE

The same predicate has to run again at dequeue, because a request that was admitted when the queue was 12 seconds deep can be stale by the time it reaches the front — the drain rate falls when the mix shifts to heavier requests, and a deadline that was comfortable at admission is not guaranteed at dispatch. Discarding at dequeue costs nothing beyond the memory already spent, and it protects the far more expensive resource, which is the tokens that would otherwise be spent generating an answer for an absent caller.

Choosing what to discard turns out to matter as much as choosing when. Under sustained overload the oldest request in a FIFO queue, drained first in, first out, is by construction the one closest to its deadline, so a FIFO discipline spends its capacity on exactly the requests least likely to be wanted, and goodput — answers delivered before their deadline — collapses toward zero even though throughput stays pinned at the quota.

LIFO, which reverses that order and dequeues the most recent arrival first, is the counter-intuitive entry and it is right for the reason that matters: during overload the most recently arrived request is the one whose caller is definitely still waiting. Netflix and Google have both published on serving the newest requests first under load for exactly this reason. Its cost is starvation of the requests at the bottom of the stack, which is acceptable when combined with a deadline discard that removes them rather than leaving them to rot, and unacceptable if those requests represent work that must eventually complete.

Ten times demand against a fixed ceiling means someone is told no#

The requirement as written — no request dropped — cannot be met by scheduling, because scheduling redistributes a fixed quantity of service and the quantity is short by a factor of nearly five. Roughly 79 percent of the promotion's requests will not receive a model-generated answer within their caller's patience, and the design question is which 79 percent and what they see instead. There are only a few levers, and they compose.

Applying the first three turns 2,000 offered requests per minute into 1,200 against a capacity of about 740, which sheds 38 percent rather than 79 percent, and the shed portion gets a designed experience rather than a spinner that never resolves. That experience is the part teams skip. A 503 that renders as a browser error page is worse than a queue; a 503 that the client turns into "our assistant is busy with the promotion — here are the offer details, or try again in 30 seconds" is an acceptable product state, and it requires the Retry-After value to be real rather than a constant, because the client will use it.

Verifying any of this before the promotion rather than during it means driving the whole path at 2,000 requests per minute against an account whose quota is deliberately scaled down, so that the ratio of offered load to ceiling matches what the real event will produce, and then asserting the things above: that shedding starts at the predicted queue depth, that the shed response renders as designed, that the queue's memory ceiling holds, and that the backlog is empty within a minute of the load stopping. That rehearsal is a specific piece of engineering with its own pitfalls, and chapter 7 is about doing it properly.

Shedding is not the same as failing#

A shed request is a deliberate, immediate, correctly-shaped answer that the service is at capacity, issued in single-digit milliseconds and costing no tokens. A failure is a request that consumed budget, occupied memory, and produced nothing. Conflating them makes shedding look like an admission of defeat, when it is the mechanism that keeps the served fraction fast: the 21 percent that get through during the promotion are answered in under three seconds precisely because the other 79 percent were not allowed to compete for the same quota.

Load shedding is also a different mechanism from the per-tenant rate limiting of 15.5, and the two are often wired together incorrectly. Rate limiting enforces an entitlement — this customer bought 60,000 tokens per minute and may not exceed it, whatever the system's overall state. Shedding responds to scarcity — the system as a whole cannot serve everyone right now, regardless of entitlement. A tenant well inside their entitlement can and will be shed during an event like this promotion, and a tenant over their entitlement is limited even on an idle Sunday. Two policies, two triggers, and both need to be legible in the response so that a support engineer reading a 429 or 503 can tell which one fired.

Deadline propagation is the piece that makes all of this work and the piece most often absent. The admission code above reads req.deadline, which has to come from somewhere: a client-supplied budget header, a gateway-assigned default derived from the route's SLO — the service level objective, the response time the route publicly commits to — or a context deadline threaded through every hop. Without it, every layer invents its own timeout, the outermost one usually fires first, and the inner layers keep working on a request whose caller is gone — which reproduces the original failure one level down.

None of this stops the load from arriving. It decides who is served and what everyone else is told, and both a shed 503 and a provider 429 land at a client that now has to decide whether to try again. That decision, made identically and simultaneously by every client, is how a ninety-second slowdown becomes a thirty-minute incident, which is the subject of the next chapter.