Admission control must reserve the completion nobody has seen yet

The summariser team from the previous chapter built the obvious defence after their incident, which was admission control: a check inside their own service, ahead of the provider, that decides whether a request can be afforded before it is allowed out. Before dispatching, each request counts its prompt with the model's tokenizer, takes that many tokens out of a local per-minute budget set to 285,000 — a deliberate 5 percent margin under the 300,000 the account allows — and waits if the budget is empty. In steady traffic it worked exactly as designed. Provider 429s went to zero for eleven days, and then a Monday-morning batch from a customer's overnight ingestion pushed 1,900 requests into ninety seconds and the 429s came back, at 11 percent, with the local budget dashboard showing a comfortable 78 percent utilisation the whole time.

The local budget was measuring three quarters of the traffic. A prompt of 740 tokens produces a completion of about 205, and the provider charges both against the same per-minute window, so a limiter that reserves only the prompt admits roughly 1.28 times as much work as the quota permits. In steady state that overshoot was absorbed: request arrival was smooth enough, and the provider's own accounting forgiving enough at the minute boundary, that a 22 percent undercount stayed invisible. The burst removed the slack, and the difference between what the limiter thought it was spending and what the account was actually charged became a rejection rate.

Reserve the maximum, capture the actual#

The mechanism that fits this problem is the one a card pre-authorisation uses. A petrol station does not know what you will pump, so it holds an amount large enough to cover the worst case at the pump, lets you fill up, and then captures the real figure and releases the rest. The hold is deliberately larger than the expected charge because the alternative — holding the average and hoping — means the customers who fill a large tank are exactly the ones who overdraw the account.

Applied here, admission reserves prompt_tokens + max_tokens — the second of those being the request parameter that caps how long the completion is allowed to run — and settles against usage.total_tokens when the response lands. The prompt half is known before dispatch. The completion half is not knowable at all at that moment; the only thing you can assert about it is the ceiling you yourself set in the request, and that ceiling is therefore the correct reservation. Reserving the typical output instead is a bet that the requests arriving in a given minute resemble the historical average, and that bet loses precisely when it matters, because bursts are usually correlated — a batch of contracts, a retry storm of the same failed job, a promotion that draws one kind of user — so the minute in which you are closest to the ceiling is also the minute in which every request wants a long answer.

The release matters more than it looks. With max_tokens at 512 against a mean completion of 205, every settled request hands back about 60 percent of the completion half of its hold, so the effective capacity of the limiter depends on how quickly holds are settled. A request that takes 4 seconds end to end occupies its full reservation for 4 of the 60 seconds in the window, which costs about 7 percent of that hold's value in the aggregate; a streaming request that takes 30 seconds — one whose response is delivered piece by piece over a connection held open while the model generates — occupies it for half the window and costs far more. Long streams are therefore expensive to the limiter in a way they are not to the provider, and the mitigation is to settle incrementally as chunks arrive rather than only at the end.

class TokenLedger:
    """Per-minute token budget with reserve-then-settle accounting."""

    def __init__(self, tpm_budget: int) -> None:
        self.budget = tpm_budget
        self.spent = 0          # settled usage in the current window
        self.held = 0           # reserved but not yet settled

    def reserve(self, prompt_tokens: int, max_tokens: int) -> int | None:
        hold = prompt_tokens + max_tokens
        if self.spent + self.held + hold > self.budget:
            return None          # caller queues or sheds
        self.held += hold
        return hold

    def settle(self, hold: int, actual_tokens: int) -> None:
        self.held -= hold
        self.spent += actual_tokens

    def release(self, hold: int) -> None:
        """Request failed before the provider charged anything."""
        self.held -= hold

The window boundary is the part of this that resists a clean answer. A request admitted at second 58 of one minute, streaming for twenty seconds, spends most of its completion in the following minute, and providers differ on whether they attribute those tokens to the window in which the request started or the window in which each chunk was produced. The difference is small for a workload of one-second calls and substantial for one where the mean call runs longer than a third of the window: at a mean end-to-end latency of 25 seconds, roughly 40 percent of any minute's completions belong, by wall clock, to the next minute. Rather than reason about which convention a provider uses, measure it — run a controlled burst that fills exactly one minute's quota with long streams and see which minute the 429s appear in — and set the local budget's margin to cover whichever answer you get.

Three details in that fifteen lines carry most of the correctness. The admission test is against spent + held, not spent, because a request that has been dispatched and not yet answered has already committed tokens the provider will charge. settle subtracts the hold and adds the real usage as two separate movements, so an under-reservation — which happens if the provider counts your prompt higher than you did — shows up as a growing gap rather than a negative balance. And release exists as a distinct operation from settle, because a connection that failed with a network error before the provider generated anything should give the full hold back, while a request cancelled mid-stream should settle whatever was generated up to the cancellation.

The size of the worst case is a design decision, not a constant#

Reserving max_tokens is only reasonable if max_tokens is a number you chose deliberately. Most SDKs default it to the model's ceiling, which on current long-output models can be 8,000, 16,000 or more, and a limiter that reserves 16,740 tokens for a request that will produce 205 will admit seventeen requests per minute on a 285,000-token budget while the account happily serves three hundred. The reservation is only as tight as the ceiling, so the ceiling has to be set per route, from the observed output distribution of that route, with enough margin that legitimate long answers are not truncated.

Choosing the per-route cap is a measurement, not a guess. Take a week of completion lengths for that route, take p99.9, and add headroom — for the summariser, p50 was 205 tokens, p99 was 430, p99.9 was 486, so 512 truncates roughly one response in two thousand and reserves 2.5 times the median. That ratio is the price of not knowing, and it is why routes with tight output distributions are much cheaper to admit than open-ended chat, where the honest p99.9 might be 4,000 tokens and the reservation is twenty times the median. For those routes, the useful lever is structural: instruct for brevity, cap the response in the prompt as well as in the parameter, and split long-form generation onto a separate budget so it cannot starve the interactive path.

Two further things break the ledger if they are not handled. Every replica holds its own budget unless the ledger is shared, so eight pods each running a 285,000-token limiter will collectively admit 2.28 million tokens per minute against a 300,000-token account. Sharing it means a round trip on the admission path — a Redis INCRBY on a key that expires at the end of the minute is the usual shape, and at sub-millisecond latency it is cheap against a call that will take seconds. What is not cheap is a shared ledger whose round trip is 40 milliseconds, because that latency lands on every request including the ones that will be rejected. And the local prompt count is an estimate with the systematic bias described in the previous chapter, so the margin between the local budget and the account quota has to be at least as large as the observed drift between your count and the provider's reported usage.prompt_tokens.

What this does not buy you#

An admission ledger is not a replacement for handling 429s. It reduces their frequency, and it cannot eliminate them, because your view of the account is incomplete: another service, a batch job, a colleague's notebook, or a second region sharing the same subscription quota can all spend from the same pool without appearing in your ledger. Any design that treats local admission as authoritative and 429 handling as dead code will discover the assumption during an incident.

Neither is it protection against a request that is individually too large. The ledger asks whether a reservation fits in the remaining budget, and a single request with a 200,000-token prompt fits comfortably in a 285,000-token window on an idle minute, gets admitted, and consumes two thirds of the account's throughput for that minute on behalf of one caller. Per-request size limits are a separate check that belongs at the same admission point, expressed as a hard ceiling on prompt_tokens + max_tokens — commonly some fraction of the per-minute budget, so that no single call can take more than, say, a tenth of a window — and the requests that exceed it need a route of their own with a different promise about latency.

It is also not fairness. A ledger decides how much total work is admitted per minute; it says nothing about whose work that is, so one tenant issuing 200,000-token requests can consume the whole budget while a hundred small tenants queue behind them. Per-tenant partitioning of the budget is a separate mechanism with its own dynamics, and it was the subject of 15.5; what changes here is only that the unit being partitioned is a reservation rather than a request.

And the ledger does not create capacity. It converts a provider-side rejection into a local decision, which is genuinely valuable — a local decision is instant, cheap, observable, and can be made per-route and per-tenant — but the number of tokens that will be served in the next minute is identical either way. Under any sustained load above the quota, admission control changes only the shape of the failure and who gets to choose it.

That choice is the part still missing. reserve returning None is the interesting branch, and the code above simply says "caller queues or sheds" as if those were interchangeable. The next chapter takes a workload that offers ten times the quota for forty minutes and shows why queueing everything satisfies the requirement as written and destroys the product.