A safe reservation is not automatically a useful one

The escalation arrived with a graph attached. A customer on a 120,000-tokens-per-minute contract was seeing rate_limit_error on roughly one call in three, and their own metering — taken from the responses they did receive — showed them averaging 58,000 tokens per minute. They were being refused at 48% of what they had been sold, and the limiter, the bucket arithmetic and the refund path were all behaving exactly as the previous chapters described.

The reservation was the whole of it. Their summarisation route left max_tokens unset, so the gateway reserved a fixed 4,000 output tokens per call on the reasoning that the route had been seen to produce that much. Actual completions on that route averaged 1,850 tokens. Every request therefore held about twice the capacity it would go on to use, the surplus came back thirty seconds later when the response landed, and in the meantime admission was making its decisions against a bucket that believed the tenant was consuming twice their real rate. The ledger was correct within a minute of every request. The admission decision, which is the only thing the reservation exists to inform, was wrong the entire time.

The bound that is safest is the one that admits nobody#

The instinct that produces this failure runs in a defensible direction: if the reservation must not be exceeded, reserve something that cannot be exceeded. The model's context limit — the largest number of tokens it will accept and emit for one call, prompt and completion together — is such a number. Hold 200,000 tokens for every call and no generation can ever overshoot, and the refund arrives promptly to correct the ledger, so the accounting is exact within one request's latency.

What that costs is visible in one division. A reservation of R against a bucket capacity of C admits at most C ÷ R requests before refusing, whatever those requests turn out to consume, because capacity is held from the moment of admission until settlement.

The first row is not rhetorical. A per-request reservation larger than the tenant's bucket capacity is refused unconditionally, which is the RequestTooLarge branch from chapter 3 firing on every call, and teams do ship this — usually by defaulting the reservation to the model's context window for routes where nobody set anything. Safety in the sense of never overshooting and usefulness in the sense of admitting the traffic you sold are different properties, and the uniform worst case maximises the first by discarding the second.

The asymmetry that makes the tuning tractable is that the two errors have very different shapes. Over-reserving costs throughput, continuously, in proportion to the ratio, and it is invisible in the ledger because the refunds keep the totals right. Under-reserving costs an overshoot bounded by the number of concurrent requests times the per-request shortfall, and the bucket absorbs it by going negative and refusing until it recovers, which is a self-correcting error rather than an unbounded one. If 50 requests each under-reserve by 600 tokens, the bucket ends the minute 30,000 tokens down and throttles for eighteen seconds at a 100,000-per-minute refill. That is a worse minute; it is not an incident. Reserving at a percentile in the p75 to p90 range of the route's own distribution is generally the better trade, and reserving at p99 is a quieter way of reserving the worst case.

Three routes, three answers#

The route with max_tokens set to 500 has nothing to estimate. The provider will not emit a 501st token, so the reservation is the exactly counted input plus 500, and the only tuning left is whether 500 is the right cap for the feature. This is the case worth engineering toward: a gateway that injects a route-level default max_tokens on every request that arrives without one converts the hard estimation problem into the easy one, and it does so with a change to a config file.

The unbounded route that produces anywhere from 200 to 4,000 tokens needs a distribution, kept per route and per model, updated from settled requests, and read at a percentile you choose deliberately. A rolling window of the last few thousand completions is enough, and a t-digest — a sketch that answers percentile queries from a few kilobytes of state instead of from the raw samples — or a coarse histogram avoids storing them.

class RouteReserve:
    """Per-(route, model) output estimate from a rolling window of settlements."""

    def __init__(self, floor: int = 256, ceiling: int = 8192, q: float = 0.90):
        self.digest = TDigest()      # updated on every settled request
        self.floor, self.ceiling, self.q = floor, ceiling, q
        self.n = 0

    def observe(self, output_tokens: int) -> None:
        self.digest.update(output_tokens)
        self.n += 1

    def reserve_output(self, max_tokens: int | None) -> int:
        if max_tokens is not None:
            return max_tokens                     # nothing to estimate
        if self.n < 200:                          # cold route: be conservative
            return self.ceiling
        est = int(self.digest.quantile(self.q))
        return max(self.floor, min(self.ceiling, est))

The cold-start branch is where this design is least satisfying and worth stating plainly: a route with no history reserves its ceiling, so the first few hundred requests on a newly deployed route run at the concurrency the previous section called wasteful. Seeding the digest from a sibling route, or from a canary run, shortens the window; nothing removes it. Keying the digest per tenant as well as per route sharpens the estimate for customers whose usage is unlike the average, at the cost of more cold starts and more state, and is worth doing only for the handful of tenants large enough for their own distribution to be well populated.

The streaming route is different in kind rather than in degree, because the settlement event may never arrive. When a client disconnects mid-stream, there is no final message with a usage block to read, and a finally block that refunds reserved - used with used still at zero returns the whole reservation for a generation that really did consume 900 tokens before the socket closed. Streaming responses carry incremental usage, so the fix is to keep a running count as deltas arrive and settle against whatever that count reached.

async def guarded_stream(tenant: str, lease: Lease, req) -> AsyncIterator[Event]:
    streamed_out = 0
    try:
        async with client.messages.stream(**req) as stream:
            async for event in stream:
                if event.type == "message_delta":
                    streamed_out = event.usage.output_tokens
                elif event.type == "message_start":
                    streamed_out = 0
                yield event
    except (asyncio.CancelledError, ClientDisconnect):
        # client hung up: the provider stops billing at roughly this point,
        # but the tokens already generated were really spent.
        raise
    finally:
        settled = lease.input_tokens + streamed_out
        await bucket.refund(tenant, max(0, lease.reserved - settled))
        metrics.reservation_efficiency.observe(settled / lease.reserved)

Whether a cancelled stream actually stops the meter depends on the provider and on your transport: cancelling the HTTP request usually stops generation within a token or two, and a gateway that proxies through a queue may keep generating after your client is gone. Measuring your own case with a deliberate mid-stream abort, then comparing the provider's usage export for that request against what you counted, is a twenty-minute experiment that decides whether your refund is honest.

Where reservations live, and what their loss costs#

A reservation is state with a lifetime, so it needs an owner and an expiry. Keeping it as a lease record — a small row naming the tenant, the amount held, and the time after which the hold is presumed abandoned — beside the bucket, in the same Redis keyspace and the same hash slot, means the reserve and the lease write are one script, and a background sweeper can refund anything whose lease has outlived the maximum plausible generation time.

That expiry is the lease's TTL, its time to live, after which the store deletes the record and the sweeper treats the held capacity as free. The TTL wants to be comfortably longer than your slowest legitimate generation and much shorter than a minute of budget — 180 seconds against a 60-second refill window is a reasonable starting point, because it lets an unusually slow 8,000-token generation settle honestly while capping how long a leaked reservation can hold capacity hostage. Without the sweeper, a process killed between reserving and refunding leaks its reservation permanently, and a rolling deploy across twelve nodes during a tenant's peak can quietly retire a third of their bucket until someone flushes the key.

Redis itself failing is the decision people postpone and then make at 3 a.m. Failing closed — refusing every request whose limit cannot be checked — turns a cache outage into a total inference outage, and failing open, admitting them instead, removes every limit at the moment the platform is least healthy. The workable middle is a per-node local bucket, sized at the tenant's limit divided by the node count with a safety factor, that takes over when the shared store is unreachable: it over-admits by the number of nodes in the worst case, which is bounded and survivable, and it keeps the account ceiling from being approached by an uncontrolled fleet. Reservations in flight at the moment of a failover are lost either way, which is what the lease TTL is really insuring against.

The input half is countable, which is not the same as counted

Calling the input side exact requires that you count what the provider will actually bill, and the two drift apart in specific places. A local tokenizer run over the user's message misses the system prompt your gateway injects, the tool schemas serialised into the request, the few hundred tokens of retrieved context added downstream of where you counted, and images, which are billed at roughly (width × height) / 750 tokens and can add 1,100 tokens for a single 1,092-pixel-square screenshot. In one measured comparison across a week of production traffic on a RAG route, locally counted input averaged 3.4% below the provider's reported input_tokens, with a p99 gap of 11% concentrated in the requests that carried attachments.

Two fixes are available and they suit different routes. Counting after templating — serialising the exact payload you are about to send and tokenizing that — removes most of the gap and costs a millisecond or two of CPU per request. Where the payload is assembled downstream or includes modalities your local tokenizer cannot price, the provider's token-counting endpoint is authoritative and costs a network round trip you may not want in the admission path. Either way, tracking the drift as a metric and folding a small multiplier into the reservation is what stops a systematic undercount from quietly becoming a systematic overshoot.

Diagnosing the customer who is throttled at half throughput#

The metric that makes this class of complaint a five-minute investigation is reservation efficiency: settled tokens divided by reserved tokens, recorded as a histogram per route and per tenant. A route whose efficiency sits at 0.47 is admitting half the traffic it could, and no amount of staring at the bucket or the refill rate will show that, because both are behaving correctly. Pair it with the pattern that identifies the symptom from the outside — a tenant with a nonzero rate_limit_error count whose observed tokens per minute is well under their configured limit — and the two together separate over-reservation from a genuine limit that is simply too low.

There is a plausible alternative diagnosis to rule out. If the customer's traffic arrives in tight bursts, refusals can come from capacity rather than from the reservation size, in which case efficiency will look healthy near 0.85 and the bucket will show as empty at the moments of refusal. Comparing the p50 efficiency against the refusal timestamps distinguishes them: uniform refusals with low efficiency is an estimate problem, clustered refusals with good efficiency is a burst-capacity problem, and the second is fixed with the capacity dial from chapter 2 rather than with the estimator.

Reservation efficiency is not a service level and should not be driven to 1.0, since an estimator that never over-reserves is one that under-reserves half the time by construction. Somewhere between 0.75 and 0.9 means the estimate is close enough that admission reflects reality, with enough slack that overshoot stays in the self-correcting range.

Everything in the last three chapters has assumed the phrase "the tenant's bucket" refers to something unambiguous. The next chapter starts with a customer who is over quota while every bucket the system holds for them is inside its limit.