Forty tenants at the p95 is three times your ceiling

Six weeks of closed beta produced a distribution rather than an answer. Across twelve tenants, the median ran 8,000 tokens per minute during business hours, the p95 tenant ran 140,000, and one customer's Monday-morning reconciliation job peaked near 900,000 tokens per minute for about ten minutes and then went quiet for a week. General availability was scheduled for forty tenants against a provider account ceiling of 2,000,000 tokens per minute, and the limit had to be a number in a config file by Friday.

The proposal on the table was 150,000 tokens per minute for everyone, set just above the observed p95 so that almost no legitimate request would ever be refused. It reads the distribution correctly, it is defensible from data, and forty tenants at 150,000 is 6,000,000 tokens per minute — three times what the account can serve. Limits whose sum exceeds the ceiling by that margin are not enforcing anything collectively; they delegate the real decision to the provider's 429s, which arrive as a platform-wide failure affecting every tenant simultaneously rather than as one tenant being told to slow down.

The opposite instinct fails just as clearly. Dividing 2,000,000 by forty gives 50,000 each, which sums exactly to the ceiling and refuses the p95 tenant every single day, while leaving the median tenant's 8,000-token allocation 42,000 tokens idle. Both numbers come from correct arithmetic on real data. The gap between them is the decision nobody has made yet.

Read the distribution, then set tiers against it#

A single per-tenant number is the wrong shape for a distribution with a 100:1 spread between its middle and its tail. Tiers cost almost nothing to implement — the config in chapter 5 already carries a tier field — and they let the limit track what a customer actually bought.

Each column refuses a different failure. The tokens-per-minute figure refuses a tenant whose generation volume outruns what they pay for. The burst column, which is the bucket capacity from chapter 2 and is deliberately smaller than the rate for the two larger tiers, refuses the instantaneous claim that a full minute's allocation would otherwise permit. Requests per minute refuses the many-small-calls shape from chapter 3, which no token figure constrains. The in-flight cap refuses the concurrency that degrades latency for everyone else. The monthly budget refuses the sustained overspend that a rate cannot see, and it is the control that turns a rate limit into a cost limit.

Setting these from the mean would produce numbers around 30,000 for everyone and refuse the upper half of the distribution constantly. The percentiles are the right reading, taken per tenant per minute rather than per request, and paired with the concurrency each tenant actually runs, because a tenant at 140,000 tokens per minute across 8 concurrent long generations and one at the same rate across 200 short ones present very different problems to the same ceiling.

The total is 4,000,000 against a 2,000,000-token ceiling, which is an oversubscription ratio of 2.0. Oversubscription is selling more capacity in total than you hold, on the expectation that not every buyer claims theirs at the same moment, and here it is a deliberate choice rather than an oversight.

Oversubscription is a bet on coincidence, so measure the coincidence#

Every shared-capacity system is oversubscribed. Airlines sell more seats than the aircraft holds, banks lend deposits they hold a fraction of, and network operators sell contention ratios that make the arithmetic explicit. What separates a workable oversubscription from a reckless one is that the workable version has measured how often the claims coincide and has decided in advance what happens when they do.

The beta data answers the first half. Across six weeks, the sum of each tenant's individual peak was 1,340,000 tokens per minute, while the highest total the platform ever actually served in a single minute was 780,000 — a coincidence factor of 0.58. Peaks do not line up, because tenants are in different time zones, run their batch work at different hours, and have different traffic shapes. Projecting that factor onto the launch configuration gives an expected simultaneous claim near 2,300,000 against the 4,000,000 of limits sold, which still exceeds the ceiling and says the bet needs help rather than that it is unplaceable.

The projection is also the weakest number in this chapter and should be treated as such. Twelve tenants is a small sample for estimating a coincidence factor, the factor improves with tenant count only when demand is independent, and the demand here is not independent: Monday mornings, month-end closes and business hours are shared clocks that correlate every customer's peak. The way to hold this honestly is to assume the factor degrades as you grow, re-measure it monthly, and treat any month where the observed simultaneous claim exceeds 70% of the ceiling as the trigger for a provider quota increase rather than a surprise.

The second half of the bet is the part that is engineering rather than estimation. Oversubscription is safe when the aggregate has its own limiter, so that the scarce resource is rationed by your policy instead of by the provider's refusals.

GLOBAL_CEILING = 2_000_000          # provider account TPM
GLOBAL_GUARD   = 1_700_000          # 85%: leave the provider room for retries

TIER_FLOOR = {                      # protected share when the pool is scarce
    "enterprise": 0.60, "scheduled": 0.60, "growth": 0.40, "starter": 0.25,
}

async def acquire_with_global(p: Principal, route: Route, messages) -> Lease:
    lease = await admission.acquire(p.tenant_id, route, messages)   # ch.3 stack
    ok, retry = await global_bucket.reserve(GLOBAL_GUARD, lease.reserved)
    if ok:
        return lease

    # The pool is the binding constraint. Ration by tier and by route class
    # before refusing anyone: batch work waits, interactive work degrades.
    await admission.release(lease)
    if route.klass == "batch":
        raise Deferred(queue="batch", retry_after=retry)
    floor = TIER_FLOOR[p.tier] * tenant_limit(p.tenant_id)
    if await tenant_rate(p.tenant_id) < floor:
        # under their protected share — admit against reserve capacity
        await global_bucket.reserve_from_reserve(lease.reserved)
        return lease
    raise RateLimited(kind="global_capacity", retry_after=retry)

Three properties of that code do the work. The guard sits at 85% of the account ceiling so that the provider never sees the platform at its limit and retries have somewhere to go. Batch-class routes are deferred rather than refused, which converts a scarcity event into latency for work that does not need to be interactive. The tier floor guarantees each tenant a protected share of their own limit even when the pool is exhausted, so a starter-tier customer is never starved to zero by an enterprise neighbour — a global limiter without a fairness rule simply moves the noisy-neighbour problem from the provider to your own front door, since first-come-first-served under contention favours whoever generates the most traffic.

The Monday customer, and whether to raise a limit#

Whatever the enterprise tier is set to, the reconciliation job at 900,000 tokens per minute will breach it, and the request to raise the limit will arrive with a customer-success manager attached. Raising it is a reasonable outcome and it should be conditional on four things, each of which corresponds to a way the raise could hurt.

The first is a scheduled window, agreed in writing, naming the day, the hour and the duration. A peak you know about is a capacity plan; the same peak arriving unannounced is the incident from chapter 6. The second is a check that the committed peaks in that window still fit: if two other enterprise tenants have Monday 09:00 windows, the sum of committed peaks for that hour has to stay under the guard, and the answer may be that the customer gets 06:00 instead. The third is a question about the workload itself — a reconciliation job over an archive is almost always batchable, and a batch lane with a lower priority and its own ceiling serves it at a quality the customer cannot distinguish while removing it from the interactive pool entirely. The fourth is the provider quota: if the committed peaks push past 80% of 2,000,000, the raise depends on an account increase that takes days to arrange, so the conversation starts with the provider rather than with the config file.

What should not happen is raising the limit because a breach was noisy. A per-tenant limit raised without a corresponding decision about the aggregate is a limit moved from a place where it refuses one tenant to a place where the provider refuses everyone, which is the arithmetic this chapter opened with.

The ceiling is a number you can negotiate and route around#

Treating 2,000,000 tokens per minute as a fact of nature makes the arithmetic harder than it needs to be. Provider ceilings rise with spend tier and with a request, usually over days rather than months, and the trigger for asking is a figure you already compute — the sum of committed peaks approaching 80% of the current ceiling. Asking early is cheap and asking during an incident is not, because the increase does not land in the fifteen minutes you have.

The second lever is the gateway from 15.4, which can hold credentials for more than one account or more than one provider and route around a pool that is scarce. That changes the shape of the problem in a way worth being precise about: two accounts of 2,000,000 tokens per minute are not one account of 4,000,000, because a tenant pinned to a model that only one provider serves cannot borrow the other pool, and a request routed to a second provider mid-conversation loses whatever prompt cache it had accumulated. Overflow routing works well for stateless, model-flexible work — classification, extraction, embedding — and poorly for long interactive sessions. The practical arrangement is to treat the primary pool's guard as the number the tier table sums against, and the secondary pool as the destination for the batch class that the global guard defers, which turns a queue into a slower lane rather than a wait.

Putting the numbers in front of traffic#

None of these figures deserve to be trusted before they have been run. The sweep from chapter 1 is the acceptance test: replay production-shaped traffic at the tier limits, with the shadow limiters recording verdicts and enforcing nothing, and read the would-reject counts per tenant per route. Two weeks of shadow mode against real traffic usually finds one or two routes whose reservation estimate is badly calibrated and one tenant whose tier is wrong, both of which are cheaper to discover in a log than in a support queue. Then enforce for new tenants first, where there is no expectation to violate, and migrate existing tenants with a fortnight's notice and their own shadow numbers attached to the email.

The limiter that ships from this is not finished, because the distribution it was set from will move. Tenants change tiers, a new feature lengthens completions on a route by 40% overnight, and a provider raises your account quota. The quarterly loop that keeps it honest is short: re-read the per-tenant tokens-per-minute percentiles, recompute the coincidence factor from the last quarter's observed simultaneous peaks, check the reservation efficiency histograms for routes that have drifted, and confirm that the sum of committed peaks still sits under the guard. Publish the limits in your API documentation with the response headers that expose remaining budget, so that customers can back off before they are refused, and keep the four corners of the load-test grid in CI so that the next person to change a default learns about it from a failing test.