Rate limiting and quotas for LLM traffic

Read these first

• AI gateways and proxies
• Denial-of-wallet

What you will know

• Implement a token-bucket limiter charged in tokens rather than requests, including output-token reservation and refund, and verify burst behavior

  Concurrent generations can exceed the budget before final usage is known unless capacity is reserved up front and unused allocation is returned.

• Set per-tenant quotas that survive an abusive user

  Bind accounting to authenticated tenant identity and enforce both rate and cumulative budgets so key rotation or distributed callers cannot evade controls.

• Detect a single tenant consuming the shared budget

  Alerting needs tenant share, token burn rate, and remaining global headroom because aggregate traffic can look normal while one customer creates noisy-neighbor pressure.

• Choose the limits from measured usage

  Use token-rate distributions, burstiness, concurrency, and spend ceilings rather than average request volume when setting thresholds.

• Run a load test that compares request-based and token-based limits under mixed prompt and completion sizes

  Equal request rates can produce radically different token spend, queue pressure, and time-to-exhaustion when generation lengths vary.

Chapters

1. 1

   Five hundred requests a minute, fifteen times the tokens

   Run the same fixed 500 requests per minute twice, once asking for 200-token summaries and once for 3,000-token drafts, and a 600-per-minute request limiter passes both runs without noticing the fifteenfold gap in spend. Almost all of that difference is output, the part a request counter structurally cannot see. This is the load test that settles whether your limiter measures capacity or merely traffic.

   Read chapter
2. 2

   Debiting on response admits everything already in flight

   Charging the bucket when the provider reports usage gives a perfectly accurate ledger and no control: 200 concurrent requests each check a balance that predates all of them, and a 100,000-token minute becomes 400,000 spent. Admission has to reserve an estimate up front and refund the remainder afterwards. Bucket capacity is a separate policy decision — it is the burst a rested tenant may take in one go, and it has to be one you can absorb.

   Read chapter
3. 3

   A token bucket alone still draws 429s from a request ceiling

   Moving the unit of account to tokens does not retire the request limiter, and after two chapters of request counts being blind to output length the reader is primed to conclude that it does. Providers publish a requests-per-minute ceiling beside the tokens-per-minute one, so a tenant comfortably inside budget can still be refused. Connection pools, worker slots and request-scoped downstream calls to the retriever and tool APIs are all sized in requests rather than tokens, and the admission check with its reservation bookkeeping costs something per request whatever the token count is. Both limiters stay, and what you owe is the composition rule: which one refuses what, and in which order they are consulted.

   Read chapter
4. 4

   A safe reservation is not automatically a useful one

   Holding the provider's context limit for every call is a guaranteed upper bound and a poor admission rule: effective concurrency collapses to the tenant's limit divided by that constant, so short-prompt customers get throttled as though every generation were maximal. A route with max_tokens set reserves that; an unbounded route reserves a high percentile of its own history; a stream clients abandon needs the refund to fire on disconnect. Decide where reservations live and what their loss does.

   Read chapter
5. 5

   One tenant, three keys, three buckets, no limit

   A customer holding production, staging and data-science keys with a bucket per key stays inside every individual limit while sitting well over quota in aggregate. Bind the bucket to the authenticated tenant those keys resolve to and keep the key itself as a reporting dimension. Pair the token rate with a cumulative budget for the period, since a tenant who never breaches a per-minute rate can still burn a month's allocation in a day.

   Read chapter
6. 6

   Share without its rate of change is a ranking, not a detector

   An unannounced backfill doubles platform p95 for four minutes and starts drawing 429s while aggregate throughput stays inside its weekly range and every per-tenant request rate looks ordinary — both dashboards honest, both blind. Detection needs each tenant's share of token burn, how fast that share is moving, and the headroom left in the global pool. A fixed 30% threshold fires constantly on your largest customer and never on one going from 2% to 20%.

   Read chapter
7. 7

   Forty tenants at the p95 is three times your ceiling

   Beta shows a median tenant at 8,000 tokens per minute, a p95 tenant at 140,000, and one customer peaking near 900,000 every Monday, against a 2,000,000 account ceiling and forty tenants at launch. Setting everyone just above the observed p95 reads the distribution correctly and then sums to three times the ceiling, handing the real limiting job to provider 429s that land platform-wide. Oversubscription is fine once you decide what happens when the sum is claimed at once.

   Read chapter