One tenant, three keys, three buckets, no limit

The finance review flagged Northwind before engineering noticed anything. Their contract entitled them to 120,000 tokens per minute and a monthly allocation of 450,000,000 tokens; the usage export showed them peaking at 341,000 tokens per minute on weekday mornings and on track to finish the month at 780 million. Engineering's first response was that the limiter must be broken, because the tenant dashboard showed no refusals and no breaches. Both were right. The limiter kept one bucket per API key, Northwind held three, and every one of those buckets spent the morning peak comfortably under its own 120,000-token ceiling.

$ rlctl inspect --account northwind --window 09:05-09:06
key                              label            tokens/min   limit    state
sk-live-8f21…                    production          118,400  120,000   ok
sk-live-2c07…                    staging             104,900  120,000   ok
sk-live-b9d4…                    data-science        117,700  120,000   ok
                                 -----------------------------------------
                                 account total       341,000  120,000   (unenforced)

The bottom line of that output is not a limit that was breached. It is a number the system never computed, because nothing in the enforcement path had a concept that the three keys were the same customer. Each key had been issued through a self-service console, each got a bucket at the default limit when it was created, and the limit that appears on the contract was applied three times rather than once.

The bucket belongs to whoever the contract names#

Rate limiting has always been an identity question first and an arithmetic question second, and the identity you choose determines what evasion looks like. Cloud APIs learned this in the era when quota was attached to a key and customers discovered that more keys meant more quota; the fix, adopted almost universally, was to make the key an authentication artifact and the account or project the unit of accounting. Inference workloads make the same choice more urgent, because per-key defaults are usually generous — you set them so that a new integration works out of the box — and because the resource being over-claimed is a shared provider ceiling rather than your own elastic compute.

Binding to the tenant means resolving credentials to a tenant identifier at authentication and carrying that identifier, never a client-supplied one, into the limiter. The distinction is worth being blunt about: a X-Tenant-Id header that the caller sets is not an identity, it is a suggestion, and a limiter keyed on it can be zeroed by anybody who reads your documentation. The tenant must come from the verified credential — the key record you looked up, or a signed claim inside the JWT, the JSON Web Token whose signature your service checks and whose contents therefore cannot be edited by the caller. The resolved result is the request's principal: the identity everything downstream, including the limiter, is entitled to trust.

@dataclass(frozen=True)
class Principal:
    tenant_id: str        # the accounting identity — from the credential
    key_id: str           # reporting dimension only
    key_label: str        # "production" | "staging" | "data-science"
    tier: str

async def authenticate(request) -> Principal:
    record = await keys.lookup(hash_key(request.headers["authorization"]))
    if record is None or record.revoked_at is not None:
        raise Unauthorized()
    return Principal(
        tenant_id=record.tenant_id,
        key_id=record.id,
        key_label=record.label,
        tier=record.tier,
    )

async def admit(p: Principal, route: Route, messages) -> Lease:
    # Enforcement is keyed on the tenant. The key rides along as a label so
    # dashboards can still attribute burn, and so support can answer
    # "which of our keys is doing this?" without changing the ceiling.
    lease = await admission.acquire(p.tenant_id, route, messages)
    metrics.tokens_reserved.labels(
        tenant=p.tenant_id, key=p.key_label, route=route.name
    ).inc(lease.reserved)
    return lease

Losing per-key visibility is the objection this change usually meets, and it is a false trade. The key stays in the metric labels, the usage rows and the logs, so the question "which of our keys is burning the budget" is still answerable, and it is answerable more usefully than before because the answer now comes with a share of a single enforced total. What the key stops being is a ceiling of its own.

Sub-limits per key are a reasonable thing to want on top of that, and the rule that keeps them safe is that a sub-limit may only subdivide the tenant's allocation, never add to it. Northwind can cap their staging key at 20,000 tokens per minute so that a runaway test cannot starve production, and that check runs before the tenant check and refuses first; the tenant bucket remains the authority, so the sum of the sub-limits being misconfigured to 400,000 changes nothing about what the account can actually claim. Buckets that compose by addition are the bug this chapter opened with, dressed as a feature.

A rate limit does not bound a total#

Northwind's rate limit, enforced correctly at the tenant, permits 120,000 tokens every minute of every day. Held at that rate continuously, their 450,000,000-token monthly allocation is gone in 3,750 minutes, which is 62 hours — two and a half days into a thirty-day month, without a single refusal and without the limiter ever having been wrong. Even a sedate 30% of the rate ceiling, which no dashboard would look at twice, finishes the month's allocation on the ninth day.

That gap is the denial-of-wallet exposure from 7.9 arriving through a control that is working as designed, and closing it requires a second instrument with a different shape. A rate is a derivative; a budget is an integral. They are enforced at different cadences, they fail in different directions, and one does not approximate the other.

Budgets are cheap to enforce precisely because they are cumulative and monotone: increment a counter with the settled usage, cache the total per node with a short refresh interval, and read the cached value at admission. Being a few seconds stale on a 450-million-token budget costs nothing, whereas being a few seconds stale on a rate bucket is the race from chapter 2.

Most gateway stacks express both in configuration rather than code, and the LiteLLM proxy's Budgets and Rate Limits documentation is a usable template for the shape even if you run something else.

tenants:
  - id: northwind
    tier: growth
    tokens_per_minute: 120_000        # rate: refill, from the contract
    burst_tokens: 40_000              # capacity: the instantaneous claim
    requests_per_minute: 1_200        # the ceiling chapter 3 kept
    max_inflight: 60
    budget:
      period: monthly
      tokens: 450_000_000
      soft_thresholds: [0.8, 0.95]    # notify; no enforcement change
      on_exhausted: degrade           # refuse | degrade | queue
      degrade_to: claude-haiku-4-5
    keys:
      - label: production             # sub-limits subdivide, never add
        tokens_per_minute: 90_000
      - label: staging
        tokens_per_minute: 20_000
      - label: data-science
        tokens_per_minute: 30_000

The on_exhausted field is the one that needs a human decision rather than a default. Refusing is honest and turns a budget overrun into a visible outage for that customer, which is right for a free tier and wrong for an enterprise account mid-quarter. Degrading to a cheaper model keeps the product working at a lower quality and a fifth of the cost, and it needs the product side to have agreed what degraded output looks like. Queueing to an asynchronous batch lane suits workloads that are not interactive and is unavailable to a chat surface. Whichever you choose, the two soft thresholds exist so that the first anyone hears of it is not the enforcement.

Reading a budget without paying for it on every request

The budget check sits in the same hot path as the bucket, and it would be a shame to add a round trip to a control that tolerates being seconds out of date. Each node keeps a cached view of the tenant's period total, refreshed asynchronously, and settles the authoritative counter with the same figures the refund uses.

class PeriodBudget:
    """Cumulative spend per tenant per period. Eventually consistent by design."""

    def __init__(self, refresh_seconds: int = 10):
        self.cache: dict[str, tuple[int, float]] = {}     # tenant -> (used, at)
        self.refresh = refresh_seconds

    async def state(self, tenant: str, limit: int) -> BudgetState:
        used, at = self.cache.get(tenant, (0, 0.0))
        if time.monotonic() - at > self.refresh:
            used = await store.get_period_total(tenant, current_period())
            self.cache[tenant] = (used, time.monotonic())
        frac = used / limit
        if frac >= 1.0:
            return BudgetState.EXHAUSTED
        return BudgetState.WARN if frac >= 0.8 else BudgetState.OK

    async def settle(self, tenant: str, tokens: int) -> None:
        await store.incr_period_total(tenant, current_period(), tokens)

The worst-case overshoot from that staleness is one refresh interval of full-rate burn — ten seconds at 120,000 tokens per minute is 20,000 tokens, which against a 450-million-token budget is a rounding error of 0.004%. Compare that with the rate bucket, where ten seconds of staleness is the entire failure mode of chapter 2, and the difference in required freshness between the two instruments becomes concrete rather than a matter of taste. It also means the budget counter can live in your primary database rather than in the cache tier, so a Redis outage does not lose a month of accounting.

The migration is a customer-facing change

Switching Northwind from three buckets to one cuts their effective ceiling from 360,000 tokens per minute to 120,000, which is what the contract says and is also a two-thirds reduction in what they have been running on for months. Deploying that as a silent bug fix produces an incident report from the customer's side and a difficult call. The sequence that avoids it is the one the rest of this topic keeps arriving at: compute the tenant-level total in shadow for a fortnight, identify every account whose aggregate exceeds their contracted rate, and go to those accounts individually with their own numbers, a date, and either a tier upgrade or a lead time to adjust their workload. Multi-key accounts are usually a small minority of tenants and a large fraction of the excess, so the list of conversations is short.

There are also principals that resolve to no obvious tenant, and they need an answer before the code ships rather than a None at 3 a.m. Internal service traffic — evaluation harnesses, batch backfills, the support tool that replays a customer's request — should be its own synthetic tenant with its own limit, so that an evaluation run cannot consume a paying customer's allocation or the platform's headroom anonymously. Unauthenticated or free-trial traffic has no durable identity by definition, and the honest position is that it cannot be rate limited reliably at the tenant level; it gets a small shared bucket, a low per-IP ceiling as a speed bump, and a total budget for the whole category that you are willing to lose.

What binding to the tenant does not buy you#

Tenant-keyed accounting fixes an aggregation error and nothing beyond it. A stolen or leaked credential still burns the legitimate tenant's own budget, at their own rate, and the limiter will enforce that faithfully while the money disappears — which is a detection and credential-hygiene problem rather than a limiter one. Key rotation still works, and should: the new key resolves to the same tenant and inherits the same bucket mid-flight, which is the property that makes rotation safe rather than an evasion.

Within a tenant, one runaway internal user can still consume the whole tenant allocation and starve their colleagues, and the tenant bucket has no opinion about that. If your product exposes end users directly, a per-user layer nested under the tenant ceiling is worth adding, again as a subdivision. Across tenants, nothing here prevents one customer sitting at 100% of a legitimately large allocation while everyone else queues behind the shared account ceiling, because from the limiter's point of view they are behaving perfectly.

That last case is the one that produces the worst incidents, since every per-tenant control is satisfied and every per-tenant dashboard is green while platform latency doubles. Chapter 6 takes an unannounced backfill that did exactly that and works out which signals would have caught it.