Five hundred requests a minute, fifteen times the tokens

The load generator ran the same scenario twice on a Tuesday afternoon, against the same staging deployment, holding a fixed 500 requests per minute for eight minutes each time. The only difference between the two runs was one line in the fixture: the first asked the model for a two-sentence summary of a support ticket, the second asked for a first-draft reply to the same ticket, and both were driven from the same 180-token prompt template with the same corpus of tickets. The gateway's rate limiter — a token bucket admitting 600 requests per minute per tenant — logged zero rejections in either run. Its dashboard put the tenant at 83% of limit, twice, and it would have reported 83% if the second run had asked for a thirty-page report.

That scene assumes some vocabulary, and this is the wrong topic to leave it assumed. Rate limiting is the rule a service applies at its front door to cap how much work any one caller may start in a given stretch of time; callers over the cap are refused rather than served, conventionally with an HTTP 429 Too Many Requests response carrying a hint about when to try again. A quota, or budget, is the same instinct over a longer horizon — a total a caller may consume across a day, a month or a billing period, after which they are cut off, downgraded, or billed differently. Both exist because the capacity behind the door is finite and shared, so without them a single caller with a retry loop, a batch job someone started at lunchtime, or a customer whose traffic quadrupled overnight can take all of it and leave everybody else queueing.

The token bucket in the story above is the usual mechanism for the first of those. A bucket holds a number of permits and is topped up at a steady rate; an arriving request takes permits out, and a request that finds too few is refused until the top-up catches up. What those permits stand for is the question this whole topic turns on, and the limiter above spent one permit per request regardless of the request's size. A tenant is whichever customer or account a bucket is attached to, which for a B2B product is normally the company that signed the contract rather than an individual end user.

You meet this from both sides on the same afternoon. The model provider meters your account and refuses you when you exceed what it sold you, and your own product has to meter its customers before they exhaust that account between them, which is work that lands in the gateway from 15.4 — the component every call already passes through.

The provider's usage export for those two eight-minute windows did not agree that the runs were alike. Input was identical to the token, because the prompts were identical. Output was not, and output is where both the money and the serving capacity live.

Every row that the limiter can observe is flat, and every row that determines whether the platform survives the hour moves by an order of magnitude. That is the whole problem in one table, and it is not an artifact of a contrived fixture — the two fixtures are two ordinary features of the same product, summarise and draft, sitting behind the same endpoint and the same tenant credential.

What a request counter is a proxy for, and when the proxy holds#

Counting requests is a good control in the systems most backend engineers learned it in, and it is worth being precise about why. On a CRUD service, the work behind a request is bounded by the shape of the handler: a primary-key read touches one row, a paginated list touches at most the page size, and a write touches one transaction. The dearest request on such an endpoint might cost five or ten times the cheapest, which means a request count is within a small constant factor of the CPU, the database time and the bandwidth it stands in for. Under that condition a request limiter is a capacity limiter wearing a convenient unit, and the convenience is real, because requests are countable at the front door with no knowledge of what the handler will do.

Inference breaks the bound in a specific place. The cost of a generation is dominated by the number of tokens the model emits, one forward pass each, sequentially, each one holding its KV entry on the accelerator for the duration; the prompt by contrast is processed in a single batched prefill and is cheap per token, which is why providers price output at roughly five times input. The number of output tokens is not a property of the endpoint. It is chosen by the caller through max_tokens, influenced by the prompt, and ultimately decided by the model itself when it emits a stop token. Two calls to the same route, from the same tenant, in the same second, can differ by 100:1 in the resource they consume, and nothing at admission time distinguishes them unless you look.

The providers themselves do not pretend otherwise. Anthropic's published rate limits, documented under Rate limits in the Claude platform docs, are enforced as three separate ceilings — requests per minute, input tokens per minute, output tokens per minute — and a 429 tells you which one you hit in the response headers.

HTTP/1.1 429 Too Many Requests
retry-after: 24
anthropic-ratelimit-requests-limit: 4000
anthropic-ratelimit-requests-remaining: 3612
anthropic-ratelimit-input-tokens-limit: 2000000
anthropic-ratelimit-input-tokens-remaining: 1631000
anthropic-ratelimit-output-tokens-limit: 400000
anthropic-ratelimit-output-tokens-remaining: 0
anthropic-ratelimit-output-tokens-reset: 2026-03-17T14:31:00Z

Read the headers of a real refusal and the diagnosis is already made: 3,612 requests of headroom remaining and zero output tokens. Any limiter of yours that speaks only the first line will report a healthy tenant right up to the moment the account stops serving anybody, and the gateway you put in front of the provider in 15.4 is exactly where that mismatch becomes visible, because it is the one component that sees both your admission decision and the provider's answer.

Designing the run that tells you which one you have#

The comparison is worth building as a repeatable load test rather than reasoning about, because the interesting output is not a pass or a fail but a pair of curves that diverge. Hold the request rate fixed, sweep the completion length across the range your product actually produces, and instrument three things per run: the tokens per minute actually billed, the verdict your current limiter would give, and the verdict a token bucket sized to your provider's ceiling would give. Running the token limiter in shadow — evaluating it and recording the decision without enforcing it — is what lets you run this against production traffic later without risking a rejection storm.

import asyncio, time, statistics
from dataclasses import dataclass, field

@dataclass
class ShadowTokenBucket:
    capacity: int                 # tokens
    refill_per_sec: float         # tokens/sec
    tokens: float = field(init=False)
    last: float = field(default_factory=time.monotonic)
    would_reject: int = 0

    def __post_init__(self):
        self.tokens = float(self.capacity)

    def observe(self, billed: int) -> bool:
        now = time.monotonic()
        self.tokens = min(self.capacity,
                          self.tokens + (now - self.last) * self.refill_per_sec)
        self.last = now
        if billed > self.tokens:
            self.would_reject += 1
            self.tokens = 0.0
            return False
        self.tokens -= billed
        return True

async def run(client, rps: int, seconds: int, max_tokens: int, bucket):
    billed, latencies, provider_429 = [], [], 0
    async def one():
        nonlocal provider_429
        t0 = time.monotonic()
        r = await client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=max_tokens,
            messages=[{"role": "user", "content": TICKET_PROMPT}],
        )
        latencies.append(time.monotonic() - t0)
        n = r.usage.input_tokens + r.usage.output_tokens
        billed.append(n)
        bucket.observe(n)

    async with asyncio.TaskGroup() as tg:
        for tick in range(seconds):
            for _ in range(rps):
                tg.create_task(one())
            await asyncio.sleep(1)

    return {
        "tpm": sum(billed) / seconds * 60,
        "p95_latency_s": round(statistics.quantiles(latencies, n=20)[18], 2),
        "shadow_rejects": bucket.would_reject,
        "provider_429": provider_429,
    }

The number the sweep produces that nothing else produces is time-to-exhaustion, and it is the one to put in the report. Take a token bucket holding 300,000 tokens and refilling at 300,000 per minute, which is a sane per-tenant allocation for run A's traffic with headroom to spare. Run A burns 190,000 tokens a minute against a 300,000-a-minute refill, so the bucket never leaves full and the tenant could double their traffic without noticing. Run B burns 1,590,000 a minute, a net drain of 1,290,000 a minute against the same refill, which empties a full bucket in 300,000 ÷ 1,290,000 of a minute — fourteen seconds. The same tenant, the same request rate, the same code path, and the difference between a limit that is never approached and one that is gone before the first health check reports.

The bottom row is the one that stops people arguing that the request limit could simply be tightened. Sixty requests a minute of 3,000-token drafts and 500 requests a minute of 200-token summaries are the same load on the provider and on your account ceiling, and no setting of a request limit treats them as the same, because a request limit tight enough to contain the first refuses seven-eighths of the second. There is no threshold that makes the wrong unit into the right one; the two workloads are only equal when measured in the unit that is actually scarce.

Fixed completion lengths understate what production does

Runs A and B each held the completion length constant, which is what makes the table legible and what makes it optimistic. A real route produces a distribution, and the distributions inference workloads produce are usually skewed and often bimodal: a support-reply route in this product had a median completion of 240 tokens, a p90 of 1,100 and a p99 of 3,800 — a p90 of 1,100 means one completion in ten runs longer than 1,100 tokens, and a p99 of 3,800 that one in a hundred runs longer than 3,800 — because most tickets get a short acknowledgement and a minority get a full explanation with three cited passages. A limiter tuned against the median is wrong 10% of the time by a factor of four, and the times it is wrong are correlated, since a wave of complicated tickets arrives together during an incident at the customer's end.

Adding a third run drawn from the route's real completion histogram is therefore worth the extra fixture. It burns 470,000 tokens per minute against run A's 190,000 at the same request rate, and, more usefully, it produces queue depths that the fixed-length runs do not, because variance rather than the mean is what makes a queue grow. Two workloads with identical mean completion length and different variance have different p95 latencies at the same utilisation, which is the standard queueing result and is the reason a mixture run belongs in the suite alongside the corners.

The input side varies less dramatically and not negligibly. A retrieval-augmented route's prompt length is a function of how many passages the retriever returned and how long they were, which in this system ranged from 400 to 26,000 tokens across the same endpoint. Prompt caching complicates the arithmetic further, because a cache hit bills the cached prefix at a fraction of the base input rate, so two requests with identical prompt lengths can differ several-fold in billed input depending only on whether the previous call happened within the cache's lifetime. None of that is visible to a request counter either, and it means a token limiter reading the provider's billed figures and one counting raw prompt tokens locally will disagree — a discrepancy worth deciding about deliberately rather than discovering during a reconciliation.

What the comparison does not establish#

Switching the unit of account to tokens is a statement about what you meter, and it settles less than it appears to. It does not give you cost control, because a rate expressed per minute says nothing about the total a tenant may spend across a month — a customer sitting at 40% of a generous per-minute rate around the clock can exhaust a quarterly budget in a fortnight without ever being refused, which is the denial-of-wallet shape from 7.9 arriving through a limiter that is functioning exactly as configured. It does not give you fairness between tenants, since a per-tenant limit that every tenant is entitled to claim at once is a promise your account ceiling may not be able to keep. It does not address queueing or latency, which are governed by how many generations are in flight rather than how many tokens they will eventually cost, so a concurrency cap remains a separate control with a separate job.

It also does not retire the request limiter, however strongly the table above argues for tokens. The provider publishes an RPM ceiling alongside the token ceilings for reasons of its own, and your gateway has its own per-request costs — connection slots, worker threads, the retrieval and tool calls each generation fans out into — none of which shrink because a completion turned out to be short.

The immediate obstacle is arithmetic rather than philosophy. Charging a bucket in tokens requires knowing how many tokens to charge, and at the moment the admission decision has to be made, the input length is countable and the output length does not exist yet. The tempting repair is to wait for it: let the request through, and debit the bucket when the provider reports usage in the response, which is the only figure that is unarguably correct. Chapter 2 runs that design against two hundred concurrent requests and shows what a perfectly accurate ledger costs.