The dimensions of a quota

The account — the billing and ownership container a cloud provider groups a company's resources under, called a subscription on Azure and a project on Google Cloud — is where a cloud model platform meters you. Metering happens on several axes at once, and admission works the way traffic shaping works on a network interface: several independent token buckets refill in parallel, and a request needs a token from every bucket to be admitted. A request with budget in three of them and none in the fourth is rejected, and the rejection carries no obligation to say which bucket was empty unless you read the headers.

A token bucket is worth describing concretely rather than invoking, because everything else in this chapter is a consequence of its shape. Picture a counter topped up at a fixed rate and capped at some maximum. An arriving request removes as many units from it as the request needs. If the counter holds enough, the request is admitted and the counter drops by that amount; if it does not, the request is rejected and the counter is left alone. The counter refills whether or not anybody is calling, which is why a quiet stretch leaves headroom and a busy one does not, and why every remedy for emptying it is some version of arriving later or arriving smaller.

Two properties of that arrangement produce every surprise in this chapter. The buckets are independent: each has its own refill rate, its own maximum and its own state, and spending from one neither drains nor replenishes another. And admission is a conjunction rather than a choice — a request needs its units from every bucket at the same moment — so the constraint that binds at any instant is whichever bucket is emptiest, which need not be the one anybody is watching. The word "units" is doing different work in each of them, too. The request bucket counts one per call. The input-token bucket counts one per token in the prompt. The concurrency ceiling is not a refilling bucket at all; it is a count of requests currently in flight, decremented when a response completes rather than on any schedule.

That structure is why quota conversations go wrong. A single number invites a single check, and the check almost always lands on requests per minute, because that is the dimension consoles put on the front page and the one a person can estimate from a traffic graph.

The dimension the dashboard shows is rarely the one that binds#

Here is a representative set of ceilings for one model in one region, alongside the offered load at a moment when that account was being throttled. Offered load is the work arriving at the endpoint, which is a different quantity from the work the endpoint agrees to serve.

The offered load behind those figures was 240 interactive requests a minute at about 1,200 input tokens each, which is 288,000 tokens, plus 60 batch requests a minute carrying documents that averaged 28,000 tokens, which is 1,680,000. Long documents dominate the token dimension so completely that the batch contributed 20% of the requests and 85% of the input tokens.

Both rows of the table describe that same traffic, and reading them off it takes two additions. On the request axis, 240 interactive calls plus 60 batch calls is 300 a minute against a ceiling of 500, which is the 60% the console was showing. On the input-token axis, 288,000 plus 1,680,000 is 1,968,000 against a ceiling of 2,000,000, which is 98.4%. The share figures fall out of the same two sums: 60 of 300 requests is one in five, and 1,680,000 of 1,968,000 tokens is a little over 85%. Nothing about the platform produces that asymmetry. It is a ratio inside the workload — a batch document at 28,000 tokens is roughly twenty-three times the size of a 1,200-token chat turn, so one of them displaces about twenty-three of the other on the axis that counts tokens and exactly one of them on the axis that counts calls.

Utilisation at 98.4% of a per-minute ceiling is not a comfortable margin, because these buckets are usually enforced over a sliding window rather than a wall-clock minute, so a ten-second stretch in which a batch's workers happen to align produces an instantaneous rate well above the average and the bucket empties.

The reason there is a token dimension at all is that a request is not a unit of work on this hardware. Serving a 28,000-token document occupies accelerator memory for the whole prefill and holds a key-value cache proportional to sequence length for the whole decode, so it costs the platform something like twenty times what a 1,200-token chat turn costs, and metering by request would let one tenant consume a rack while another consumed a core. Anyone who has sized a connection pool against a database that also enforces a statement timeout and a temp-disk quota has met this structure before: several independent resources, each with its own ceiling, and the one that binds depends on the shape of the work rather than on its volume.

The concurrency row exists for a third reason again, and separating it from the token rate is worth the paragraph because the two are easy to conflate. A request in flight holds accelerator memory for its entire life: the key-value cache grows as the sequence grows and is not released until the final output token has been emitted. What that request costs the platform is therefore memory occupied for a duration, which is a different quantity from tokens per minute and does not follow from it. A workload can sit comfortably under every rate ceiling and still exhaust the memory of the machines serving it, purely by holding many long requests open at once — a hundred slow requests and a hundred fast ones consume the same token rate and very different amounts of memory. That is what the fourth row bounds, and it is why the remedy for hitting it is fewer simultaneous requests rather than smaller ones or fewer ones overall.

Burst shape matters more than the averages suggest. A batch runner with 38 workers, each finishing a 28,000-token document and immediately submitting the next, produces submissions in clusters rather than a smooth stream, and if 20 of those workers submit within the same two-second interval that is 560,000 input tokens in two seconds — an instantaneous rate of 16.8 million a minute against a 2 million ceiling. The sliding window absorbs some of it, the bucket drains, and the next arrivals are rejected even though the minute-averaged figure is still under the line. A dashboard sampling at one-minute granularity can therefore show 98% utilisation and 20% rejections at the same time without either number being wrong.

Both steps of that instantaneous rate are worth seeing rather than accepting. Twenty workers each submitting a 28,000-token document is 560,000 tokens. Expressing an amount observed over two seconds as a per-minute rate means multiplying by thirty, because two seconds is a thirtieth of a minute, and 560,000 times thirty is 16.8 million a minute — more than eight times a ceiling of 2,000,000. The claim is not that the workload is eight times too large; averaged across the minute it is comfortably under the line, as the table says. The claim is that a ceiling expressed per minute but enforced over a shorter window has to be satisfied at every instant, and an instant taken from a clustered arrival pattern looks nothing like the minute that contains it.

The rejection says which bucket, if you look:

HTTP/1.1 429 Too Many Requests
retry-after: 12
x-ratelimit-limit-requests: 500
x-ratelimit-remaining-requests: 197
x-ratelimit-reset-requests: 34s
x-ratelimit-limit-input-tokens: 2000000
x-ratelimit-remaining-input-tokens: 0
x-ratelimit-reset-input-tokens: 12s
x-ratelimit-limit-output-tokens: 200000
x-ratelimit-remaining-output-tokens: 118400

One hundred and ninety-seven requests of headroom and zero input tokens, in the same response. Header names vary across platforms and some expose fewer dimensions than they enforce, but the shape is consistent, and logging these values on every 429 is what turns a multi-day support ticket into a one-line answer. The remedies differ by dimension, which is the practical reason to know which one bound: a request-rate problem is fixed by pacing, a token-rate problem is fixed by chunking the documents, cutting the prefix, or moving the batch to a window where interactive traffic is not also spending the budget.

The belief that produces a four-day support ticket here is not carelessness, and it is worth naming because it is very nearly right. A console showing one number for one model in one region is showing a real ceiling, really enforced, with a real utilisation figure measured against it. Nothing on that page is false. What the page does not say is that it is one of four, and that the one it chose to put on the front is the one a person can estimate from a traffic graph — requests can be counted by eye, tokens cannot. A reader who treats a single utilisation figure as the utilisation figure has made exactly one wrong assumption, that the dimension displayed is the dimension enforced, and every conclusion after it follows correctly from a false premise. That is why the remedy is a habit rather than an insight: log the limit, remaining and reset triple for every dimension the rejection carries, and the premise stops being something anyone has to assume.

Admission reserves the output you asked for, not the output you got#

The output-token bucket has a property that surprises people, and it connects directly to the endpoint defaults from the first chapter. Most platforms cannot know how long an answer will be until it has been generated, so admission control reserves your declared max_tokens against the output budget at the moment the request is accepted, and refunds the difference when the response completes.

# Reserved against the output-tokens-per-minute bucket: 4096.
# Actually generated, measured over 10,000 production calls: mean 260, p99 480.
resp = client.messages.create(
    model=MODEL,
    max_tokens=4096,          # a copy-pasted default nobody revisited
    messages=messages,
)
print(resp.usage.output_tokens)   # 251

The reason admission reserves rather than measures is a sequencing problem with no way around it. Admission has to happen before generation, and the number of output tokens a request will produce is not known until generation has finished, because the model stops when it stops. A platform that admitted on actual output would be deciding after the resource had already been spent, which is not admission control at all. So it charges the budget the only figure available at admission time — the ceiling you declared — and refunds the unused part when the response completes. Your max_tokens is therefore not a limit on what you receive. It is a claim on shared capacity, made on your behalf, held for as long as the request is in flight.

That turns a harmless-looking default into a throughput ceiling, and the arithmetic is one division done twice. At 200,000 output tokens per minute, a max_tokens of 4096 admits 48 requests a minute regardless of how short the answers turn out to be, because 200,000 divided by 4,096 is a little under 49. Setting it to 600, which is p99 plus a margin, admits 333, because 200,000 divided by 600 is 333. The same traffic, the same model and the same answers differ in admitted request rate by nearly sevenfold, on the strength of one number that has no effect whatsoever on any individual response. This is the single most common cause of a service that throttles at implausibly low request rates while its billed output usage sits at a few percent of the ceiling. The fix is one line, and the diagnostic is the gap between reserved output tokens and billed output tokens.

Three numbers on the console, and only one of them is a promise#

Platform consoles typically show a service quota at the account level, a rate limit attached to a specific model in a specific region, and an entry for provisioned throughput. The first two govern on-demand capacity, which is the shared pool the platform meters per token as you use it. They look like three views of one ceiling and they are three different kinds of object.

The last row separates the third column from the other two. A quota you do not use costs nothing, which is why "ask for a raise" is nearly free advice; reserved capacity you do not use is billed at the same rate as reserved capacity you saturate, which makes it a spending decision rather than a capacity setting. It also does not stack the way a larger quota would — the reserved allocation is a separate serving path, and traffic beyond it is either rejected or falls back to the on-demand pool with the on-demand limits still applying. Whether it is worth buying is the arithmetic in the last chapter of this topic.

The error taxonomy matters for the code written around these. A per-minute rate limit produces a 429 with a retry-after and will succeed on a later attempt, so it is exactly the case retries exist for. An account service quota being exhausted often produces a different status entirely — a ServiceQuotaExceededException, or a 403 — and no amount of waiting fixes it, because the ceiling is administrative rather than temporal. Retrying that one burns time and money and gets nowhere.

Every dimension is scoped to a region, including the one you failed into#

Almost every ceiling above is scoped to a region as well as to a model. An account with 2,000,000 input tokens per minute in eu-central-1 does not have 2,000,000 anywhere else; it has whatever the platform's default for a new region is, which is frequently a small fraction of a production allocation and sometimes zero until model access is requested there at all.

That turns a regional failover into two simultaneous problems. Traffic that used to be spread across two regions arrives in one, doubling offered load against a bucket sized for a fraction of it, and the region you failed into may need a support request before it will serve the model at any rate. The failover mechanism is healthy, the DNS is correct, the credentials work, and the endpoint returns 429 for every request. Checking this means reading a quota table per region rather than per model, and it takes ten minutes if done before the failure rather than during it.

The two on-demand limits also interact in a way that is easy to get wrong when several teams share a subscription. A per-model rate limit is not carved out of the account quota as a private allocation; it is an additional ceiling applied to your calls, while the account total is consumed by everyone at once. Two teams each holding a 2,000,000-token-per-minute per-model limit on a subscription whose account ceiling is 3,000,000 will each see their own limit as comfortable and will still throttle each other, and neither team's dashboard shows the other's consumption. Add the two per-model limits and they come to 4,000,000 tokens a minute against an account ceiling of 3,000,000, so the two allocations, both correctly granted, describe more traffic than the account is able to carry. Each team's own limit is genuinely not binding on that team's own traffic, each team's own utilisation figure is genuinely comfortable, and the requests are rejected anyway. Whether a platform models it this way has a documented answer, and it is worth reading before promising anyone a throughput figure.

There is a limit to what observability buys here, and it is worth stating plainly: nothing in your logs or headers tells you what the other teams on the subscription are doing. A remaining-tokens count of zero may mean your batch spent the budget or may mean a data science team started a backfill nine minutes ago, and telling those apart is an organisational problem rather than a technical one.

The teammate who says "we got the quota raised, so we can drop the retry and backoff code" has confused the ceiling's height with its existence. Raising a limit moves the point at which throttling starts; it does not remove throttling, because the burst shape that produced a 429 at 2,000,000 tokens a minute will produce one at 4,000,000 as soon as the workload grows into it, and because the other three buckets were not raised. A larger quota makes throttling rarer, which makes the retry path less exercised and therefore less trustworthy, which is an argument for testing it rather than deleting it.

That coupling to other teams is also why the obvious next step — deliberately driving an account to its ceiling to find out what your client does there — is not a test that can be run casually.