A request is one unit of RPM and thousands of units of TPM

A document-summarisation endpoint at a legal-tech company had run at roughly 300 requests per minute for six months without a single throttle. The account it called was limited to 500 requests per minute and 300,000 tokens per minute, and the dashboard the on-call team watched showed request rate against the 500 line: a comfortable 60 percent, flat, boring. Then marketing shipped a landing page, traffic rose 27 percent to about 380 requests per minute, and the provider began returning 429s on roughly one call in six. The on-call engineer looked at the dashboard, saw 76 percent of the request limit, and opened a support ticket claiming the provider was throttling below the documented quota.

The support ticket was closed with an explanation rather than a fix, and the explanation is where this topic starts. A hosted model endpoint does not sell you a machine with a known number of cores; it sells you a share of a shared fleet, metered by two counters that run side by side and reset every minute. RPM, requests per minute, counts calls, one unit per call whatever the call contains. TPM, tokens per minute, counts the text those calls carry — tokens being the sub-word fragments a model reads and emits and is billed by — so a single call spends one unit of RPM and anywhere from a few hundred to a few hundred thousand units of TPM. When either counter is exhausted the provider refuses further calls with HTTP 429, Too Many Requests, until the window rolls over. Capacity planning for model traffic is the work of deciding ahead of the traffic whether the demand a feature will place on those two counters fits underneath them, how much capacity to buy if it does not, and what the service should do during the minutes when it does not fit anyway. It shows up in three places: the quota page of a provider console, where the pair of numbers is set; the readiness review before a launch, where somebody has to state how much traffic the feature will bring; and the incident, where that statement turns out to have been made in the wrong unit.

The provider was not throttling below the documented quota. The landing page had brought a different kind of document — sales contracts and lease agreements rather than the two-page memos the endpoint had been fed until then — and the mean prompt had grown from 620 tokens to 740. Completions grew too, because a longer contract produces a longer summary, from a mean of 190 to 205. That is a 17 percent increase in tokens per request arriving on top of a 27 percent increase in requests, and the product of the two is what mattered.

The account had been at 81 percent of its token quota for six months while the dashboard everyone trusted said 60 percent. There was never much headroom; it was just on a counter nobody was watching. A capacity plan that tracks the wrong counter does not fail gradually, because the two counters move independently and the safe-looking one keeps looking safe right up to and through the incident.

Every account has a crossover token count#

Two limits on the same account define a single number that tells you which one binds: divide the token limit by the request limit. This account allows 300,000 tokens and 500 requests each minute, so it implicitly assumes an average request of 600 tokens. A workload whose mean request is lighter than 600 tokens will exhaust the request counter first; a workload heavier than 600 will exhaust the token counter first. The summariser's mean request was 810 tokens before the launch and 945 after, so it was on the token side of the crossover the entire time and the request limit was never going to be the thing that stopped it.

The crossover is worth computing once for every account you own and writing next to the quota in the runbook, because it converts a two-dimensional problem into a one-line check. It also tells you which increase to ask the provider for. Requesting more RPM on an account whose workload sits at 945 tokens per request buys nothing; the token counter binds at 317 requests per minute regardless of how high the request ceiling goes.

The number is only as stable as the mix that produced it, which is the part that catches teams out. The summariser's crossover never moved, because it is a property of the account, but the workload's position relative to it moved by 17 percent in a week without a single line of code changing. Any input whose size is chosen by users rather than by you — uploaded documents, pasted transcripts, conversation histories that grow turn by turn, retrieved chunks whose count depends on a similarity threshold — makes mean request weight a live variable. Treating it as a constant measured once during design is how a plan that was correct in March becomes wrong in June.

TPM_LIMIT = 300_000
RPM_LIMIT = 500

crossover = TPM_LIMIT / RPM_LIMIT           # 600 tokens per request

mean_prompt = 740
mean_completion = 205
mean_request = mean_prompt + mean_completion  # 945

if mean_request > crossover:
    binding = "TPM"
    max_rpm = TPM_LIMIT / mean_request       # 317 requests/min
else:
    binding = "RPM"
    max_rpm = RPM_LIMIT

print(binding, round(max_rpm))               # TPM 317

Nothing in that calculation is specific to a provider. Any platform that publishes a pair of per-minute limits — and most of the ones covered in 2.19 publish exactly that pair — has a crossover, and any workload has a mean request weight to compare against it. What differs between platforms is the accounting detail underneath: whether the completion is charged at the moment it is generated or estimated up front from max_tokens, whether the minute is a sliding window or a fixed bucket, and whether the counter is enforced on sub-minute intervals so that a burst inside one second is rejected even though the minute as a whole would have fit. Those details decide how much of the nominal quota you can actually reach, and they are worth measuring against your own account rather than reading off a docs page.

Fifty QPS is a number about the wrong dimension#

Take a capacity plan that says a new chat feature is "sized for 50 QPS" against an account limited to 5,000 RPM and 2,000,000 TPM, where a typical request carries a 4,000-token prompt and generates an 800-token completion. Fifty QPS is 3,000 requests per minute, which is 60 percent of the request limit, and a reviewer who checks only that line will approve the plan and possibly ask for request headroom while they are at it.

The crossover for that account is 2,000,000 divided by 5,000, which is 400 tokens per request. The workload's actual weight is 4,800 tokens, twelve times the crossover. At 3,000 requests per minute the workload demands 14.4 million tokens per minute against a ceiling of 2 million, so it is over by a factor of 7.2, and the maximum request rate the account can sustain is 2,000,000 / 4,800, which is about 417 requests per minute, or roughly 6.9 QPS. The plan is not slightly optimistic. It is asking for seven times the capacity the account has, and the check that passed was a check on a counter that had nothing to do with the constraint.

The sentence a product manager will accept is that the provider charges and limits us by tokens rather than by requests, and each of our requests is twelve times heavier than the average request this account's quota assumes, so a request rate on its own says nothing about whether the workload fits. That framing survives the follow-up question, which is always some version of "so what do you need instead".

What you need instead is a demand figure in tokens per minute, stated with a distribution rather than a mean, plus the request rate as a secondary number for the cases where it binds. Producing it means measuring four things about live or prototype traffic. The prompt token distribution matters more than its mean because a p95 prompt of 18,000 tokens — p95 being the length only one prompt in twenty exceeds — on a p50 of 4,000 will dominate the minutes it appears in. The completion distribution matters for the same reason and is harder, because completion length depends on the model and the instruction rather than on the input you can inspect. The arrival pattern has to be measured at the granularity the limit is enforced at, so an hourly average is close to useless when the counter resets every minute — a workload averaging 800k TPM over an hour can spend six of those minutes above 2M. And the mix matters, because a single endpoint usually serves several call shapes whose weights differ by an order of magnitude, and the mean across them moves whenever the mix moves, which is exactly what happened to the summariser.

What the token count does not tell you#

Sizing in tokens is not the same as costing in tokens, and the two use different weights. Providers price input and output separately, with output typically several times more expensive, so a workload that is 90 percent prompt by token count can be under half prompt by spend. A request of 4,000 prompt and 800 completion tokens is 4,800 units against the TPM limit either way, but at, say, $3 per million input and $15 per million output it costs $0.012 for the prompt and $0.012 for the completion. Cached prompt prefixes widen the gap further, because a cached read is discounted on the bill while still being counted, in full or in part depending on the provider, against the throughput limit. Use the token plan to answer "will it fit" and a separate calculation to answer "what will it cost".

Nor does a TPM number account for every limit an account carries. Platforms commonly enforce a concurrency or in-flight request cap alongside the per-minute counters, and a workload with long streaming completions can hit that cap at a request rate far below the TPM ceiling: 200 concurrent streams each running 30 seconds is only 400 requests per minute, which may be nowhere near either published limit, and it will still be refused if the concurrency cap is 100. Batch and fine-tuning endpoints usually have their own separate quotas that do not draw from the same pool, and image or audio modalities are often metered in units that are not tokens at all.

Token accounting is also an estimate on your side of the wire, however exact it looks. Counting a prompt with the model's tokenizer gives you the user-visible text, and the request that actually leaves your process carries a system prompt, tool and function schemas serialised into the payload, any retrieved context, and per-message structural overhead that the provider counts and your naive len(encode(user_text)) does not. Teams routinely find their local count sits 8 to 15 percent below the usage.prompt_tokens the provider reports back, and a systematic 12 percent undercount is a systematic 12 percent overshoot of the quota. Reconciling your estimate against the reported usage on every response, and alerting when the ratio drifts, costs almost nothing and catches the day someone adds a fourth tool definition to the shared client.

Finally, having token headroom does not mean the requests will be fast. Throughput and latency are different properties of the same deployment, and a shared endpoint running at 40 percent of its token quota can still show a time-to-first-token that doubles when a neighbouring tenant sends a burst. Capacity planning answers whether your demand fits inside a ceiling; it does not promise anything about the shape of the response time distribution underneath that ceiling, and a plan that quietly conflates the two will pass its own review and disappoint its users.

Knowing the binding number is a different problem from enforcing it. The summariser team could compute 317 requests per minute in an afternoon, but a service cannot admit a request based on a mean, and at the moment it has to decide whether to dispatch, it knows the prompt exactly and knows nothing at all about the completion that request will produce. That gap is where the next chapter starts.