Bounding in-flight tokens

The second published allowance is tokens per minute: a ceiling on how many tokens the account may present in a rolling window, frequently split into separate input and output pools. It is on the same documentation page as the requests-per-minute number and it is the one that tends to be left out of the arithmetic, because the concurrency bound derived in the previous chapter — a count of calls in flight, from the published rate and the observed latency — mentions calls and says nothing about their size.

That works until prompt sizes grow. A job whose calls carry 250 tokens each is limited by request rate and never notices the token pool; the same job with a retrieved-context section in its template carries 3,200, and the token pool becomes the binding constraint while the request gauge reads exactly what it always did. This chapter derives the second bound and builds the gate that enforces it, a gate whose slots have sizes.

Two budgets, drained by different traffic#

Converting the token allowance into an in-flight bound is the same Little's law derivation as before — in-flight equals rate times latency — run on a quantity that varies per call rather than one that is always exactly one:

in_flight_calls  = rpm_allowed / 60 * headroom * latency_seconds
in_flight_tokens = tpm_allowed / 60 * headroom * latency_seconds

Take an account rated at 500 requests and 200,000 input tokens per minute, running a summarisation job at a 1.4-second p50. The request allowance permits 500 / 60 × 0.8 × 1.4 = 9.3 calls in flight. The token allowance permits 200,000 / 60 × 0.8 × 1.4 = 3,733 tokens outstanding at any instant. If each call carries about 3,200 input tokens plus 256 reserved for the answer, then five calls hold 17,280 tokens in flight against an allowance of 3,733, which is 4.6 times over, and the bound that binds is roughly one call.

Each of those numbers is the same three operations applied to a different allowance. On the request side, 500 divided by 60 gives calls per second, the 0.8 headroom shaves a fifth off that, and multiplying by the 1.4-second latency gives the 9.3 calls that may be outstanding without the sustained rate passing the allowance. On the token side the identical three operations run on 200,000 rather than on 500 and give 3,733 tokens outstanding. The workload is then priced against them: 3,200 input tokens plus 256 reserved for the answer is 3,456 per call, five of those in flight is 17,280, and 17,280 over 3,733 is about 4.6. Dividing the other way is the more useful direction — 3,733 over 3,456 is a fraction above one — so the token allowance admits a single call at a time while the request allowance is still offering nine.

Both readings are correct simultaneously, which is why this is hard to see from a dashboard: request concurrency at 55% of its ceiling, request rate at 214 per minute against 500, and three quarters of the traffic refused.

The rows do not shade into one another; the binding limit flips between them on a change of prompt template. Reranking is limited by request count by a factor of four, the review-enriched summary is limited by tokens by a factor of nine, and all four are the same client calling the same account with the same key. A single derived constant is correct only for the workload it was measured on, which is the same failure the latency table in the previous chapter described from the other direction.

The middle column is one division, and it is what makes the table readable. Calls a minute that the token allowance permits is 200,000 divided by the budgeted tokens per call: 200,000 over 282 is 709, over 3,456 is 58, over 98 is 2,040, and over 13,024 is 15. Set each of those against the 500 calls a minute the request allowance permits and the binding limit falls out with no further reasoning — 709 and 2,040 sit above 500, so requests bind, while 58 and 15 sit below it, so tokens bind. The last row's in-flight figure comes from the same three operations at a 6-second latency, giving 16,000 tokens, and a single 13,024-token call fits inside that with nothing left over for a second. Nothing about the client changed between any two of those rows. The only thing that moved is how many tokens a call carries.

The last row shows a ceiling on the work rather than on the concurrency. Fifteen contract summaries per minute is what 200,000 tokens per minute buys when each summary reads 12,000 tokens of contract, and no client-side setting changes that. A thousand contracts is 67 minutes of wall clock at the account's absolute best, and every useful lever is upstream of the client: shorten the input, split the document and summarise the parts, cache the shared prefix, or buy more quota.

Telling which budget refused you#

A 429 carries no useful information in its status code, and a job needs to know which pool it exhausted, because the two failures call for opposite corrections. Every major provider answers that question twice — once in the error body:

{
  "type": "error",
  "error": {
    "type": "rate_limit_error",
    "message": "This request would exceed your organization's rate limit of 200,000 input tokens per minute."
  }
}

and once in the rate-limit headers present on every response, including successful ones:

Sampling both remaining counters into gauges on every response costs nothing and turns the next diagnosis into a glance. The characteristic shape of a token-bound failure is remaining-requests sitting comfortably above half its limit while remaining-tokens bottoms out within the first eight seconds of each minute and stays at zero until the window resets — which is also a picture of what the job could have been doing with the 60% of request quota it is leaving on the floor.

The belief this chapter argues against is worth writing out, because it is what lets the failure last for weeks. It is that the in-flight gauge is a statement about how hard the client is pushing. It is not. The gauge counts calls, the provider is metering two quantities, and only one of them is calls — so a gauge pinned exactly at its configured value is entirely consistent with three quarters of the account's traffic being refused. A reader who comes away able to say "concurrency looks fine" on the strength of a call gauge alone has taken away the thing that was wrong.

The third row is the one that costs an afternoon. If both counters look healthy and the provider is still refusing you, the limit being enforced is not one of the two on your account page. Per-model pools are the common cause, since a large model and a small one frequently draw on different allowances and only one of them appears in the dashboard you were reading. Shared organisation ceilings are the next, and they have the property that your job's behaviour did not change at all — somebody else's did.

The reservation you did not know you were making#

A provider cannot know how long a response will be until it has generated it, so admission control — the check it runs against your remaining allowance before it agrees to serve a request at all — charges the budget using the ceiling you supplied. max_tokens is a reservation, debited when the request is admitted and reconciled downwards when the response completes.

The causal chain there is short and every link is checkable. A provider has to decide whether to admit a request before it runs the model, because the purpose of admission control is to refuse work it lacks capacity for rather than to discover the shortfall halfway through generating an answer. Deciding requires a number for how much of your allowance this request will consume. The input side it can count exactly, since the prompt is in its hands. The output side does not exist yet and cannot be predicted from the prompt, so the only figure available is the upper bound you supplied. max_tokens is therefore not a safety limit the provider ignores when the answer turns out to be short; it is the quantity the admission check runs against, and the answer being short is established several seconds after the decision that mattered.

That turns a habit into a throughput bug. Setting max_tokens to 4096 because it is a safe default, on a call whose answer is a two-sentence summary of about 40 tokens, budgets 4096 tokens against an allowance of 200,000 per minute:

# 4,346 tokens of budget for a 40-token answer
{"model": MODEL, "max_tokens": 4096, "messages": [...]}

# 282 tokens of budget for the same answer
{"model": MODEL, "max_tokens": 32, "messages": [...]}

A 250-token prompt with a 4096-token reservation costs 4,346 tokens of budget instead of 282, cutting what the account can serve from 709 calls per minute to 46. Fifteen times the throughput disappears into space reserved for output nobody generates, and the only visible symptom is a rate-limit error mentioning a number far larger than anything the job appears to be sending.

Both figures in that sentence come from one addition and one division. A 250-token prompt carrying a 4096-token reservation budgets 250 plus 4096, which is 4,346 tokens against the window. What the account can then serve per minute is 200,000 divided by that, which is 46 calls, against the 709 it serves at a budget of 282. The ratio between 709 and 46 is about fifteen, and it is very close to the ratio between 4,346 and 282 for a reason worth noticing: the division is linear in the budget, so multiplying the per-call cost by some factor divides the throughput by the same factor. That is what makes an over-large ceiling expensive in exact proportion rather than approximately.

How large that effect is depends on the provider's accounting, which is worth reading rather than assuming. Some meter input and output against separate pools, some meter one combined pool, and cached prefix tokens are frequently discounted or excluded — a job with a large shared system prompt and prompt caching enabled, so that the provider reuses its work on the unchanged opening of every call, may have several times the token headroom a naive estimate suggests. Whether the unused reservation is refunded on completion or held for the rest of the window also changes how much an over-large ceiling costs.

A gate whose slots have sizes#

Enforcing a bound on outstanding tokens needs a semaphore with weighted acquisition, which asyncio does not provide. The primitive is short, and the interesting parts are the two edge cases rather than the counter.

class TokenGate:
    """A weighted semaphore: callers reserve an estimated token cost."""

    def __init__(self, capacity: int) -> None:
        self._capacity = capacity
        self._available = capacity
        self._cond = asyncio.Condition()

    @contextlib.asynccontextmanager
    async def reserve(self, tokens: int) -> AsyncIterator[None]:
        weight = min(tokens, self._capacity)      # oversized items must still fit
        async with self._cond:
            while self._available < weight:
                await self._cond.wait()
            self._available -= weight
        try:
            yield
        finally:
            async with self._cond:
                self._available += weight
                self._cond.notify_all()           # one release may free several waiters

Clamping the weight to the capacity is not a rounding convenience. A contract call estimated at 13,024 tokens against a gate sized at 3,733 would wait on a condition that can never become true, so the job stops with nothing in flight, no error and no log line, which is among the hardest failures to diagnose from outside. Clamping admits it alone and accepts the overshoot, which the provider answers with a 429 you can retry — and a rejection you can see is worth more than a deadlock you cannot.

notify_all rather than notify is the second detail. Releasing 3,456 tokens may satisfy one large waiter or thirty small ones, and waking a single arbitrary waiter can leave capacity idle behind a waiter that does not fit. Waking everyone costs loop scheduling proportional to the waiter count, which is negligible at the queue depths this gate produces.

Both bounds run together, in the acquisition order the previous chapter established, because satisfying one says nothing about the other:

CALL_GATE = asyncio.Semaphore(9)          # 500 rpm ÷ 60 × 0.8 × 1.4 s
TOKEN_GATE = TokenGate(3_733)             # 200,000 tpm ÷ 60 × 0.8 × 1.4 s
MAX_OUTPUT = 256


async def summarise(client: httpx.AsyncClient, entry: Entry) -> str:
    body = prompt_for(entry, max_tokens=MAX_OUTPUT)
    estimated = count_tokens(body) + MAX_OUTPUT
    async with CALL_GATE, TOKEN_GATE.reserve(estimated):
        response = await client.post(ENDPOINT, json=body)
    response.raise_for_status()
    payload = response.json()
    ESTIMATE_DRIFT.observe(payload["usage"]["input_tokens"] - count_tokens(body))
    return payload["content"][0]["text"]

The call gate is slack there — nine permitted, one admitted by the token gate — and it stays in the code because the binding constraint moves when the template does, and a bound that only exists while it is the tight one is a bound that will be missing on the day it is needed.

This gate bounds tokens outstanding, which is a different instrument from the token-bucket limiter the previous chapter paired with the semaphore. If you want a clock on tokens as well, aiolimiter's AsyncLimiter takes a weighted acquisition, so AsyncLimiter(160_000, 60) with await limiter.acquire(estimated) paces token spend per minute directly. Running both matters for the same reason it did for requests: the rate limiter alone will happily let 160,000 tokens' worth of payload sit in your process simultaneously if latency is high, and the in-flight gate alone drifts as latency moves.

The estimate itself comes from the provider's published tokenizer where there is one, and from a character heuristic where there is not. Four characters per token is close for English prose and badly wrong for code, JSON and non-Latin scripts, where it can under-count by half — which matters because an under-estimate makes the gate admit more than the account allows. The usage block on every response is ground truth, and exporting the difference between estimate and actual is what says whether the gate is holding a real bound or a comfortable fiction.

Where a token bound stops helping#

The gate constrains the rate at which tokens are presented, which is what the provider meters, and it says nothing about what those tokens cost. Two hundred thousand tokens per minute is an allowance, not a budget: a job that runs inside it for six hours has spent 72 million tokens and violated nothing. Spend control is a separate counter with a separate stopping rule, and conflating the two produces a job that is perfectly compliant and financially surprising.

Streaming complicates the reservation in a way worth knowing before it surprises you. A streamed response holds its slot and its reserved tokens from the request until the final chunk, so a call that reaches its first token in 400 ms and finishes in 9 seconds occupies 3,456 tokens of gate capacity for the full 9 seconds. Sizing the gate from a time-to-first-token measurement will over-admit by a factor of twenty, and the symptom is a job that looks responsive on every latency dashboard while being refused constantly.

That factor is a ratio between two durations rather than a rule of thumb. A call reaching its first token in 400 ms and finishing in 9 seconds holds its reservation for the whole 9 seconds, and 9 seconds is a little over twenty times 400 ms. Size the gate from the first duration and it will admit roughly twenty times as many calls as the second duration can support. Every latency dashboard reports the first number, because time-to-first-token is what a user experiences, which is why this particular mis-sizing survives review by people who are looking at real measurements of a real system.

Like every gate so far, this one lives inside a process, so eight workers each holding TokenGate(3733) present 29,864 tokens in flight to an account that permits 3,733. Dividing capacity by the replica count works while the replica count is fixed and stops working the moment autoscaling touches it; the alternatives are a shared counter in Redis or a metering proxy in front of the provider, meaning a small service every client calls through so the counting happens in one place. Chapter 7 returns to this once bulk jobs and interactive traffic draw on the same allowance.

The instrument this chapter adds is therefore a second derivation and a second gate: tokens in flight bounded by tpm / 60 × headroom × latency, with the call bound kept alongside it so that whichever limit is tight today is enforced without a code change tomorrow. Both bounds are estimates built from a published limit and an observed latency, and providers do not publish the whole truth. Burst allowances, per-model pools, organisation-wide ceilings shared with other teams and short-window smoothing all mean the real serving capacity is not what the arithmetic says — usually more, occasionally less. That makes the derived bound a safe starting point rather than an answer, and it raises the question of how far above it you should push, which is where the 0.8 headroom factor gets challenged.