Crossing the ceiling on purpose is the only way to know what happens

A launch on Tuesday is expected to triple traffic. In the readiness review someone asks what happens if the primary provider throttles, and the answer is that there is a fallback to a second provider, written fourteen months ago, with full unit test coverage against a mocked 429. A quick query settles how often it has run: the fallback_engaged counter has been zero for every one of those fourteen months. The code has never executed outside a test that asserts it does what its author assumed the provider does.

That is the actual state of the capacity plan. Every mechanism in this topic — the reservation ledger, the deadline shedding, the jittered retries and the breaker, the regional routing classes — has a branch that only runs when the ceiling is crossed, and none of those branches has been observed under real conditions. The way to observe them is to cross the ceiling deliberately, somewhere it does not matter, which is cheaper than it sounds because the cost of crossing a ceiling scales with the ceiling.

Provision a staging account with a quota you can exhaust: 10,000 tokens per minute rather than the 500,000 production has. At a mean of 1,200 tokens per request that ceiling is about eight requests per minute, so driving three times the limit for twenty minutes costs roughly 600,000 tokens, or about $1.80 of spend. Reproducing the same ratio against production would take 30 million tokens, around $90, and would push a shared account into rejection for whoever else is using it. The small quota is not a compromised version of the test; it is the same test at a ratio you can afford to run every week.

Assert the whole path, not the branch you wrote#

The temptation is to assert that the fallback fires. That is the least interesting property, because it is the one the unit test already covers. What the drill is for is everything the mock could not know.

The mid-stream case is the one that most reliably surprises people. A provider that rejects before generation returns an HTTP 429 with headers, which is easy to catch. A provider that rejects after the response has begun has already sent a 200 and a set of headers, so the rejection arrives as an event inside the stream itself — inside the server-sent events body, the sequence of data: lines that streaming completions are delivered as — and the exception handler wrapped around the call never sees it because the call succeeded. Capturing what actually comes back is the point of the exercise:

HTTP/1.1 429 Too Many Requests
retry-after: 6
x-ratelimit-limit-tokens: 10000
x-ratelimit-remaining-tokens: 0
x-ratelimit-reset-tokens: 5.94s
x-ratelimit-remaining-requests: 47
content-type: application/json

{"error": {"message": "Rate limit reached for gpt-4o-mini in organization
 org-8f2 on tokens per min (TPM): Limit 10000, Used 9840, Requested 1210.",
 "type": "tokens", "param": null, "code": "rate_limit_exceeded"}}

Two things in that response are worth writing assertions against. The remaining-requests header shows 47, which is chapter 1's crossover restated at the moment of failure: the request counter had ample headroom while the token counter was exhausted, and any alerting keyed to the request headers would have been silent. And the reset value is 5.94 seconds while Retry-After says 6, which means a retry policy that ignores the header and uses its own exponential schedule will either return too early and fail again or wait far longer than necessary.

Driving the account there is a short piece of code. The driver ramps offered load past the ceiling, holds it, and records the shape of everything that comes back rather than aggregating it away.

async def drive(target_tpm: int, seconds: int, tokens_per_req: int) -> None:
    rate = target_tpm / tokens_per_req / 60          # requests per second
    observed: list[dict] = []
    deadline = time.monotonic() + seconds

    async def one() -> None:
        t0 = time.monotonic()
        try:
            r = await client.chat.completions.create(**REQ, stream=True)
            chunks = [c async for c in r]
            observed.append({"outcome": "ok", "chunks": len(chunks),
                             "ttft": ..., "latency": time.monotonic() - t0})
        except RateLimited as e:
            observed.append({"outcome": "429", "retry_after": e.retry_after,
                             "headers": dict(e.headers), "phase": e.phase,
                             "latency": time.monotonic() - t0})

    async with asyncio.TaskGroup() as tg:
        while time.monotonic() < deadline:
            tg.create_task(one())
            await asyncio.sleep(random.expovariate(rate))   # Poisson arrivals

Poisson arrivals matter more than the average rate. A driver that fires requests on a fixed interval produces perfectly smooth load, which never happens in production and which hides exactly the burst behaviour the ceiling test exists to find. Sampling inter-arrival times from an exponential distribution gives clustering at the same mean, and the clusters are what push a marginal minute over the limit.

The second provider is a second capacity plan#

Diverting traffic to another provider moves the problem rather than removing it, and the receiving system has its own ceiling, its own accounting and its own tokenizer. A fallback account that was created during an evaluation and left on a trial quota of 30,000 TPM cannot absorb 200,000 TPM of diverted load, and the drill's most valuable finding is often that number, discovered on a Thursday afternoon rather than during the launch.

The tokenizer difference is subtler and affects every reservation. The same English paragraph can differ by 8 to 15 percent between two vendors' tokenizers, and code or non-Latin scripts diverge further, so a workload measured at 200,000 TPM against provider A may present as 225,000 TPM against provider B. The reservation logic from chapter 2 therefore needs per-provider token counting rather than one shared estimate, and the per-route max_tokens caps need checking against each provider's parameter semantics, since the field name, the inclusive-or-exclusive treatment of the prompt, and the behaviour when the cap is reached all vary.

Ordering is the last thing to pin down, and it is a policy question with a measurable answer. If the client retries three times with full jitter before considering the fallback, the expected delay before provider B is asked anything is the sum of three jittered waits, which with a 1-second ceiling averages about 1.5 seconds and can reach 3. Against a 25-second deadline that is affordable; against a 3-second conversational budget it is not, and the correct policy is to fall back on the first 429 and retry only within the fallback. The drill measures the actual distribution, which is the only way to know which of those you built.

What this drill is not#

It is not a load test. A load test asks how much throughput a system sustains and what its latency looks like at each level, and it wants production-sized capacity and a long steady run. This asks what the system does in the seconds either side of a limit it has already reached, and it wants the smallest possible ceiling so that reaching it is trivial. The two rigs answer different questions, and running the ceiling drill on production-sized infrastructure mostly buys a large bill and a long feedback loop.

The absolute numbers from a small-quota account do not transfer, either. Crossing a 10,000-TPM ceiling tells you the shape of the rejection, the ordering of the code paths and the correctness of the accounting; it tells you nothing about whether 500,000 TPM is enough for the launch, which is the sizing question chapter 1 answers from measured demand. Conflating the two produces a green report on a test that never asked the sizing question.

A fault-injection proxy that returns synthetic 429s is a useful complement and a poor substitute. It gives you determinism, the ability to test in CI, and the ability to reproduce a specific header combination on demand, and it can only ever return the shape you programmed into it, which is the shape you already believed. The real small-quota account returns the shape you did not know about — the mid-stream error, the missing header, the message that changed format in a provider update last month. Use the proxy for regression tests of the handling code and the real account for discovering what the handling code should do.

Nothing here stays true indefinitely. Providers change error payloads, add and remove rate-limit headers, and adjust whether a rejection lands before or during generation, generally without an announcement that reaches you. That makes the drill a recurring item — before each significant launch, and on a schedule otherwise — and it argues for a small synthetic canary in production — a scripted request the system sends to itself on a schedule, purely to check that a path still behaves as it did — that deliberately trips a low per-route budget once an hour and alerts if the response shape stops matching the assertions.

Putting the topic to work looks like a short sequence run against any service before it takes real load. Measure the token distribution of each route and compute the crossover against each region's own quota, so the plan is stated in tokens per minute rather than in QPS. Set per-route max_tokens from the observed p99.9 and reserve prompt plus that cap at admission, settling against reported usage. Give every request a deadline, bound the queue in seconds of work rather than in requests, and design the shed response as a real product state. Configure one retry policy with full jitter and a budget, disable the ones in the SDK and the gateway, and put a breaker in front. Build the unit ladder before asking finance for reserved capacity, and know which region exists for latency and which for residency. Then take a staging account with a deliberately small quota and run the whole thing over its ceiling, because every one of those decisions is an assumption until it has been crossed on purpose.