Reserved throughput is billed on allocation, not on usage

Finance has a vendor quote and a question. Reserved throughput — capacity the provider sets aside for one account at a fixed price for the term, held whether or not anything is sent through it, and sold by the major platforms as provisioned throughput units — is quoted in units of 100,000 tokens per minute at $2,000 per unit per month; on demand, where you are charged only for the tokens you actually send, is $3 per million tokens. Production measures 250,000 TPM at the median, 900,000 TPM at p99, and 40,000 TPM overnight. How many units should we buy?

The answer that arrives first from most engineering teams is nine, because nine units covers the p99 and nine units means never being throttled again. Nine units is $18,000 a month, and at a 250,000-TPM median it runs at under 30 percent utilisation, which means most of those units cost more per token than simply buying the tokens. It also converts a bill that tracked demand into a fixed cost that does not fall when a customer churns or a feature is deprecated. And it does not actually deliver "never throttled", because p99 is not p100 and the traffic above 900,000 TPM still has to go somewhere.

One unit is 4.32 billion tokens, and it costs $2,000#

Reserved throughput is a rate, and a rate held for a month is a quantity. A unit of 100,000 tokens per minute, held for every minute of a 30-day month, is 100,000 × 60 × 24 × 30 tokens, which is 4.32 billion. Those same tokens bought on demand at $3 per million would cost $12,960. The unit costs $2,000. Dividing gives the break-even: a unit pays for itself once its average utilisation across the month exceeds $2,000 / $12,960, which is 15.4 percent.

UNIT_TPM        = 100_000
UNIT_COST_MONTH = 2_000
ON_DEMAND_PER_M = 3.0
MINUTES         = 60 * 24 * 30          # 43,200

tokens_per_unit  = UNIT_TPM * MINUTES               # 4.32e9
on_demand_value  = tokens_per_unit / 1e6 * ON_DEMAND_PER_M   # $12,960
breakeven        = UNIT_COST_MONTH / on_demand_value          # 0.154

That number is unintuitively low, and it is the reason reserved capacity is worth taking seriously rather than dismissing as an enterprise procurement ritual. A unit only has to be busy about one minute in seven, averaged over the month, to be cheaper than on demand. Where the intuition goes wrong in the other direction is in assuming the whole allocation shares that utilisation. It does not, because demand is not flat, and each additional unit you buy is used only during the hours when demand reaches its band.

Buy units one at a time and check each against the break-even#

The median, p99 and overnight figures are three points on a curve, and the decision needs the curve. A representative day for this workload, from the same measurements, spends its 24 hours roughly like this: 8 hours overnight at 40,000 TPM, 4 hours of ramp at 150,000, 8 hours of business load at 250,000, 3 hours of afternoon peak at 450,000, and one hour at the 900,000 p99. Mean demand across the day is 215,000 TPM, which is 9.31 billion tokens a month, or $27,930 if every token is bought on demand.

Now evaluate each unit separately. Unit 1 serves the band from 0 to 100,000 TPM, so it is fully used in every hour except the overnight ones, where demand of 40,000 uses 40 percent of it. Unit 3 serves 200,000 to 300,000 TPM, so it is idle overnight and during the ramp, half used during business hours, and full during the two peaks. Each unit's average utilisation is the demand that falls inside its band, divided by the band's width, weighted by hours.

The ladder collapses fast, which is what a peaky workload always does. Three units is the confident answer and the fourth is a coin flip sitting one and a bit points above break-even. Checking the totals confirms that the decision is flat near the optimum: three units costs $6,000 plus 1.89 billion overflow tokens at $5,670, for $11,670; four units costs $8,000 plus 1.17 billion overflow tokens at $3,510, for $11,510. The two options differ by $160 a month, so the tie-break is not money — it is that three units carries less commitment risk if the traffic forecast is wrong, and four units leaves more headroom before overflow behaviour becomes visible. Against the $18,000 that nine units would cost, either is a saving of about $6,400 a month, and against $27,930 of pure on-demand it is a saving of roughly 58 percent.

Everything above the reserved band has to be served by something, and that something is the on-demand deployment, reached either by your own router or by the platform's own spillover feature. Azure's provisioned deployments can be configured to spill over to a standard deployment when the provisioned capacity is saturated, which is documented in "Operate provisioned throughput deployments in production" along with the utilisation metric you need to watch. Whichever mechanism carries the overflow, the request that spills is subject to on-demand quotas, on-demand latency variance, and on-demand availability, so the overflow path is a second capacity plan rather than an afterthought — and it is a path that, on the numbers above, carries 20 percent of your tokens.

A reserved allocation that sits at 80 percent utilisation on unit 1 and 33 percent on unit 3 is also an invitation to put more than one workload on it, since the bands that are idle overnight are idle for every team at once. Sharing works, and it needs a policy at the front door rather than shared credentials, because otherwise the first workload to burst consumes the allocation the interactive path was bought for. "Sharing Azure OpenAI Provisioned Throughput across use cases" walks through the usual arrangement, with API Management in front holding per-consumer token budgets and deciding which callers may use the provisioned deployment and which are sent to on demand. The routing rule that matters is that latency-sensitive traffic gets the reserved capacity and everything batch-like is sent to the overflow path by default, rather than both competing and the loser discovering it during a peak.

Utilisation is measured in the platform's units, not in yours#

The arithmetic assumes a unit of reserved capacity is 100,000 tokens per minute of your traffic, and real platforms do not sell it in those terms. Provisioned throughput units are sized as a processing rate, and how many tokens per minute a unit yields depends on the model, on the ratio of input to output tokens, and on the request shape, because generating output is substantially more expensive per token than reading input. "Provisioned throughput for Foundry Models" gives the mapping and the sizing calculator, and the practical consequence is that the tokens-per-minute figure you get from a unit is a function of your workload rather than a constant on a price list. A workload that shifts from summarisation, which is input-heavy, to long-form drafting, which is output-heavy, can lose a third of its effective capacity on the same allocation without anyone changing a setting.

The other measurement that has to be right is the demand curve itself. The ladder above weighted utilisation by hours in a representative day, which is only valid if days resemble each other. A workload with a heavy Monday and a quiet weekend has a different answer, and a workload driven by monthly reporting cycles — three days at the end of every month at four times normal — will show a median that justifies two units and a lived experience of severe throttling seventy-two hours a month. Build the ladder from at least a full month of minute-resolution data, and if the shape is dominated by a periodic spike, consider whether the spike is shiftable, because a batch that can run overnight lands in the hours where units 1 and 2 sit at 40 percent utilisation and costs nothing extra at all.

The fact that would make you buy no units is a forecast horizon shorter than the commitment term. If the reservation is a twelve-month commitment and the feature it serves is a pilot with a quarterly review, or the model it is pinned to will be deprecated in five months, then the relevant number is not utilisation but the probability that the traffic still exists at month nine. Reserved capacity is normally tied to a specific model and version in a specific region, so a model upgrade mid-term can strand the allocation. The same conclusion follows if the measured mean demand is below roughly 15,000 TPM, which is where even the first unit falls under break-even, or if the workload is entirely asynchronous and can use a batch endpoint at a discount instead, since batch trades latency for price and reserved throughput is a way of buying latency.

There is a softer version of the same objection that catches teams who have done the arithmetic correctly. Reserved capacity removes rejections only in the band it covers, and the value of removing them is not uniform across that band. Three units eliminate throttling for the 80 percent of tokens below 300,000 TPM, which were rarely being throttled anyway because they sat well inside the on-demand quota; the throttling everyone actually remembers happened during the afternoon peak, in the bands where the ladder says not to buy. If the reason finance is being asked for money is an incident rather than a cost model, the ladder will produce a purchase that is financially optimal and does nothing about the incident, and the honest thing to say is that the peak is a shedding and overflow problem rather than a procurement one.

Reserved throughput is also not a rate-limit exemption. It replaces one ceiling with another, and its failure mode is different rather than absent: a saturated provisioned deployment does not usually reject cleanly, it queues and stretches latency, so the symptom is a time-to-first-token that drifts from 300 ms to several seconds while the error rate stays at zero. That is harder to alert on than a 429 and easier to misdiagnose as a model problem. Nor is it a discount on tokens you were already buying; it is a bet that a given band of demand will show up, and the bill arrives whether it does or not.

Every number in this chapter has been a global one, and capacity is not global. Units are allocated in a region, quotas are enforced per region and per subscription, and the question of which regions to buy in turns out to be governed less by arithmetic than by two goals that most design documents state in the same sentence without noticing that they conflict.