Costing provisioned throughput

Provisioned throughput is reserved inference capacity: a platform sets aside a defined rate of work for your account, bills you for the whole term whether or not you use it, and in exchange stops metering you per token on that path. It is sold under several names — model units, provisioned throughput units, PTUs — and it is the thing procurement asks about after the first throttling incident, usually framed as buying a bigger quota.

It is not a bigger quota. A quota is a ceiling on you and costs nothing when unused; a reservation is a promise to you and costs the full rate for every minute held. That difference makes the buy-or-not decision an arithmetic problem with three inputs: how much of a unit the workload actually consumes, what shape that consumption has through the day, and what the reservation costs against per-token pricing for the same volume. This chapter works all three, and the arithmetic is easy. It gets skipped because its inputs are in a unit nobody on the team has ever measured.

A unit is denominated in the platform's capacity measure, not in requests#

Each unit is a rate of work rather than a count of calls. For a typical long-document workload the platform's capacity documentation defines one unit as 600,000 weighted tokens per minute, where an output token counts as six input tokens, because output tokens are generated one at a time and occupy the accelerator far longer than a prefill token does.

That last clause is the part worth unpacking, because a weighting factor looks arbitrary until the mechanism behind it is in view. Processing the prompt — the prefill — puts every token of it through the model together, because the whole input is known in advance and the hardware can work across it in parallel. Generating the answer cannot be done that way. Each output token is conditioned on the token before it, so producing one requires a full pass through the model, and producing a thousand requires a thousand passes. An output token therefore occupies the accelerator for roughly a pass of its own, while an input token shares a pass with all of its neighbours. A platform selling capacity by the minute has to charge for occupancy rather than for count, and one weighting factor applied to the output side is the simplest instrument that does it.

Whether that factor is six, and whether one unit is 600,000 weighted tokens a minute, are figures from one platform's capacity documentation for one class of model, and both get revised. What does not change is the shape underneath them. A workload's unit consumption is a weighted sum in which the output side counts for several times its face value, so any estimate built from request rates alone is wrong by whatever the output mix happens to be, which is the reason to measure rather than to reason from a traffic graph.

That weighting is why average requests per second is useless as an input. Two workloads at the same request rate can differ by an order of magnitude in units consumed. A filings workload at a mean 26,400 input tokens and 900 output tokens per document is at the heavy end: 26,400 + 6 × 900 = 31,800 weighted tokens per request.

Read that sum out and the distortion is visible in one line. The 26,400 input tokens count as themselves. The 900 output tokens count as five thousand four hundred. So the output is about three per cent of the tokens anybody involved would say the request contained, and about seventeen per cent of the capacity the platform is selling. Two workloads with identical request rates and identical documents, differing only in how long their answers are, therefore land in different places on the purchase decision, and the size of the gap between them is set by the weighting rather than by anything either workload does differently.

Getting from usage records to units is a short script, and it has to run over the platform's own metered counts rather than a local tokenizer's estimate, for the reasons the first chapter set out.

from collections import defaultdict
from statistics import mean, quantiles

WEIGHT_OUTPUT = 6            # from the platform's capacity documentation
UNIT_WEIGHTED_TPM = 600_000  # one unit, same source

per_minute = defaultdict(int)
for r in invocation_log_records:      # 14 days, one row per request
    minute = r.ts.replace(second=0, microsecond=0)
    per_minute[minute] += r.input_tokens + WEIGHT_OUTPUT * r.output_tokens

values = [v for _, v in sorted(per_minute.items())]
q = quantiles(values, n=100)
print(f"mean  {mean(values)/UNIT_WEIGHTED_TPM:6.2%} of a unit")
print(f"p50   {q[49]/UNIT_WEIGHTED_TPM:6.2%}")
print(f"p99   {q[98]/UNIT_WEIGHTED_TPM:6.2%}")
print(f"max   {max(values)/UNIT_WEIGHTED_TPM:6.2%}")
print(f"minutes above one unit: {sum(1 for v in values if v > UNIT_WEIGHTED_TPM)}")

Fourteen days is the shortest window that captures a weekly cycle, and the last line is the one that decides the purchase. A mean of 15% and a max of 300% describe completely different capacity problems, and only the percentiles distinguish them. Per-minute buckets also hide sub-minute bursts, which matter because admission control operates on a shorter window, so it is worth re-running the same script at ten-second granularity to see how much higher the true peak is. The invocation log from the previous chapter is where those rows come from, which is one more reason to have switched it on before it was needed.

Platforms publish capacity calculators that take a request rate, an average input size and an average output size, and return a unit count. They are useful for a first estimate and systematically optimistic for one reason: averages. A calculator fed the mean of this workload returns 0.15 units and a recommendation to buy one, which is arithmetically correct and operationally wrong, because it has no field for the shape of the demand curve.

There is one more input a calculator cannot supply. Reserved capacity has its own admission behaviour when you exceed it, and the two possibilities — a reservation that throttles at its ceiling, or one that spills to the on-demand pool and bills for what it spills — produce very different bills and very different failures. Which one applies is a platform behaviour and sometimes a per-deployment setting, and it belongs in the arithmetic rather than in a discovery after the first peak.

The arithmetic on one unit#

A fourteen-day run on the filings workload produced a clean picture: a quiet overnight period, a flat working day, and a single sharp peak when a regulator's daily filing bundle lands and the workload processes it as fast as it can.

Every cell in that table is a multiplication, and the rows are worth checking rather than accepting, because a capacity argument nobody has done the arithmetic for is an argument about opinions. The overnight window is 360 minutes at 30,000 weighted tokens a minute, which is 10.8 million. The working day outside the peak is 1,060 minutes at 90,000, which is 95.4 million. The peak is twenty minutes at 1,800,000 offered — three times what one unit can serve — so the unit serves its own 600,000 a minute for those twenty minutes, which is 12 million, and the 24 million it could not take has to go somewhere else. The three window lengths add to 1,440 minutes, which is a day, and the served column adds to 118.2 million.

One unit supplies 600,000 weighted tokens a minute, so 864 million a day. This workload consumes 118.2 million of them, which is 13.7% utilisation: the unit sits idle for the equivalent of twenty and a half hours out of every twenty-four, and idle reserved capacity is billed at exactly the same rate as saturated reserved capacity.

Both of those are single divisions and both are worth seeing, because they are the two numbers the purchase turns on. 118.2 million against 864 million is 13.7%. The idle share is the remainder of that, and 86% of twenty-four hours is twenty and a half — not twenty and a half hours during which nothing arrives, but twenty and a half hours' worth of capacity that arrives, is paid for, and is not used. The invoice does not have a column for the difference.

The spill is the other half. During the twenty-minute peak the offered load is three times what the unit can serve, so 24 million weighted tokens a day have to go somewhere, and on most platforms that somewhere is the on-demand pool — with the on-demand quota from earlier in this topic still applying to it, unchanged. Reserved capacity does not raise those limits; it runs beside them.

The reservation line is $29,850 a month against a total on-demand bill of $12,437, which is where a proposal describing this as 2.4 times current spend gets its multiplier. Adding the $2,099 the spill still costs takes the committed option to $31,949, and the annual row is twelve months of each column with nothing else in it.

The peak is the only twenty minutes when p99 is at risk#

Tail latency is the usual justification, so test it against the shape above. For the 1,420 minutes a day when offered load is at or below 15% of a unit, tail latency is already fine — there is no queueing, no throttling, and p99 sits near p50 because the endpoint is not contended. Reserved capacity makes those minutes more predictable in the sense that a noisy neighbour on shared capacity can no longer affect them, which is worth something, and it is worth something during a period when nothing was going wrong.

The p99 that appears in an SLO report is made almost entirely of requests from the twenty-minute peak, because that is when queueing, throttling and retry delays happen. During the peak, one unit serves a third of the offered load and the other two thirds are throttled or served on-demand at whatever latency the shared pool is giving that afternoon. The commitment therefore buys predictability for the window where predictability was not the problem and leaves the window that generates the breaches essentially as it was.

The reasoning that leads there is sound at every step: latency is bad, reserved capacity improves latency, therefore buy reserved capacity. What the chain omits is that a capacity guarantee is a ceiling on what the reservation can absorb, and a reservation smaller than the peak does not cover the peak.

Break-even is a utilisation figure, and it is higher than people expect#

There is a load at which reserving is clearly right, and computing it gives the conversation a target rather than an opinion. The comparison needs one intermediate quantity that none of the tables states outright: what a weighted token costs on demand at this workload's particular input–output mix. It is recoverable from the tables themselves. Offered load is 10.8 plus 95.4 plus 36 million weighted tokens a day, the last of those being twenty minutes at 1,800,000 a minute, which is a little over four and a quarter billion a month — and the on-demand column bills $12,437 to serve it. That works out at a shade under three thousand dollars per billion weighted tokens. The same rate checks out against the other column, which is the point of doing it: 24 million spilled a day is seven hundred and twenty million a month, and that many weighted tokens at that rate is the $2,099 on the spill line.

A fully saturated unit delivers 25.92 billion weighted tokens a month, which is 864 million a day across a month. Bought as on-demand tokens at the rate just derived, those would cost about $75,560, against $29,850 for the reservation, so reserved capacity at full utilisation is roughly 60% cheaper per token. Dividing gives the break-even: $29,850 against $75,560 is a little under 40%, and that is the sustained utilisation at which the reservation stops being the more expensive way of buying the same tokens. Measured utilisation is 13.7%, about a third of it. The gap is not a rounding error, and no negotiation on the unit price closes it, because the unit price is already reflected in the 40%. Moving the break-even down as far as 13.7% would take a discount of roughly two thirds on the reservation, which is not the size of concession a procurement conversation produces.

That number also says what would change the answer. Sustained daytime load rising from 15% to 45% of a unit — plausible if two more filing types arrive — makes the reservation cheaper than on-demand for the steady traffic while leaving the peak exactly as unsolved as it is now. A peak of 0.8 units instead of 3.0, which is what spreading the bundle over three hours rather than twenty minutes would produce, converts the whole workload into something a single unit covers end to end, and that reshaping costs nothing but a change to a scheduler. If the filings have no latency requirement of their own, an asynchronous batch endpoint at roughly half the on-demand token price handles the peak better than any reservation would, and leaves the steady interactive traffic on-demand.

The last row is the one to re-derive before anybody quotes it. It is an estimate at roughly half the on-demand token price, and how far the bill actually falls depends on how much of the volume can move to an asynchronous path — the whole workload, or only the twenty-minute peak — and on whether the platform's batch discount applies to input tokens, output tokens or both. Those are scheduling and pricing questions with documented answers rather than things to model. The row is in the table to show the direction, which is that the cheapest option on it costs less than the status quo rather than more.

A one-year term binds you to one model version#

The second argument in that approval note was locking the price in, and a term does that in both directions. Model prices in this market have fallen sharply and repeatedly, and a capable successor at a third of the price within six to twelve months is the base case rather than an optimistic one. A one-year commitment signed today means that when such a model ships, the remaining term is a sunk obligation on the old one, and the workload either keeps running the more expensive model to use capacity already paid for or moves and pays twice.

How bad that is has a factual answer, and it should be asked before signing rather than after: is the commitment portable to successor models within the family, at what conversion ratio, and with what notice? Platforms differ. Some sell reservations against a capacity pool that any supported model can draw from, which makes migration a configuration change; others bind a reservation to one model version, which makes it a stranded asset the day that version is superseded. A one-year commitment on a portable pool is a genuinely different instrument from a one-year commitment on a specific version, and the words on the order form are where the difference lives.

Reserved capacity is worth buying when a workload is flat and heavy, and this one is neither. The general form of the test is three numbers rather than an opinion: sustained utilisation against a break-even you computed from the platform's own unit price, the peak as a multiple of the allocation, and the portability clause on the order form. All three come from measurements that take a fortnight to collect and an afternoon to run, and all three are cheaper than the alternative of finding out three quarters into a term.

What carries forward from this topic is a short list of things to have running before anyone asks. Measure throughput in the platform's own weighted unit and keep the series, because it is the input to every capacity conversation and it takes two weeks to collect. Keep a shadow comparison and a billed-token ratio alerted, because the platform ships changes you do not deploy. Keep invocation logging on with a retention period matched to the longest audit lookback in your agreements, and keep the request id on the business record. Keep jitter and a capped attempt count in the client, and a concurrency limit in front of every batch. None of these are large pieces of work, and each is the difference between answering a question in an afternoon and discovering that the evidence, the measurement or the headroom needed to exist three months ago.