Price per million tokens

Every hosted model is sold by the token, where a token is the sub-word unit the model reads and emits — roughly three-quarters of an English word on average — and the price is quoted per million of them. That unit is convenient and it is also the source of most cost surprises in this part of the stack, because a provider's pricing page does not publish one number per model. It publishes at least two, usually four or five, and the single figure people carry away from it is an average they have computed themselves without noticing. Quoting one number without the weights is like quoting an average shipping cost without saying whether the parcels are envelopes or pallets. The two teams in the takeaway differ by a factor of four for exactly that reason: one is summarising long documents into short answers, the other is expanding short prompts into long ones, and the token counts they report are the same while the mix behind them is inverted.

The two numbers that a single price averages away#

Output tokens cost more than input tokens across essentially every provider, typically by three to five times, and the reason is mechanical rather than commercial. Input tokens are processed in a single parallel forward pass over the whole prompt, which saturates the accelerator efficiently. Output tokens are generated one at a time, each requiring a full pass through the model, and each one occupies memory in the key-value cache for the remainder of the request. Generation is the expensive phase, so it is priced as the expensive phase.

Why one of those phases can be parallel and the other cannot is the step underneath the price difference, and it turns on a single property of the model. The computation at any position depends on the tokens at or before it and on nothing after it. For a prompt, every one of those tokens is already known the moment the request arrives, so the work for all of the positions can be issued at once and the accelerator has thousands of independent multiplications available to fill itself with. For generation, the token at each position is precisely the thing being computed, so the input to the next step does not exist until the current step has finished. The hardware is handed one position's worth of work at a time, which is nowhere near enough to keep it busy, and the model's entire weight set has to be read out of memory for each of those steps regardless of how little arithmetic is being done with it. Same model, same weights, and a cost per token that differs by roughly the ratio between a saturated accelerator and a nearly idle one.

The consequence is that the shape of a workload determines its cost more than the model's headline rate does. Two workloads on the same $3/$15 model:

Every cell in that table is one multiplication, and working them by hand once is what makes the rest of this chapter arguable rather than memorised. A price quoted per million tokens is a rate, so the cost of any other number of tokens is that number divided by a million, times the rate. The summariser sends 8,000 input tokens, which is eight thousandths of a million, and eight thousandths of three dollars is twenty-four thousandths of a dollar — the $0.0240 on the input row. Its 250 output tokens are a quarter of a thousandth of a million, and a quarter of a thousandth of fifteen dollars is a shade under four thousandths, which is the $0.0038 below it. Those two add to $0.0278 for one request, and fifty thousand requests come to $1,390 because the per-request figure is a constant and multiplying by the count is the only step left.

The blended row is the same division run backwards, and it is the one worth understanding rather than reading off. That $0.0278 bought eight thousand two hundred and fifty tokens in total. Expressing it per million means scaling the token count up to a million and scaling the cost by the same factor, which is legitimate because both sides of a rate scale together: a million divided by eight thousand two hundred and fifty is a little over a hundred and twenty-one, and $0.0278 multiplied by that is $3.37. The drafting assistant's request is one thousand two hundred and fifty tokens for $0.0146, and a million divided by that count is exactly eight hundred, so its blended rate is $0.0146 times eight hundred, which is $11.68. Nothing about the model differs between those two lines. What differs is that one workload's tokens are mostly the cheap kind and the other's are mostly the expensive kind, and a blended rate is nothing more than the average price of a token in one particular mix.

The same model, quoted honestly, is worth $3.37 per million tokens to one team and $11.68 to the other. Neither number is a property of the model. Either can be produced from your own logs before you sign anything, because the usage object on every completion response carries the split already:

{
  "id": "resp_01H9",
  "model": "some-model-2",
  "usage": {
    "input_tokens": 8214,
    "output_tokens": 247,
    "cache_creation_input_tokens": 7900,
    "cache_read_input_tokens": 0,
    "reasoning_output_tokens": 0
  }
}

Those last three fields are where the second layer of surprises lives. Prompt caching lets a provider keep the processed state of a prompt prefix and reuse it on later requests that begin with the same text, which saves the work of recomputing it, and cached input is billed on a different schedule from ordinary input — a premium to write an entry, commonly around 25% above the base input rate, and a steep discount to read one, commonly 90% below it. A workload with a long stable prefix and a short varying suffix can therefore cut its input bill by most of its value, while a workload that changes the first token of every prompt sees no benefit at all and pays the write premium repeatedly.

The word prefix is doing all the work in that sentence, and it is the word people read past. What the provider stores is not your text but the state the model computed while reading it, and that state has a strict dependency running one way: the entry for a position is a function of every token at or before it. The stored work for the opening stretch of a prompt is therefore reusable by any later request whose opening stretch matches to the character, and reusable by no request that differs at the very first token, because a difference there changes every state after it. A prompt assembled as a fixed instruction block followed by a variable ticket has a long reusable head. The same prompt with a timestamp, a request identifier or the customer's name interpolated at the top has none at all, and it is the arrangement of the template rather than its content that decides which of the two you are running.

The third of those fields answers a question about the same bill from the other end. Reasoning output tokens are billed as output even though the user never sees them, which for the reasoning variants covered in the earlier topic on model types can mean the visible answer is a small fraction of what you paid to generate. They are billed that way because they were generated that way: hidden or not, each was produced by one sequential pass through the model, which is the step the output rate is pricing. A field showing a large reasoning count beside a short answer is not an anomaly to be queried with the provider — it is the workload telling you what it actually costs.

Everything else the pricing page charges for#

A pricing table with two columns is already better than one, and it is still not a cost model. Four further mechanisms move real bills by more than the difference between most candidate models.

Batch processing offers a discount, typically 50%, for requests you submit as a file and collect within a window measured in hours. That is a genuine halving for anything asynchronous — nightly enrichment, backfills, evaluation runs, classification of a document backlog — and it is unavailable for anything a user is waiting on. A workload that is 70% asynchronous and 30% interactive should be modelled as two workloads, because treating it as one at the interactive rate overstates its cost by a third.

Context tiers change the unit price above a threshold. A model advertised at $3 per million input tokens may charge $6 above 200,000 tokens of context, applied to the entire request rather than to the excess.

"Applied to the entire request rather than to the excess" is the clause that does the damage, and it is worth spelling out because the natural reading is the other one. An income tax bracket charges the higher rate only on the portion above the line, so a small overshoot costs a small amount extra. This works the other way: crossing 200,000 tokens re-prices every token in that request, including all the ones below the threshold, at $6. A request a token under the line and a request a token over it therefore differ in price by close to a factor of two rather than by one token's worth of anything. A retrieval workload that usually sends 60,000 tokens and occasionally sends 210,000 does not pay slightly more on those requests; it pays double on them, which makes a retrieval budget cap a cost control and not only a latency control.

Rate limits determine whether you can buy the volume at all. Limits are published per minute in both requests and tokens, and the token limit is the one that binds on long-context work: an account allowed 400,000 tokens per minute can sustain about 50 requests per minute at 8,000 input tokens each, regardless of how many requests per minute the other limit permits. That figure is a single division — four hundred thousand tokens of budget divided by the eight thousand tokens each request spends is fifty requests — and the division is the right operation because the limit is a quantity consumed per token rather than per call, so requests draw from it in proportion to their length. Which of the two limits binds is therefore decided by prompt length rather than by traffic: a generous per-minute request allowance is easy to stay under on short prompts and irrelevant on long ones, because the token budget runs out first. Exceeding it returns 429s with a retry-after header, and your effective throughput becomes a queueing problem rather than a pricing one.

HTTP/1.1 429 Too Many Requests
retry-after: 26
x-ratelimit-limit-tokens: 400000
x-ratelimit-remaining-tokens: 0
x-ratelimit-reset-tokens: 26s

A twenty-six second wait is not a cost line, and it is the difference between a feature that works at the daily traffic peak and one that does not. Rate limits also tend to be tied to account spend history, which means the limit you are quoted during evaluation is often lower than the one you would eventually hold and higher than the one a new account gets — worth asking about explicitly, because a candidate that cannot be granted your peak throughput within the launch window is disqualified on availability rather than on quality.

Modality is priced on its own schedule when a candidate accepts more than text. Images are converted into a token count derived from their dimensions, so a 1,100 by 1,400 pixel page scan can consume well over a thousand input tokens on its own, and a document workload that sends four pages per request is paying several times what its text prompt suggests. Audio input and output are commonly billed per second or per audio token at rates that do not map onto the text rate at all, and a model listed as multimodal may support an input modality without supporting the corresponding output. A comparison table with a single modality column reading "text, image" is hiding a cost model; the useful cell records what each modality costs and at what resolution or sample rate.

Minimums and commitments appear at the enterprise end. Provisioned or reserved capacity is sold per unit-hour, so it is billed whether or not you send traffic, and it converts a variable cost into a fixed one in exchange for guaranteed throughput and immunity from shared-tier congestion. That is worth having when your traffic is steady and worth avoiding when it is spiky, which is the same arithmetic self-hosting demands and the subject of the next chapter.

Why the same open-weight model appears twice at double the price#

Inference marketplaces list open-weight models — the downloadable checkpoints of chapter one — served by multiple independent hosts, and it is routine to see one model with the same publisher, the same parameter count and the same name offered at $0.18 and $0.35 per million tokens on the same page. The weights are identical, so the difference is not the model. It is the serving configuration, and three variables account for most of the spread: quantisation, context length and serving tier.

Numerical precision is the first. A host serving an FP8 or 4-bit quantised build uses roughly half or a quarter of the memory per replica and generates tokens faster, so it can amortise a GPU across more concurrent requests and price lower.

Both halves of that follow from one fact about the hardware. Weights occupying half the memory leave the other half free for key-value cache, and cache capacity is what limits how many requests a card can hold at once, so the same accelerator carries more concurrent users. Generation speed on this hardware is set by how many bytes have to be read out of memory to produce a token rather than by how much arithmetic is done with them, so halving the bytes roughly halves the time per token. More users per card and more tokens per second per user multiply together, and the host divides one hourly rate by that larger product.

The quality cost of the lower precision is real, usually small on general text and larger on tasks with tight formatting or long-chain arithmetic, and it is rarely stated on the listing. The cheap listing and the expensive listing may not be the same model in any way that matters to your evaluation.

Context length is the second. Serving 128k tokens of context requires reserving key-value cache memory per request, which reduces how many requests fit on a card. A host offering the same weights capped at 16k context is selling a cheaper product, and the cap is frequently in a details panel rather than the price row.

Serving tier is the third: shared throughput with best-effort latency and no availability guarantee, against dedicated capacity with a published time-to-first-token target, which is a service-level objective the host can be held to. Batch scheduling policy and how aggressively the host packs concurrent requests onto a card shift both price and tail latency, meaning the slowest few per cent of requests rather than the typical one. So does speculative decoding, a technique in which a small fast model drafts several tokens ahead and the large model checks them in a single pass, accepting the ones it agrees with.

When two listings of one model differ by a factor of two, the useful move is to ask the host which precision they serve and at what context, and to record the answer in your comparison. If a host will not say, that is itself a data point: you cannot reproduce an evaluation against a build whose numerics are unspecified.

The columns that earn their place#

A candidate table built for a decision carries the assumptions, not just the headline. The columns that earn their place are input price, output price, cached-read price, batch availability and discount, the context threshold where pricing changes, the token and request rate limits at your account tier, precision and context for open-weight listings, and — as one derived cell — the cost of a thousand of your requests at your measured token split. That last column is the only one anyone should compare directly, and it can be computed as soon as you have a few hundred logged requests.

def cost_per_1k_requests(in_tok, out_tok, price_in, price_out,
                         cache_hit_rate=0.0, cache_read_price=None,
                         batch_discount=0.0):
    """Prices are USD per million tokens. Returns USD per 1,000 requests."""
    cached = in_tok * cache_hit_rate
    fresh = in_tok - cached
    read_price = cache_read_price if cache_read_price is not None else price_in
    per_request = (fresh * price_in + cached * read_price
                   + out_tok * price_out) / 1_000_000
    return per_request * 1_000 * (1 - batch_discount)

# ticket assistant, 350 in / 900 out, no caching, interactive
cost_per_1k_requests(350, 900, 3.00, 15.00)            # -> 14.55
# same traffic, 80% of the prompt cached, submitted as batch
cost_per_1k_requests(350, 900, 3.00, 15.00,
                     cache_hit_rate=0.8, cache_read_price=0.30,
                     batch_discount=0.5)               # -> 7.14

Read the first call before the second, because it is the table above in code. Three hundred and fifty input tokens at three dollars a million costs about a thousandth of a dollar. Nine hundred output tokens at fifteen dollars a million costs thirteen and a half thousandths. Their sum is a shade under fifteen thousandths of a dollar for one request, and a thousand requests at that price is the $14.55 printed beside the call. The second call leaves the model and the traffic alone and changes two settings: four fifths of the input moves out of the input column into the cached-read column, where the rate is a tenth of it, and the batch discount then halves whatever the total came to.

Which of those two did most of the work is worth noticing, because the intuition runs the other way. Prompt caching is the change that sounds structural, and on this workload it is close to cosmetic — input was a small minority of the bill to begin with, so removing most of a small share removes very little. The batch discount applies to everything, including the output tokens the bill is overwhelmingly made of, and halving the whole is worth more than saving nine tenths of a fraction. The transferable lesson is that a cost lever is worth exactly what it is a lever on, so the first thing to establish about any workload is which of its two token streams the money is sitting in.

Two configuration choices on an unchanged model halved the cost per request, which is a larger effect than switching to most cheaper competitors would have produced, and it is invisible to anyone comparing single prices.

Cost per token against cost per completed task#

The figure this exercise produces is still not cost per unit of work done. A cheaper model that needs two attempts to produce valid JSON, or that answers correctly in 1,400 output tokens where a pricier one uses 500, can be more expensive per completed task while winning every price comparison.

Retries belong in that figure too, and they are usually left out because they live in a client library rather than in the application code that logs cost. A request that streams half an answer and then fails on a connection reset has consumed the input tokens and the output tokens generated so far; a validation retry after malformed tool arguments pays for the whole first attempt again. On a workload where four per cent of calls are retried once, that is a four per cent surcharge nobody budgeted, and on a strict-schema extraction task where the retry rate reaches fifteen per cent it is the difference between two candidates.

Tokenizers differ between families as well, so the same 10,000-character document is not the same number of tokens across candidates — differences of 10-20% are ordinary, and larger for code or non-Latin scripts. The reason is that a token is not a unit of language but a unit chosen by whoever built that model's vocabulary, from the text they built it on. A vocabulary trained mostly on English prose spends one token on a common English word and several on a Turkish one, or on an identifier in camel case, or on a run of indentation. So the token count of a document is a property of the pair — this document, that model — rather than of the document, and the price you are comparing is a price per unit whose size changes with the candidate. A comparison run on token counts from one provider's tokenizer and applied to another's prices is wrong by that margin before anything else has been measured. Cost per successful task, measured on your own inputs, is the quantity that decides anything; cost per million tokens is an input to it.

A defensible cost column, then, is not a number copied off a pricing page. It is a derived figure computed from your own measured token split, adjusted for the batch and caching configuration you would actually run, checked against the rate limit your account tier would hold at peak, and stated per completed task rather than per token. Every one of those adjustments is available before you sign anything, and each of them moves the answer by more than the difference between most of the candidates on a shortlist.

All of that assumes there is a price to look up. For a model you host yourself there is no per-token figure anywhere, because you are renting hardware by the hour and the tokens are whatever you manage to extract from it. Turning that into a number comparable with the ones above is arithmetic you have to do yourself, and it produces a different kind of answer.