The per-request cost distribution

Denial-of-wallet is the failure in which a system puts no ceiling on what a single request, task or account is allowed to spend, so ordinary-looking traffic — or one bug, or one crafted document — becomes an arbitrarily large bill while the service itself stays healthy. The name echoes denial-of-service deliberately. A denial-of-service attack takes the product down, so everyone finds out within minutes, whereas denial-of-wallet leaves every dashboard green and produces its first symptom on an invoice that arrives weeks after the event. It became a category worth naming when serving moved from capacity bought in advance to model APIs billed per token, because the cost of handling one request stopped being fixed at deploy time and started being decided at runtime, by how much text the request drags through the model and how many times. You meet it wherever a customer's input can cause your code to call a paid model on their behalf: a chat assistant, an agent — a model called in a loop, choosing tools and reading their results until it decides it is done — that answers questions over uploaded documents, or a nightly pipeline over a customer's mailbox. The work the term names is putting ceilings in place, per request, per task and per account, and being able to say what happens to someone who reaches one.

This chapter is about the measurement that has to come first, which is the distribution of what a single request costs. A rate limit — a ceiling on how many requests one caller may send in a window, enforced at the edge and answered with a 429 too many requests — is the control most teams already have, and it is the one most often mistaken for a spend limit. The mistake is not carelessness. A limit is a ceiling, a ceiling times a unit cost gives an exposure, and the arithmetic is one line. It goes wrong at the phrase the cost of a request, which the calculation treats as a constant and which is in fact the only unbounded term in the system.

What a day of requests actually cost#

The way to find out is to take a day of usage records and price every one of them, using the token counts the provider returns on each response. Over 2.1 million requests from 18,400 active accounts on one platform, the per-request cost came back like this, written in percentiles, where p99 names the figure that ninety-nine percent of requests came in under: median €0.0018, p90 €0.011, p99 €0.19, p99.9 €1.42, maximum €14.80. Total platform spend for that day was €9,412.

Looking at the whole distribution rather than its endpoints is more informative than either, because the bands correspond to recognisable product behaviour:

Two columns of that table close exactly, and it is worth checking rather than taking on trust, because the checking is the argument. Add the share-of-requests column down all five rows and it comes to a hundred percent, which it must, since the bands are the whole distribution cut at percentile boundaries with no gaps and no overlaps. Add the share-of-spend column and it also comes to a hundred percent, which is the stronger statement: every euro of the day is attributed to exactly one band, so there is no residual sitting outside the table and no double-counting inside it. Once both columns close, the headline figure is not a separate measurement you have to believe. The top one percent of requests is the last two rows — nine tenths of a percent plus one tenth — and their spend shares are the last two entries of the other column. Add those two and you have sixty-two percent. The concentration claim is arithmetic performed on a table printed above it.

Reading the same table as gaps rather than as levels says why that concentration happens, which the totals alone do not. Walk down the cost column and each band's prices are around an order of magnitude above the band before it. Walk down the request column, from the third band onward, and each band holds roughly a tenth of the requests of the band before it. Those two movements are in opposite directions at close to the same rate, and a band's share of spend is its request count multiplied by what its requests cost. When one factor falls by about the same proportion the other rises, the product barely moves — which is what the spend column does across the top three bands, none of them far from a quarter or a third of the day. That is the practical meaning of a heavy tail: not that the last band is large, but that no band is negligible, so there is no point in the distribution below which the money stops mattering.

Sixty-two percent of the spend sits in the top one percent of requests. That concentration is the practical reason a request counter is the wrong instrument: it weights every request identically while the money is almost entirely in a thin tail, so the metric you are limiting and the metric you are paying for barely correlate. Once you have looked at the distribution, quoting the product of a bounded count and an unbounded unit is a way of publishing a number rather than a way of knowing one.

Where the eight-thousand-fold spread comes from#

The gap between €0.0018 and €14.80 is a factor of about 8,200, and it is not noise. It is the product of four terms that each vary independently, three of which are influenced by whoever is sending the request.

The row that surprises people is the input tokens. Eight and a half million input tokens on a request whose user uploaded a single 240-page contract does not mean the contract was eight million tokens; it means the conversation ran 34 model calls, and every one of those calls resent the whole accumulated context. A 250,000-token context resent 34 times bills as 8.5 million input tokens even though the user supplied the text once. The provider charges per token processed, and each call reprocesses everything.

That arithmetic runs backwards as well, and running it backwards is the check to perform on any expensive request in your own logs. Divide the billed input by the number of model calls and you recover the average size of the context each call carried. Do it on this row and the answer lands on about a quarter of a million tokens per call — the same figure the paragraph above names for the context. Two statements about this request, taken from different columns of the table, arrive at the same number, which is what tells you the request is a resend problem rather than an unusually large upload. When that division instead returns something near your typical prompt size, the expense is coming from the number of calls; when it returns something enormous, the context itself is the problem and the loop is incidental. One division separates the two causes, and they have different fixes.

Model tier contributes a factor of roughly 19 on the input side between a small model and a frontier one — frontier being the trade's word for the largest and most capable model a provider currently sells, priced accordingly — and if your product routes by task complexity, or lets the user pick, or escalates to the big model when the small one returns low confidence, then the caller has partial control over that factor too. Tool calls add charges that do not appear in the token bill at all: a web search at €0.005 per query, a sandboxed code run at €0.0004 per CPU-second, a document reindex that quietly re-embeds 40,000 chunks. Those land on different invoices from different vendors and are invisible to anyone looking only at the model provider's dashboard.

Written as a formula, the cost of one request is roughly

PRICES = {  # euros per million tokens
    "small":    {"in": 0.80, "cache_read": 0.08, "out": 4.00},
    "frontier": {"in": 15.00, "cache_read": 1.50, "out": 75.00},
}

def request_cost_eur(calls, tool_charges_eur=0.0):
    """calls: list of usage records returned by the provider, one per model call."""
    total = 0.0
    for c in calls:
        p = PRICES[c["tier"]]
        total += (
            c["input_tokens"] * p["in"]
            + c["cache_read_input_tokens"] * p["cache_read"]
            + c["output_tokens"] * p["out"]
        ) / 1_000_000
    return total + tool_charges_eur

Three things in that function are worth reading rather than skimming. It loops over calls, one iteration per model call, because a request is not a call and the fan-out is the outer factor in everything that follows. It bills cached input on its own line at its own rate, because the provider charges a reduced price for a prefix it has already processed, and a workload that reuses the same prefix prices differently from one that does not even when the token counts are identical. And it divides by a million at the end of each term because price sheets are quoted per million tokens, which is the unit conversion people drop when they estimate this on a whiteboard and the reason those estimates come out a factor of a million wrong in either direction. The tool charges are added outside the loop and outside the token arithmetic entirely, since nothing about them is priced per token.

Nothing in that function is bounded by a request counter. The rate limiter sees one increment whether calls has one element or thirty-four, whether each element carries four thousand input tokens or four hundred thousand, and whether the tier is the cheap model or the expensive one. It counts the outer loop and is blind to everything inside it.

The two terms that dominate#

If you have to name the inputs that separate a cheap request from an expensive one, name context size and fan-out, in that order, because they multiply each other. Fan-out here is the number of separate model calls one incoming request ends up making, which is one in a plain question-and-answer product and thirty-four in the request above.

Context size dominates because of the resend. A conversation that grows by 6,000 tokens per turn does not cost 6,000 tokens per turn; by turn twenty the context is 120,000 tokens and that turn alone bills 120,000 input tokens. Fan-out dominates because it is the multiplier on that growing number, and because in an agent it is decided at runtime by the model rather than at design time by you. A loop that is usually three iterations is not a loop bounded at three.

The two terms interact in a way that makes averages misleading. If your agent averages four iterations and your average context is 20,000 tokens, the average request bills around 80,000 input tokens, and it is tempting to reason about capacity from that product. But iterations and context size are positively correlated in every agent workload I have measured, because the runs that need many steps are the runs that accumulated a lot of retrieved material, and each extra step both adds to the context and pays for all of it. A run at twice the average iteration count is not twice the average cost; it is closer to four times, and the tail of the joint distribution is where the €14.80 request comes from.

Model tier is third and is usually a decision your routing layer makes, so you control it. Output length is fourth, and it is the one term people reach for first because max_tokens is the parameter sitting right there in the request. Output is genuinely expensive per token — five times the input price on most price sheets — but a completion cannot exceed the model's output limit, which is a few thousand to a few tens of thousands of tokens. The input side has no equivalent structural ceiling once a loop is resending a growing context.

Prompt caching changes the arithmetic without changing the shape. Providers offer it because the front of a prompt is usually identical from one call to the next — the system instructions, the tool definitions, the document — so they will keep the processed form of that prefix for a few minutes and charge a reduced rate when the next call presents the same bytes. When a stable prefix is cached, the reread bills at roughly a tenth of the normal input price, which turns that 8.5-million-token request into something closer to €2 if the cache hits on every call. It is the single highest-leverage cost lever available, and it is also why cost distributions get wider rather than narrower after you adopt it: cache-friendly workloads get ten times cheaper while cache-hostile ones — where each turn mutates the front of the context — do not move at all.

What a request-per-minute limit is genuinely for#

None of this makes a 60-per-minute limit pointless. It is doing real work on a different axis. It bounds concurrency, which bounds how much of your own serving capacity one account can occupy, how many connections it holds open, how deep your queues get and whether other users see latency — the subject of topic 2.12, on concurrency and throughput. OWASP, the Open Worldwide Application Security Project whose top-ten risk lists most security teams work from, keeps one for large language models, and its tenth entry — LLM10, on unbounded consumption — is careful about this split, separating resource exhaustion — saturating compute, memory and throughput — from cost exhaustion, the denial-of-wallet case where the system stays perfectly healthy and the invoice does not. A request-per-minute ceiling is a good control for the first and close to useless for the second, because the expensive requests are slow, so an attacker running the €14.80 pattern needs nowhere near 60 per minute. Six a minute would do it, and would generate €5,300 an hour without ever touching the limiter.

That figure is one multiplication and it is worth doing rather than accepting, so the shape is visible. Six requests a minute over sixty minutes is three hundred and sixty requests in the hour. Each of them, priced at the most expensive request the day actually produced, gives the hourly total. Nothing in that calculation touches the limit, because three hundred and sixty an hour is six a minute and the ceiling is sixty a minute — the traffic sits at a tenth of what the limiter would refuse. This is the shape to hold on to: the constraint and the cost are measured in different units, so satisfying the constraint tells you nothing about the cost, and an attacker optimising for cost will run comfortably inside the constraint rather than against it.

As an analogy, and it is only an analogy, a limit on how many parcels you may post in a day bounds the count and not the postage, because postage is charged by weight and the parcel count says nothing about what is in them. It is a useful picture for one reason: it makes obvious that two ceilings in different units do not substitute for one another. It breaks at the point that matters most here. A postal service refuses a parcel above some weight, so there is a per-parcel ceiling in the units that are billed, and the product of the two limits really is a bound. A request has no such ceiling by default. Its billed size is decided at runtime by how much text the loop drags through the model and how many times, and nothing in the request path refuses one for being too expensive unless you put something there. The analogy holds for the mismatch of units and fails at exactly the property this chapter says you have to install.

Tokens per minute are closer to money and are not money#

Token-per-minute limits, which most providers enforce on their side and some teams mirror on theirs, are a step nearer the bill because they meter the thing you are actually charged for. They are still not euros. The same million tokens costs €0.80 on a small model and €15 on a frontier one, so a workload that routes to the expensive tier spends nineteen times as much under an identical token ceiling. A token limit also does not see tool charges at all, which means an agent that fans out into paid search calls and sandboxed code runs can sit comfortably under a generous TPM ceiling while running up a bill from a vendor the token counter has never heard of.

The same gap runs through cost attribution. The per-request, per-user and per-feature attribution described in topic 2.16, on tracking cost per request, is what makes a distribution like the one above available in the first place, and it is worth checking that it covers tool vendors and not only the model provider, because the tool charges are the part that goes missing. An attribution layer that reconciles to the model provider's invoice and ignores the search vendor's will report a distribution whose tail is too thin, and the tail is the entire subject.

The tail is what legitimate traffic looks like#

One detail about that €14.80 request matters more than any of the controls described later in this topic. Nobody was attacking. It was a real customer, a real 240-page contract, and an agent that legitimately needed many passes to answer the question asked. The top percentile of this distribution is not a population of abusers; it is deep agent runs over large uploads, which is the feature the product was sold on. Any ceiling placed on the per-request cost is therefore a ceiling that will, sooner or later, land on somebody doing exactly what they pay for, and a control designed as though the tail were hostile will be turned off in its first week. Knowing the distribution also gives no bound on tomorrow's, because the terms that produce the tail are decided at runtime by an input you do not control: it is the evidence for a ceiling, not a substitute for one.

It is worth being precise about where the €1,278,720 went wrong, because the usual diagnosis is that the number is too big and the real defect is more specific. The multiplication itself is sound: a per-user daily ceiling on requests times a per-request cost is a per-user daily ceiling on spend, and that is the right operation. The defect is in the second factor. A cost per request substituted into that product has to be a figure that holds for every request being counted, and the maximum holds for one of them. Compare the result with what the platform actually did that day — one account's supposed ceiling against the whole platform's real spend across all of its active accounts — and the ratio runs to well over a hundred, which is the arithmetic saying that the substituted constant does not describe the traffic. The mean cost of a request that day, total spend divided by request count, is under half a cent; the figure used was the largest single draw. Substituting a maximum where the product needs a typical value inflates by whatever the spread happens to be, and the spread here is the factor of 8,200 the chapter opened on. That reframing is what makes the finance question answerable at all: the problem was never the multiplication, it was that the system offered no value for the second factor that was true of more than one request.

A rate limit is a good control that answers a different question from the one finance asked. It bounds a count in a window, which bounds concurrency and protects the service, and it is blind to the four terms that decide what a request costs: how many model calls the request makes, how large the context is by the time each of them is issued, which tier serves them, and what the tools charge on invoices the token counter never sees. Between the cheapest and the most expensive request on one ordinary day, those terms produced a factor of 8,200.

The consequence is that a maximum-exposure figure cannot be extracted from a system that has no ceilings, however much usage data it has recorded. It has to be chosen, by placing ceilings per request, per task and per account and then quoting them, together with what happens to somebody who reaches one. Until that exists, the honest answer to "what is the worst case" is a distribution and an admission, which is a better thing to send than €1,278,720.