Per-case cost ceilings

A correctness suite is blind to price by construction. Every check written so far asks whether the output was right, and a change that makes the output right more expensively passes all of them, because it is a change in the input. Prompt tokens are the usual vehicle: retrieval that returns more context, a system prompt that grows a section, few-shot examples added to fix a hard case — worked pairs of a customer message and the answer it should get, included in the prompt so the model can imitate them.

This chapter puts a number on that and then decides who owns the number. The first half is mechanical: providers return a usage block with each response, a price table turns it into currency, and an assertion compares the result against a ceiling stored beside the case. The second half is where these checks actually fail in practice, and it is social rather than technical. A ceiling set to today's average plus headroom is tripped by the first genuine improvement, and a team under deadline pressure responds by raising the number — after which the assertion tracks whatever the prompt happens to cost and has stopped constraining anything. The chapter covers the file format, the approval path and the expiry that make that harder, the separate ceiling that output length deserves for reasons that have nothing to do with money, and the division of labour between this check and the billing alert that does see real money.

The numbers are in the response already#

Providers return a usage block with each response, and the run record of chapter 3 keeps it alongside the model that served the call. Reading it is the whole measurement:

PRICES_PER_MTOK = {                      # USD per million tokens
    "claude-opus-5":   {"in": 5.00, "out": 25.00, "cache_read": 0.50, "cache_write": 6.25},
    "claude-sonnet-4": {"in": 3.00, "out": 15.00, "cache_read": 0.30, "cache_write": 3.75},
}

def case_cost_usd(record: RunRecord) -> float:
    total = 0.0
    for call in record.llm_calls:
        p = PRICES_PER_MTOK[call.model]
        total += (call.input_tokens            * p["in"]
                + call.output_tokens           * p["out"]
                + call.cache_read_input_tokens * p["cache_read"]
                + call.cache_write_input_tokens* p["cache_write"]) / 1_000_000
    return total

Three things in that function are the reason to compute cost rather than assert on token counts alone. It sums over llm_calls, because an agent case is not one request — a run that looks up an order, reads a policy document and then answers makes three or four calls, each of which resends the system prompt, so a change to that prompt is multiplied by the loop length before it reaches the bill. It reads call.model from the record rather than from config, so a case served by a fallback is priced as what actually ran. And it prices cached input separately, at roughly a tenth of the standard rate, because providers offer prompt caching: the processed form of a request's unchanging prefix is stored for a few minutes and re-used on later requests that start with the same bytes, billed at the reduced cache-read rate.

Where those numbers come from settles what can be measured. When they arrive settles what can be enforced, and the two questions are easy to run together. A usage block is attached to a response, so the cost of a call is known the instant that call returns and not a moment before it. After the first call of a case the harness holds the exact cost of the first call; after the second it holds the exact cost of the first two. The running total is therefore available inside the loop, at every point where the decision to issue another request has not yet been taken.

That availability is what makes a ceiling enforceable rather than merely reportable, and the difference is not one of strictness. A check that runs after the case has finished is a report: every token has already been bought, and the only thing the check can still influence is whether the change merges. A check that runs between calls is a control: it can decline to issue the next request, which is the one moment at which the spending can still be prevented rather than described. An eval suite runs the first kind, because a case has finished by the time its assertions run, and that is the right kind for this job — the eval exists to stop a diff, not to stop a request. The same arithmetic wired into the agent's own loop in production is the second kind, and it is what stops one pathological conversation spending a day's budget inside a retry storm.

A control that trips mid-case then has to decide what happens next, and both answers cost something. Abandoning the case loses its correctness verdict, so a budget breach arrives disguised as a correctness failure and the next person debugs the wrong thing. Recording the breach and continuing keeps the verdict at the price of spending the money the control existed to save. An eval suite avoids the choice entirely by measuring after the fact, which is another way of saying its ceiling is a gate on a merge and never a cap on a bill.

One asymmetry inside the loop decides how far ahead either kind can see. Input tokens are countable before a request is sent, because the prompt is assembled by your own code and can be measured before it leaves. Output tokens are not: how long an answer runs is settled as it is generated, and the only prospective control over it is the cap on completion length that the request itself carries. So a prompt-token ceiling can be checked before anything is spent, a completion-token ceiling has to be imposed rather than checked, and a cost ceiling — which mixes the two — can only be estimated in advance and confirmed afterwards. That is why the budget file below carries a separate line for prompt tokens and another for cost, instead of one number standing for both.

The price table belongs in the repository, versioned, with the date it was checked. Prices change, and a cost assertion computed from numbers somebody typed from memory eighteen months ago fails in the least useful way available: quietly, in the wrong direction.

Unit cost, and the bill it does not predict#

Work the arithmetic for a prompt change that takes average prompt tokens from 1,200 to 3,400 across a three-call agent loop, and the per-case number comes out like this:

Every figure in that table comes from the same two operations, and the last row is worth trusting only once the first has been walked down. Start with input cost before the change. The prompt is 1,200 tokens and the case makes three calls, and the whole prompt is charged on every one of them: the API keeps no state between requests, so an agent loop re-sends the entire conversation each turn and the system prompt is paid for once per call rather than once per case. That is three times 1,200, or three thousand six hundred prompt tokens for the case. The price table quotes $3.00 per million input tokens, and a price quoted per million becomes a price per token by dividing by a million, so those tokens cost three thousand six hundred millionths of three dollars, which is $0.0108. The same arithmetic on 3,400 tokens a call gives ten thousand two hundred tokens and $0.0306.

The +183% appears twice, on the token row and on the cost row, and the reason it is the same number twice is worth stating rather than checking. The token increase is the difference between 3,400 and 1,200 taken as a proportion of the 1,200 it started from, which is the 183% in the table. Cost is that token count multiplied by two constants — the number of calls, which did not move, and the price per token, which did not move — and a constant common to both sides of a ratio cancels, so the percentage survives the conversion into dollars unchanged. Had the change also added a call to the loop, the constants would not have cancelled and the two rows would differ.

Output is charged at $15.00 per million, and the table's completion figure is per case rather than per call, so 350 tokens cost 350 millionths of fifteen dollars, a shade over half a cent, which the table shows as $0.0053 at the four decimal places it keeps. Ten more tokens after the change take it to $0.0054. Adding the input and output terms gives $0.0161 before and $0.0360 after.

The headline +124% is the difference between those two totals divided by the first, and what to notice is that it sits well below the +183% on the input row. It has to. The total is a mixture of two terms that moved by very different amounts, and a mixture rises by the average of its parts' increases weighted by their shares of the starting figure. About two thirds of the $0.0161 was input and about one third was output, so a large increase on two thirds of the base and a negligible one on the remaining third must land between the two and nearer the larger. Stated as a multiple instead of a percentage, $0.0360 divided by $0.0161 is a little over 2.2 — and a rise of 124% and a multiple of a little over 2.2 are the same fact written two ways, since the multiple is the percentage increase plus the original whole.

The suite row is one multiplication. Three hundred cases each priced at the same average makes the run cost 300 times $0.0161, or $4.83, and 300 times $0.0360, or $10.80. The $5.97 gap can be read either as the difference of those two totals or as 300 times the per-case difference of just under two cents; the two agree because multiplication distributes over subtraction, and their agreement is a free check on the whole column. What the row assumes is that the average holds across the basket, and an average is compatible with one case in the suite costing several times the mean while the mean stays comfortably under the ceiling. That is the argument for asserting per case as well as per run, which a later paragraph returns to.

A 2.2x unit cost is enough to justify failing a pull request. It is also not the same number the invoice will show, and the gap runs in a direction worth understanding before anyone quotes the suite's figure at finance. Production conversations run longer than eval cases — a median support thread makes five or six model calls rather than three — so a fixed prompt delta is multiplied by a larger number in production than in the suite. And appending to a system prompt invalidates the cached prefix: where roughly 70% of production input tokens were previously billed at the cache-read rate, every request pays full price until the new prefix warms, on a cache with a five-minute TTL — the time an unused entry survives before it is evicted — that a low-traffic hour never reaches. An eval suite, which runs each case cold, never sees the caching regime at all and so under-reports the real delta.

Prompt caching matches on an exact run of bytes from the start of the request, and that single property decides the whole outcome. The provider keeps the processed form of a prefix and may re-use it only for a later request that begins with the same bytes; the first byte that differs ends the match, and everything from that point on is processed again at the full rate. A system prompt sits at the very front of a request, so text appended to it falls inside the region every later request depends on, and the stored entry is invalidated for all of them rather than only for the requests that have anything to do with the new text. The same property explains the repair in the scenario below: moving the examples ahead of anything that varies per request restores a long identical opening, and the length of that identical opening is what decides how much of the prompt can be served from cache.

The size of the effect is worth doing once. With cache reads priced at roughly a tenth of standard input, a request in which seven tenths of the input is cached pays a tenth of the usual rate on those seven tenths and the full rate on the remaining three, which comes to a little over a third of what the same tokens would have cost uncached. Losing the cache therefore raises the price of tokens the request was already sending, before a single new token has been added — and the change added tokens too. Those two effects multiply rather than add, which is why a suite that never experiences the caching regime reports a floor on the change rather than an estimate of it.

That is the honest limitation of a cost assertion measured in a lab, and it is worth stating on the pull request rather than discovering later: the suite measures uncached unit cost, which is an upper bound per call and an unreliable predictor of the bill. What it is very good at is detecting direction and magnitude of a change, immediately, attributed to a diff.

A budget is a number with an owner#

The obvious threshold is the current average plus some headroom, and it is the version that fails. A ceiling anchored to today's average is tripped by the first genuine improvement, and the cheapest way past an obstacle nobody owns is to edit it. A budget that survives is derived from something outside the prompt. If a support conversation must cost under $0.05 in model spend to hold the margin the product was priced at, and the median conversation is five calls, then the per-call budget is $0.01 and the per-case eval ceiling follows from that with stated headroom. The number now has a reason attached, so raising it is a claim about the product's economics that somebody has to make and defend, rather than a build-fixing edit.

# evals/budgets.yaml — owned by @platform-costs (CODEOWNERS)
defaults:
  max_prompt_tokens_per_call: 2000
  max_cost_per_case_usd: 0.020
  rationale: >
    Target model spend per support conversation is $0.05 at a median of
    5 calls. 2000 prompt tokens leaves ~35% headroom for retrieval.
  review_by: 2026-11-01

overrides:
  - case_id: refund_multi_item_dispute
    max_cost_per_case_usd: 0.045
    rationale: "6-item order requires full policy doc in context."
    approved_by: "@dhernandez, 2026-06-14"
    review_by: 2026-09-14

Three fields do the governance work. rationale forces the number to be justified in words, which is what makes a future reader able to tell a considered ceiling from a copied one. approved_by records that a person agreed, and CODEOWNERS — the file in which a repository assigns named owners to a path and requires their review on it — makes that approval mandatory for any change to the budgets, so raising a budget is a review conversation rather than a one-line commit inside an unrelated pull request. review_by gives overrides an expiry, which is the only mechanism that reliably removes them; an override with no expiry is permanent by default.

The division inside that rationale is worth doing slowly, because of the step it does not take. Spreading $0.05 across five calls gives $0.01 a call, and both quantities came from outside the prompt: the margin the product was priced at, and the length of a median conversation. Turning a per-call budget into a per-case eval ceiling needs one further step that no file can take for you, since an eval case in the table above makes three calls rather than five. A ceiling that does not say which loop length it assumes is comparing two different quantities, and is right only by accident. A reviewer's job on a budget change is to make somebody state that assumption out loud — how many calls the number is priced for, and whether the cases it governs run to that length — and to ask it again the next time the agent grows a step.

The suite then reports two different things, and confusing them is why cost checks get switched off:

Drift is what makes the check useful on changes that are nowhere near the ceiling. A change from 600 to 1,400 prompt tokens is a 133% increase that stays comfortably inside a 2,000-token budget, and a comment on the pull request saying so is how a team notices a trend before it becomes an incident. Report the delta on every run; fail only on the ceiling.

The two signals answer different questions, and neither answer contains the other. A ceiling asks whether the current number is affordable and has no memory of what the number used to be. A drift comparison asks whether this diff moved the number and holds no opinion about whether the result is affordable. That 133% — the step from 600 to 1,400 measured against the 600 it started from — is a large movement and comfortably inside the 2,000-token budget at the same time: guilty on one signal, innocent on the other. A single number cannot report both, which is why the column that blocks a merge and the column that annotates a diff are separate columns rather than two thresholds on one measurement.

Output tokens deserve their own ceiling for a reason that has nothing to do with money. Completion length is the term that dominates latency, because generation is sequential and input is processed in parallel, so a change that adds 400 tokens to the average answer adds real seconds to the user's wait while adding cents to the bill. A separate max_completion_tokens_per_call therefore acts as a proxy latency guard that is stable under any concurrency you happen to run the suite at, which is a property the next chapter spends its whole length establishing that wall-clock does not have.

The last structural choice is per-case versus per-suite. A ceiling on each case catches the one prompt that grew; a ceiling on the total cost of a suite run catches the change that added a retrieval step firing on every case, each within budget individually. Both are one line, they fail on different regressions, and the aggregate one has the useful side effect of telling you what your own eval suite costs to run — a number a surprising number of teams discover only when someone asks why the CI account's spend doubled.

Where the billing alert is right#

The argument that cost belongs in the provider's dashboard rather than in the eval suite is half correct, and the half it gets right is important enough that you want both. Billing sees actual money: real traffic mix, retries and their duplicated generations, background jobs, the tokens a fallback model spent during an incident, the cache hit rate at your real request cadence, and every other team on the same account. An eval suite sees a fixed basket of 300 cases under lab conditions and cannot know how many conversations you will serve next month, which is the term that dominates the invoice.

The two are looking at different factors of the same product. A month's model spend is, near enough, the number of units of work multiplied by the cost of a unit of work. An eval suite measures the second factor under fixed conditions and has no way to observe the first. Billing observes the product after the fact and has no way to separate them, which is why a unit cost that doubles while volume halves arrives as a flat month on the graph. A flat spend graph is the least informative artefact in this chapter, and it is the one most teams are relying on.

What billing cannot do is attribute or prevent. A spend graph tells you the number moved, not which of the forty merges that week moved it, and the detection lag runs from a day to a full billing cycle. By the time it fires, the change is in production, has been built on, and is expensive to revert. The eval assertion answers a narrower question — did this diff make each unit of work more expensive, and by how much — before the merge, in the review where the trade-off can still be made cheaply. Keep the budget check as a gate on unit cost and the billing alert as a monitor on total spend, and treat a surprise in the second that the first did not predict as evidence your basket no longer resembles production.

A cost ceiling is the cheapest assertion in this topic to compute and the most expensive to govern. The computation is a dictionary lookup and a sum over figures the provider already returned; nothing has to be timed, sampled or repeated. What takes the work is the number on the other side of the comparison, and the difference between a ceiling that constrains anything and one that records the status quo is entirely in where it came from, who agreed to it, and whether the override that got somebody's build green in June expires before anyone forgets why it was granted.

Cost has one property that makes all of this tractable: the figures are reported by the provider, do not depend on how you ran the suite, and come out the same whether you executed 300 cases one at a time or fifty at once. Latency has none of those properties, which is why the obvious latency assertion fails on Tuesday afternoons.