The mean cost per task is where the expensive runs hide

The release report had one cost line on it. Mean cost per task moved from $0.132 to $0.154, a rise of about seventeen percent, which sat inside what the team had come to treat as normal variation between runs of the set and got no comment. Three weeks later, a customer with unusually busy calendars generated an invoice line that made somebody go back and pull the per-task figures from that release, and the distribution behind those two means turned out to be a different story: the median had actually fallen slightly, from $0.094 to $0.089, and three tasks out of three hundred had gone from around $0.11 to $6.60, $4.20 and $3.85.

The arithmetic is unremarkable once you see it. Three tasks adding $14.43 between them, spread across three hundred, contribute $0.048 to the mean. That is comfortably inside release noise, so the summary statistic behaved exactly as designed and told nobody that a task type had become fifty times more expensive. The step-level cost attribution from topic 10.8, which puts a price on every span of a run, already gives every span a figure that reconciles against the provider invoice. What this chapter does with it is aggregate to the task and then refuse to collapse the result to one number.

The quadratic in the step count#

Cost per agent task is heavily right-skewed, meaning most runs cluster near the median while a thin tail of very expensive ones stretches out to the right and drags the mean up with it, and the skew has a mechanical cause. The message list — the running transcript of system prompt, user request, assistant messages and tool results handed to the model — grows on every iteration, and each model call re-sends the whole of it, because the model holds no state between calls and the transcript is the only thing carrying what happened earlier. Input tokens therefore accumulate quadratically in the number of steps.

Take a run that starts with a 2,000-token prompt and adds roughly 3,000 tokens per step from tool results and assistant messages. Number the model calls from the first to the last. The first sends the base prompt alone, the second sends the base plus one step's worth of additions, the third the base plus two steps' worth. In general the input billed on a call is the base prompt plus one increment for every step already taken, which is one fewer increment than the call's own position.

Adding those up across a run separates into two pieces, and it separates because addition can be regrouped freely. The base prompt appears once on every call, so it contributes the step count multiplied by the base. The increments contribute one lot on the second call, two on the third, and so on, so summing them means adding the whole numbers from zero up to one less than the step count and multiplying by the increment. The sum of the whole numbers below any count is that count multiplied by one less than itself, halved. The general form is the number of steps times the base prompt, plus the per-step increment times the sum of the whole numbers below the step count.

Evaluate that at the two step counts worth comparing. Six steps: six lots of 2,000 tokens is twelve thousand, and the whole numbers below six sum to fifteen, so fifteen lots of 3,000 tokens is forty-five thousand, giving 57,000 input tokens for the run. Forty-one steps: forty-one lots of 2,000 is eighty-two thousand, and the whole numbers below forty-one sum to eight hundred and twenty, so that many lots of 3,000 tokens is two million four hundred and sixty thousand, giving a little over 2.5 million. Both reproduce what those runs actually billed, which is the check that this is the right model of the cost.

Pricing is one more multiplication, because a token price is a rate — dollars per million tokens, applied to however many millions you sent. At $3.00 per million input tokens, fifty-seven thousand tokens is fifty-seven thousandths of a million, and three times that is about $0.17. Two and a half million tokens is two and a half times a million, and three times that is a little over seven and a half dollars — $7.63 on the unrounded count. A policy difference that looks like "it tried a bit harder" is a factor of about seven in steps and a factor of about forty-five in dollars.

Which piece dominates is the part to carry away. The base-prompt piece grows in step with the step count; the accumulated-history piece grows with the square of it, since it is an increment multiplied by a sum that has the step count in it twice. At forty-one steps the history piece is around thirty times the base-prompt piece, which is why a long system prompt is almost never where a cost investigation should start.

That mechanism also settles what the tail is made of. Every call in the run is billed at the same rate per token, and nothing about being the fortieth call rather than the first changes that rate. So the tail is not made of expensive calls. It is made of ordinary calls, many of them, each one carrying a longer history than the last, which is why sorting model calls by unit price finds nothing — the sort key is constant across the set — and why the mean is a poor detector: a distribution whose tail is generated by a multiplicative process will always have a mean that is dominated by a handful of rows and moves smoothly when those rows change. The sorts that do find the tail are by call count and by input tokens per call, both properties of the loop rather than of any single call.

The shape of the distribution is the finding#

Read that table across rather than down. No single level in it carries the finding, because there is no absolute cost per task that is right or wrong for a scheduling agent. What carries the finding is the gap between how far the p95 row moved and how far the median row moved, together with the direction: the median went down between the two releases while p95 went up by nearly a factor of four. Two statistics computed over the same three hundred runs moving in opposite directions says the runs stopped resembling each other, and no number taken from either column on its own says that.

The p95 row is where a person looking at this report stops. Its jump from $0.38 to $1.42 says that fifteen tasks in the set now cost more than a dollar where previously the same rank sat at about a third of one, and it says it in a number that cannot be produced by a single outlier, which is what makes it hard to dismiss. That last part is a fact about what a percentile is. The p95 of a set is the value one task in twenty sits at or above, and one in twenty of three hundred is fifteen tasks. Making a single task arbitrarily expensive moves it up the sorted order and shifts everything above it down one rank, changing the value at the p95 rank by at most one neighbour's worth. Dragging that rank from $0.38 to $1.42 takes roughly fifteen tasks getting expensive together, which is a task type rather than an accident.

The mean-over-median ratio is worth carrying as its own line for the same reason a p95-to-median ratio is carried for latency: it is a direct measurement of how unlike the typical run the expensive runs have become, and it is dimensionless, so it can be compared across task types with very different absolute costs. That dimensionlessness is what it licenses. A scheduling task and a document-summarisation task with wildly different absolute costs go in the same column and their skew is comparable, because dividing dollars by dollars leaves the units behind.

Its residual is that a ratio of two summary statistics moves when either end moves, and the ratio alone cannot say which. Going from 1.40 to 1.73 is consistent with the tail getting worse, with the typical run getting cheaper, or with both — and here both happened, which is only visible because the median and the mean are also reported as their own rows. A skew ratio quoted without its numerator and denominator beside it will eventually be read backwards.

The mean as a regression detector#

The mean did its job. It is the honest average of what three hundred tasks cost, and it moved by an amount that correctly reflected the total. A team watching it every release therefore believes it is watching a defence, and the belief is tempting for a real reason: the mean is the correct statistic for the question of what a given volume of tasks will cost in total, which is the question a budget forecast asks.

It fails as a detector for a reason visible in its definition. A sum has no memory of which addends moved, so three tasks each rising by several dollars and three hundred tasks each falling by a fraction of a cent enter it as the same kind of thing, and no operation on the result can separate them again afterwards. A mean cannot distinguish "a few rows moved by a large amount" from "many rows moved by a small amount", by construction rather than by bad luck, and the blind spot grows with the count: dividing by three hundred shrinks the visible effect of those three tasks by a factor of three hundred while doing nothing to shrink the invoice they generate. The quantity that stayed constant here was the mean's own behaviour — small movements the team read as release noise, throughout a change that made one task type dozens of times more expensive.

Naming the tail, not just measuring it#

A percentile is a summary of the tail, and it is not enough on its own, because the action after seeing p95 move is always to go and read a specific trace. A cost report that stops at percentiles sends the reader back to a query console; one that names the five most expensive task IDs alongside their step counts sends them straight to a trace.

@dataclass
class TaskCost:
    task_id: str
    run_id: str
    model_calls: int
    input_tokens: int
    cached_input_tokens: int
    output_tokens: int
    tool_calls: dict[str, int]      # {"list_availability": 22, "read_thread": 3}
    priced_tool_cost_usd: float     # search/enrichment APIs billed per call
    retries: int
    retry_cost_usd: float           # spend on attempts that produced nothing
    total_usd: float
    steps: int

def cost_report(costs: list[TaskCost], top_n: int = 5) -> dict:
    totals = sorted(c.total_usd for c in costs)
    q = lambda p: totals[min(len(totals) - 1, int(p * len(totals)))]
    worst = sorted(costs, key=lambda c: c.total_usd, reverse=True)[:top_n]
    return {
        "n": len(costs),
        "median": q(0.50), "p90": q(0.90), "p95": q(0.95), "p99": q(0.99),
        "max": totals[-1],
        "mean": sum(totals) / len(totals),
        "skew_ratio": (sum(totals) / len(totals)) / q(0.50),
        "retry_share": sum(c.retry_cost_usd for c in costs) / sum(totals),
        "tail": [{"task_id": c.task_id, "run_id": c.run_id, "usd": round(c.total_usd, 3),
                  "steps": c.steps, "input_tokens": c.input_tokens,
                  "top_tool": max(c.tool_calls, key=c.tool_calls.get)} for c in worst],
    }

retry_cost_usd is broken out separately from the rest because it is the one component that is pure waste by definition — tokens spent on attempts that produced no usable result. On the scheduling set it ran at 4% of total spend in normal conditions and 31% under the sustained fault fixtures from chapter 3, where a tool is made to fail for a sustained window rather than on one call so that the agent's stopping behaviour is what gets tested. That is a useful pair of numbers to have when someone proposes raising the retry limit.

Running that report on release B and reading the tail entry for the worst task gives the whole diagnosis in one row.

{"task_id": "sched-045", "run_id": "r-91d3", "usd": 6.601, "steps": 41,
 "input_tokens": 2118400, "top_tool": "list_availability"}

Forty-one steps, 2.1 million input tokens, and the dominant tool is the availability lookup — which is the widening search loop from chapter 2, where the agent extends its availability search by a day on each pass until it finds a slot everybody is free for, now with a price attached. Each field answers a different question: the dollar figure says whether to care, the step count and token count say whether the loop and its transcript were the cost, the tool name says which call repeated, and the run ID is what you paste into a trace viewer. A percentile answers none of those.

Where the money went matters more than how much there was#

A single total per task tells you a run was expensive and gives no purchase on what to do about it. The decomposition does, because each component has a different remedy.

Every priced row there can be rebuilt from a token count and a rate, which is what makes the table checkable rather than something to be believed. The input rows use the $3.00 per million quoted above: 2,118,400 tokens is a little over two and an eighth million, and three times that is $6.36, while 54,200 tokens is about fifty-four thousandths of a million, and three times that is $0.16. The priced-tool rows are a count times a unit price, so twenty-two calls at $0.004 is a little under nine cents and three calls is a little over one cent. The output rows stand as given, since the table quotes no output rate to multiply by. Adding the tail column downwards reproduces $6.60 exactly, and the median column reproduces $0.19 to within the half-cent the cached row contributes.

Doing that addition is what lets you check the share claim instead of accepting it. Input tokens are 96% of the tail task — $6.36 of $6.60 — and 84% of the median one, which is the normal shape for a tool-using agent and is the first thing to check when a cost report surprises you. It also explains why the cached-input row deserves attention: cached reads are billed at a fraction of the base rate, so the cache is a multiplier sitting directly on the term that is nearly the whole bill.

Prompt caching deserves its own row in that table because it is the one lever that attacks the quadratic term directly. Providers price a cached read at roughly a tenth of the base input rate and charge a premium of about a quarter on the write that populates the cache, with a cache entry that expires after a few minutes of inactivity. A tenth of $3.00 per million is thirty cents per million, and eighteen thousand tokens at that rate is a little over half a cent, which is the $0.005 in the median column — so the row is priced consistently with the tenth-rate claim. For an agent that re-sends a growing message list on every iteration, that is close to an ideal workload, since the prefix is stable and the calls are seconds apart. The condition is that the prefix must be byte-identical, and because a cached read costs a tenth of a full one, a run pays the full rate on the fraction that missed and a tenth on the fraction that hit, so the dominant term is multiplied by one minus nine tenths of the hit rate.

That makes cache hit rate worth reporting per run, computed as cached input tokens over total input tokens. The denominator has to be written down beside the figure, since a cached read is still an input token and counting it in or leaving it out gives two different numbers for the same run. On the scheduling set the median run sat at 0.33 and the tail runs at 0.00, and the zero is diagnostic on its own. A miss on the first call is expected, because there is nothing yet to match against. A run that makes forty-one calls and misses on every one is saying that no call's prefix matched the previous call's, and the prefix is the part of the transcript that by construction does not change — so a long run with no cache hits almost always means the prefix is unstable rather than that the run is genuinely novel, which is a claim about the code that builds the prompt rather than about the task.

Cost per successful task is the number the business asked for#

Cost per task counts the failures at the same rate as the successes, and failures are frequently the expensive ones, since a run that flails is a run that makes many calls. Dividing total spend by the number of runs that passed the assertions from chapter 1 — the observable conditions on the end state that decide whether a run counts as a success — gives cost per successful task, which is what a conversation about unit economics needs: what it costs to produce one unit of the thing the product actually sells, here one scheduled meeting.

The relation between the two numbers is fixed, and deriving it takes one line. Cost per task is total spend over the number of tasks, and cost per success is the same total spend over the number of passes. The pass rate is passes divided by tasks, so the number of passes is the pass rate times the task count. Substituting that into the second expression gives total spend over the pass rate times the task count, and dividing numerator and denominator by the task count — allowed, since the task count is not zero — leaves cost per task divided by the pass rate. Cost per success is cost per task over the pass rate, always.

The figures close against it. On release B, $46.20 of total spend over 300 tasks with 267 passes is $0.154 per task and $0.173 per success. Two hundred and sixty-seven passes out of three hundred is a pass rate of eighty-nine percent, and $0.154 divided by that rate gives back $0.173 per success. The gap widens fast as the pass rate falls: at 71% the same spend is $0.217 per success, which the same division reproduces, so a correctness regression shows up as a cost regression even when nothing about the cost mechanics changed.

The identity licenses dropping one of the two numbers from a report: given the pass rate, either converts into the other, so a dashboard carrying cost per task and the pass rate already carries cost per success. Its residual is that the derivation held total spend fixed while the pass rate moved, and that is not what happens. Failing runs are disproportionately the expensive ones, so a release where the pass rate drops is usually also a release where total spend rises. The identity is a floor on how fast cost per success moves rather than a forecast of it, and a team using it as a forecast will be surprised in the same direction every time.

Breaking the same figures down by task variant is usually more actionable than breaking them down by percentile, because a variant is a thing you can change. On this set, single-attendee same-timezone tasks cost $0.06 per success while multi-attendee cross-timezone tasks cost $0.94, a factor of fifteen, and the second group was 12% of the eval set and 41% of production traffic. That mismatch has consequences for forecasting that go well beyond the cost report, and it is the same mismatch a later chapter chases from the direction of the success rate, where an eval set whose input mix does not resemble production reports a pass rate users disagree with.

Cost tails, latency tails and volume forecasts#

Cost is not latency, and the two diverge often enough that reporting one as a proxy for the other misleads. A run that makes three sequential calls to a slow enrichment API takes eleven seconds and costs $0.04; a run that makes forty short model calls costs $6.60 and, with a fast model, may finish sooner. Both need percentile reporting and they need it separately, since the tasks in the cost tail and the tasks in the latency tail are frequently different tasks.

Percentiles taken deep into a small set are shaky, and p99 is the row to be careful with. One in a hundred of three hundred tasks is three tasks, so p99 here is computed from the behaviour of three runs and will move when any one of them changes, including for reasons that have nothing to do with the release. The p95 row rests on fifteen tasks and is correspondingly steadier, which is why it is the row to gate on while p99 is read for direction.

The mean is also not useless — it is the correct statistic for the one question it answers, which is what a given volume will cost in total. Multiplying it by projected volume is exactly right for a budget forecast, provided the eval set's mix of inputs resembles production. That proviso is doing a great deal of work and is worth stating explicitly whenever the number is quoted, because an eval set built from tidy two-attendee threads will produce a mean that underestimates a customer base full of five-attendee cross-timezone ones, and the underestimate is largest precisely in the tail that generates the invoice.

Finally, the report says what a run cost and takes no position on whether that was acceptable. Run sched-045 produced a correct meeting: the assertions from chapter 1 all pass, no unsafe action was taken, and a human scheduler dealing with three busy calendars might well have made a comparable number of attempts. Forty-one steps and $6.60 for a correct answer is either an acceptable outcome or a release blocker, and nothing in a distribution can settle that.

What settles it is a budget somebody wrote down before the eval ran. The quadratic in the step count, the percentile gaps, the component decomposition and the cost-per-success identity are all machinery for saying precisely what a run cost and where the money went, and none of them contains a threshold or can be made to produce one, because what a scheduled meeting is worth is not a question about tokens. The distribution's job is to make the expensive runs impossible to miss and to name them by task ID; deciding what to do about them is a separate act, performed in advance, by somebody who knows what the product charges.