A p95 eight times the median is a step count, not a price

With the sums finally reconciling against what the provider charged — the work of the previous chapter, which found two thirds of a claims-triage agent's money sitting in model calls made inside tools and in attempts nobody had recorded — the distribution of run cost was legible for the first time and it was lopsided. Median run: 19.2¢. Ninety-fifth percentile, meaning the level only one run in twenty exceeds: $1.52, almost exactly eight times the median. Mean: 41.6¢, which sits above the seventy-eighth percentile — a distribution whose average describes almost none of its members. The top 2.4% of runs by cost consumed 28% of April's $74,900.

A shape like that raises one question with money attached: what the runs at the top are doing that the runs in the middle are not. Two answers are available before any evidence is looked at, and they point at different parts of the system. Either a small number of calls are individually expensive, which leads to prices and model choice, or a small number of runs make very many ordinarily priced calls, which leads to control flow.

The most expensive call and the half-percent it explains#

The first thing anyone does with that is sort llm.completion spans by cost.micros descending and look at the top of the list. It is the obvious move and it produces a real answer to the wrong question. April's single most expensive model call cost 41.3¢: a prompt of 78,400 tokens against claude-opus-5, on a run that had accumulated an unusual amount of document text. The thousand most expensive individual calls in the month summed to $412. Against a $74,900 invoice, the entire top of the by-price list explains half a percent of the bill.

That is worth naming as a defence that does not work, because running the sort is not careless: it is the correct query for the question which call was expensive, and everything it returns is true. What makes it useless is that it ranks the one quantity here that barely varies. What a call costs is set by the tokens it carries and the per-token rate, both bounded in practice, which is why the largest call in the whole month came to 41.3¢. Step count is not bounded that way — it runs from 6 at the median to 31 at the ninety-fifth percentile — so the property being ranked stays nearly constant while the property driving a run's cost varies by a factor of five.

Anyone who has spent an afternoon in a CPU profiler has met this shape before. The slowest single function invocation in a trace is almost never the reason a program is slow; the reason is a function that costs forty microseconds and is called nine million times. Sorting by self-time per call finds the pathological outlier and misses the hot loop, which is why profilers show you total time and call counts side by side. Agent traces have exactly the same structure and the same trap, and the tail here is a hot loop: runs doing very many ordinarily priced things.

That is an analogy rather than a claim about how the two systems are built, and it has a price. A profiler's call-count and total-time columns are exact, because the profiler observes every invocation inside the process it is attached to; the equivalent columns here are computed over the traces a sampler chose to keep, which is a failure mode the profiler version has not got and one with a section of its own below.

Compare shapes, not prices#

The unit of comparison that works is the trajectory shape — the count of steps in a run, and the sequence of tool names within it. Both are derivable from the span hierarchy of chapter one, and neither is affected by the redaction policy of chapter two, which is what makes this the first analysis available on a redacted trace store.

WITH runs AS (
  SELECT r.trace_id,
         SUM(c.cost_micros) / 1e6                                  AS run_usd,
         COUNT(*) FILTER (WHERE s.name LIKE 'step.%')              AS steps,
         COUNT(*) FILTER (WHERE s.name LIKE 'tool.%')              AS tool_calls,
         string_agg(DISTINCT substring(s.name from 6), ',')
           FILTER (WHERE s.name LIKE 'tool.%')                     AS tools_used
  FROM otel_spans r
  JOIN otel_spans s USING (trace_id)
  LEFT JOIN LATERAL (
    SELECT SUM((l.attributes->>'cost.micros')::bigint) AS cost_micros
    FROM otel_spans l
    WHERE l.trace_id = r.trace_id AND l.name = 'llm.completion'
  ) c ON true
  WHERE r.parent_span_id IS NULL AND r.name = 'run.claims_triage'
    AND r.start_time >= now() - interval '7 days'
  GROUP BY r.trace_id, c.cost_micros
)
SELECT width_bucket(steps, 1, 41, 20) * 2 AS step_band,
       COUNT(*)                           AS runs,
       ROUND(AVG(run_usd)::numeric, 3)    AS mean_usd,
       ROUND(SUM(run_usd)::numeric, 0)    AS band_usd,
       mode() WITHIN GROUP (ORDER BY tools_used) AS typical_tools
FROM runs
GROUP BY 1 ORDER BY 1;

What that intends is a step count, a tool-call count, a set of tool names and a cost for each run, bucketed by step count. Nothing in it reads a payload, which is why it runs against a store that has none. The output of that query answered the question in about ninety seconds of reading. Step count at the median was 6; at p95 it was 31. Cost at p95 was 8.0× the median while steps were only 5.2×, and the extra factor of 1.5 is the resent prefix: each additional step also re-sends everything before it, so a long run pays for its own length twice over.

That last clause carries the argument and is an assertion until the two payments are named. What a call bills is set by how many tokens it sends, and a call inside an agent loop sends the whole conversation so far — the system prompt, the tool definitions, every earlier assistant turn and every earlier tool result — because that is the material the model needs in order to choose the next action. Step count therefore enters the total in two places: it sets how many calls get made, which is the first payment, and how long a prefix each of them carries, which is the second. Adding steps raises both at once, so the cost ratio between a long run and a short one has to come out larger than their step ratio.

One condition on that has to be stated aloud, or a reader will mispredict the next case. It holds while the steps of a long run append about as much to the message list as the steps of a short one do; a run reaching thirty-one steps through thirty-one tiny calls appending almost nothing would show no such effect. The runs in this tail are repeated tool calls with results attached, which licenses reading the gap between the two ratios as prefix growth rather than as something about prices.

The mechanism predicts the direction of that gap; the measurement supplies its size, and 8.0 against 5.2 leaves the extra factor of 1.5. Notice how small a residue that is, because the mechanism alone predicts a far larger one: whatever a run appends at its first step is sent again by every step that follows. The reason the bill does not reflect that is the prompt caching set up in the previous chapter, under which the provider keeps the tokens of a prefix it has already processed and charges a much reduced rate to read them back. The cache decides what the second payment costs.

The step-count histogram was not a smooth tail either. It had a fat body around 5–8 steps, a near-empty valley, and a second hump between 24 and 31 — which is the signature of a mechanism rather than of natural variance. A distribution with two humps has two populations in it.

The evidence there is the valley and not the hump. Claims vary continuously in how much work they need, so if step count were set by that variation alone the counts would spread outward from the body without a break: there are claims at every intermediate difficulty, and each would produce a run at some intermediate length. An empty stretch in the middle says nothing is producing runs of that length. The population reaching 24 steps did not get there by being slightly harder than the population that stops at 8, because if it had, the runs in between would exist.

Three mechanisms, and the query that separates them#

Once the tail is known to be long runs, the useful move is to enumerate what makes a loop run long and then test each against the same trace store. There are not many candidates, which is what makes this tractable.

Add the last column up before reading any single row of it. The three real mechanisms come to $9,100, $4,400 and $5,200, the null hypothesis accounts for about $1,900, and the by-price instinct for $412 — together roughly twenty-one thousand dollars, which is what 28% of April's $74,900 comes to. The tail closes, and that is the difference between an investigation that has finished and one that has merely stopped: no sixth mechanism sits behind these five, so anyone proposing a further cause has to displace one rather than add to them.

The residual is in the column heading. Those are shares of April's tail, not the total cost of each class of behaviour, and the two come apart whenever a mechanism also affects runs that never reach the top 2.4%. The empty-result loop touched about ten thousand runs in April against a tail of 4,320, so most of the runs it affected are counted somewhere other than in that $9,100. A tail share is a floor under what a fix is worth rather than an estimate of it.

The last row is the instinct the chapter opened with, kept in the table because seeing it sized next to the others is the argument. The fourth row is the null hypothesis, and it matters that it accounts for something: about $1,900 of the tail really is the agent working harder on harder claims, and that spend is correct. An investigation that finds no legitimate expensive runs has probably mis-specified its filter.

A null hypothesis here is the explanation that has to be ruled out before the others count as findings — that nothing is wrong and the expensive runs are expensive because the claims were hard. It carries a size because a size is what makes it checkable, and the signature the table records for it is a smooth relationship between step count and difficulty, which is exactly what the histogram does not show.

The largest of the three real mechanisms is worth drawing, because it is a cycle and a cycle is the one thing prose describes badly. Roughly 3% of policies live in a legacy schema that search_policy indexes under a different clause key. For those, the tool returns [] — not an error, a valid empty result. The model, correctly, treats an empty result as "my query was wrong" rather than "no such policy exists", rephrases, and searches again:

The model's reading is the right one, and that is what makes the cycle stable rather than self-correcting. An empty list is what a search returns when the query was badly phrased, and also what it returns when the clause genuinely is not there, and nothing in the result separates the two. Rephrasing is the right response to a signal carrying both meanings, and it produces a new query, another empty list and the same ambiguity. Every pass is a defensible decision and the sequence of them is a defect. The loop terminates only at the step cap, which means every affected run pays the maximum and then returns nothing useful.

The retry storm is narrower and cheaper. The estimates provider reindexes nightly between 02:10 and 02:40 and returns 503 throughout, and during that window the agent's three-attempt wrapper exhausts itself, returns an error result to the model, and the model retries the whole step — so a single logical tool call becomes nine HTTP attempts across three steps, each of which re-sends a growing prefix. Only 1.8% of daily volume falls in that window, which is why it never showed up in any daily aggregate; it is visible only when the attempt counts are bucketed by hour.

Those nine are two retry layers stacked on each other, neither aware of the other: the wrapper's three belong to the tool dispatch, and the model's three to the trajectory, since it reads the returned error as information about the call it just made. Three inside three is nine, and the cost lands as extra steps rather than as extra tool spans.

The document fan-out is a different animal and worth separating carefully, because it is not a step-count problem at all. A forty-page GP bundle produces forty ocr_document child spans, and their combined text — around 28,000 tokens — is appended to the message list and therefore re-sent, cached or not, on every subsequent step. The run takes a perfectly normal seven steps. It just carries five times the payload through all of them. Any analysis that groups purely by step count classifies these runs as unremarkable, which is why the shape query above also carries a token total. Shape records how many times the prefix was sent and nothing about how long it was, so step count and token total are independent axes and an investigation projecting onto one loses whatever lives on the other.

The sampler decides which population you are looking at#

There is a way to do all of the above correctly and still get a wrong answer, and it is worth naming because it is invisible from inside the query. Everything here is an aggregate over the traces you kept, and if the sampler that decided which traces to keep is correlated with the thing being measured, the aggregate describes a population that does not exist.

Head sampling — deciding at the root span, before the run has done anything, whether to keep its trace — is the correlation to watch, and the arithmetic is short. At the 5% rate the SDK defaults to, the 4,320 tail runs in April would have been represented by a twentieth of that, about 216 traces, spread across three mechanisms, three time-of-day patterns and several tools, which leaves a few dozen runs to stand for any single pattern. The second hump in the step-count histogram — the observation that started the whole investigation — would have been perhaps forty runs wide beside a body still holding thousands, and indistinguishable from ordinary variation. The evidence would have been present and unreadable, which is worse than absent.

Worse, some sampling implementations make the keep decision at the root span before any of the run has happened, which is fine, while others make it on the first exported span and drop spans from traces whose root was already flushed, which silently truncates long runs. Each link there is checkable. Spans are exported as they close and a run's root closes last, so a long trace has many children exported before its root arrives; an exporter flushing on a timer has forgotten the early ones by then, and a sampler keyed to the first export has nothing to attach the late spans to. The spans it drops sit at the end of a trace, and the traces with most spans at the end are the longest. Long runs are the ones in the tail. A sampler that drops the ends of long traces removes precisely the evidence.

The version that works is a tail-sampling policy: buffer the spans of a run until it completes, then decide, so the decision rests on facts about the finished run. Keep 100% of runs that error, 100% that hit the step cap, 100% above a cost threshold — the claims team used $1.00, which is a fifth of a percent of runs — and sample the rest at 10%. Storage went up by about 15% against the flat 5% policy and the tail became fully observed rather than estimated.

Two things are stacked inside that policy. That the tail must be observed rather than estimated is a requirement, and it follows from the question: a population of 4,320 runs split three ways cannot be characterised from a sample leaving a few dozen in each split. The thresholds — errors, the step cap, $1.00, a tenth of everything else — are a design decision about which runs are individually worth keeping, and a team with a different failure profile would choose differently and still meet it.

The one thing to check when introducing it is that the cost figures in your dashboards are scaled by the sampling rate of their bucket, because a store that keeps every expensive run and a tenth of the cheap ones will report a mean cost roughly double the truth if summed naively. That correction follows from the policy rather than sitting beside it as a second choice: expensive runs now sit in the retained set at ten times the weight of cheap ones, so dividing summed cost by trace count charges each cheap run once where it should count for ten.

That correction interacts with the reconciliation from chapter three: the daily gap check should be computed on unsampled counters rather than on the sampled trace store, or you will spend an afternoon chasing a discrepancy you created.

Long runs, query strings and the unit-price reflex#

Shape analysis found where the money went and did not, on its own, explain why the model behaved as it did in the loop. It established that nine consecutive searches happened and that each was a different query string — length and hash differed — but under the redaction policy of chapter two, which replaces free-text payloads with facts about their shape, the query strings themselves are structural metadata in the ninety-day tier. Whether the model was making progressively better guesses or oscillating between two phrasings is a question about the text, and answering it required a break-glass read of the seventy-two-hour full-fidelity tier while the pattern was still recent. That is the redaction policy working exactly as designed and also exactly as expensive as designed; the constraint has a real cost and it is paid here.

There is a second limit worth naming because it invites a wrong fix. Nothing in the analysis says that a long run is a wrong run. The 31-step runs in the tail mostly succeeded — they produced correct decisions, slowly and expensively. Cost tail and correctness failure are different populations with only partial overlap, and a team that has spent a week on the cost tail will be tempted to treat the trace viewer as an incident tool and start reading the expensive runs when a decision comes out wrong. Those are unrelated searches. The runs that produce a bad payout figure are usually six steps long and cost 19¢.

And the reflex to reach for a cheaper model when the cost tail is bad has to be checked against these numbers. Moving the whole agent from claude-opus-5 to claude-haiku-4-5 cuts the per-token rate by 80%, and applied to a 31-step run that is looping because a tool returned an empty list, it produces a 31-step run that costs 34¢ instead of $1.68 and still returns nothing. Fixing the legacy-schema index took one afternoon and removed $9,100 a month. Unit price is the lever you reach for when the work is right and the price is wrong, and here the work was wrong. Cutting a rate scales the runs that should not be happening down with everything else, which leaves the failure intact and its tail shorter and harder to find next quarter.

What the week bought was a change in what gets compared. A distribution with a long tail invites a search for expensive things, and in an agent there are hardly ever any: model calls cost what they cost, inside a band narrow enough that the most extreme one in a month is a rounding error against the invoice. The variation lives in how many of those ordinarily priced calls a run makes, which is a property of the trajectory rather than of the price list. Two conditions sit under the method: the comparison has to run over a set of traces that actually contains the tail, which is a statement about the sampler and not about the query, and the mechanisms have to be enumerated until they add up, because a list of causes accounting for half the money leaves the other half unnamed.

Both of those observations point the same way. Aggregate shape finds classes of expensive behaviour across many runs; it does not tell you where a single run went wrong, and the single wrong run is the thing on-call is actually holding. That question has its own method, and it starts from the observation that the step which throws the error is rarely the step that caused it.