Per-step costs that do not reconcile to the bill are wrong

The claims-triage agent this topic has been instrumenting is a loop that reads a motor insurance claim, calls tools to fetch the claim record, run optical character recognition over the attached documents and price a repair, and then writes a decision. Every model call the loop makes is wrapped in a span carrying the model name, the token counts the provider's API returns, and a cost computed at the moment the span is written. Those spans feed a cost dashboard that breaks spend down per step, per tool and per run. It was built the way the first chapter of this topic says to build it, it contains no defect anybody has been able to point at, and for its first four weeks it reported roughly a third of the money.

A per-step cost is unlike most of the numbers a system reports about itself, because somebody else is independently measuring the same quantity and sending you a bill for it. Reconciliation is the practice of summing your own attributed costs over a period and comparing that sum against the provider's figure for the same period. Without it a dashboard reports the money it happened to be told about, the shortfall is invisible because nothing anywhere is red, and the first evidence arrives as an invoice weeks later.

The run the dashboard was describing#

Start with what a priced model call is made of, because the arithmetic is short and every later number rests on it. The provider reports token usage in four categories and charges a different rate for each, and the categories exist because the four cost it different amounts to serve. Fresh input tokens are processed from scratch, and output tokens are generated one at a time, which is the expensive direction. The other two come from prompt caching, under which the provider keeps the tokens of a prefix it has already processed and charges a much reduced rate to read them back on a later request, in exchange for a one-off premium to write them. The cost of one call is therefore the sum, over those four categories, of the tokens in the category times that category's rate.

The rates in play, for reference: claude-opus-5 bills $5.00 per million input tokens and $25.00 per million output; cached input reads at $0.50 per million and cache writes at $6.25. claude-haiku-4-5 bills $1.00 and $5.00. The agent sets a cache breakpoint on the tools and system prefix and a rolling one on the last message of each turn, so from step two onward it pays the read rate for everything the previous step already sent.

How those rates are quoted explains a line of code further down that looks like a scaling bug and is not. A token count multiplied by a dollars-per-million rate is not a figure in dollars; it is a figure in millionths of a dollar, and a millionth of a dollar is a micro. That is why the pricing function below multiplies a token count by a dollar rate and returns micros with no division in it: the division by a million already sits in the units of the rate.

Every client.messages.create call in the agent loop was instrumented — that was the point of the work in chapter one, and it was done correctly. A median six-step run produced six llm.completion spans, and each carried the model name, the four token categories the API reports, and a cost computed at write time from the rate then in force.

Work the first row in full, because the whole table is that one calculation repeated and the general form is easy to believe once one instance has been seen. Step one sends 340 fresh input tokens, and at five dollars a million those contribute seventeen hundredths of a cent. It writes 1,800 tokens into the cache, and at six dollars twenty-five a million those contribute exactly one and an eighth of a cent. It generates 96 output tokens, and at twenty-five dollars a million those contribute twenty-four hundredths of a cent. There are no cache reads on the first step, because nothing has been cached yet for it to read — the cache write on this row is what creates the prefix the later rows read back. Adding the three and rounding to the nearest hundredth of a cent gives the 1.54¢ in the table.

That the cache write dominates the row is the intuition to carry, and it inverts on the very next line. Writing the cache is the most expensive thing per token in the table apart from output, so the first step pays a premium; from step two onward the same prefix reads back at a tenth of the input rate, which is why step two costs less than step one despite sending more fresh input and more output.

The rest of the table is the same shape with more rows. Add the six costs down the last column and the total comes to the 11.24¢ recorded as the run total, which is the check that the per-step figures and the per-run figure describe the same run rather than having been computed by two paths that drifted apart.

That check passing is what made the dashboard convincing, so be precise about what it establishes. It establishes that the rows sum to the row labelled total. It establishes nothing about whether the rows are all the rows. Read the table as an artefact of the instrumentation rather than as a description of the run, and the missing money is already visible in what the columns do not have. There is no row for anything that happened inside tool.search_policy. There is no row for an attempt that failed. There is one row per step, and steps are the only unit the loop knows about.

Each absence follows from where the span was opened, which was the agent loop, at the point where the loop calls the provider. The set of calls the loop can record is therefore the set of calls the loop itself makes. A model call issued inside a tool's own code is made by a function the loop invoked and then waited on, so the loop sees a string come back and has no way to know what happened in between. A request the provider billed but that never returned a usable answer does not reach the line that writes the attribute. In both cases the money left and no row was created, and no amount of care in the loop's instrumentation changes it, because the loop is not where the call happened.

Where the other two-thirds were#

Model calls made inside tools accounted for $31,600. The search_policy tool does not query a database; it embeds the query, retrieves forty candidate clauses, reranks them with claude-haiku-4-5, and summarises the top five with claude-opus-5 so the agent receives prose rather than a wall of policy text. ocr_document sends page images to a vision model. From the agent loop's point of view both are function calls that return a string, and in the trace both are single spans with a duration and no token counts at all. The tool span for search_policy averages 3.6 seconds and 5.9¢, and the dashboard recorded 0¢ for it 640,000 times in April.

This is the failure that generalises. Any tool sophisticated enough to be worth having is a candidate for containing a model call, and the agent's loop is structurally incapable of noticing, because a tool is a function boundary and cost is not part of the function's return type. Teams find it late because the tool works: nothing is broken, the estimate is correct, the latency is acceptable, and the only symptom is an invoice.

The belief that gets a team here is a reasonable one held for a good reason, and it is worth naming as a belief. Every model call in the agent loop was instrumented, and the team could show it: the span count matched the run count times the expected fan-out to within a percent. That check is real, it is the right check, and it passes. What it measures is completeness against the set of calls the loop makes, and that quantity stays at its ceiling no matter what happens inside a tool — while the property that decides whether the dashboard is right, the share of billed calls originating inside the loop rather than beneath it, falls every time somebody makes a tool cleverer. The measured quantity is constant while the dangerous one varies, which is why nothing moves on any dashboard on the day a tool starts calling a model.

Retried and failed attempts accounted for $13,200, in three distinct flavours worth separating. The SDK's built-in retry issues up to two additional requests on a 429 or a 5xx, and those requests are made below the layer that writes the cost attribute, so a run that hit rate limits twice wrote one cost row for three billed requests. A stream that errors partway through has already produced tokens and is billed for them, and the exception path in the loop returned before the span was closed, so the usage was never read. And 1.1% of runs hit the ten-step cap, at which point the loop returned {"stop": "max_steps"} and a caller that treated the run as failed discarded it — but the cost hook was wired to the success return, so ten steps of billed work left no rows behind. Abandoned work is still billed work; the provider does not care whether you liked the answer.

Escalations to a second-opinion agent accounted for $4,700. When the primary run's confidence falls below a threshold it spawns a second run against a different system prompt, and that run was correctly instrumented — with its own trace id, generated fresh, with no link back to the parent. Its spans existed. The query that built the dashboard filtered to traces whose root span was named run.claims_triage, and the escalation's root was named run.second_opinion.

Those three are worth adding up rather than reading as a list, because the sum is the finding. The dashboard reported $25,400 against an invoice of $74,900, so the money the instrumentation could not see is $49,500 — and tool-internal calls, retried and failed attempts and orphaned escalations come to that figure exactly. What a closing sum licenses is the end of the search: the three account for the whole of the gap rather than most of it, so the dashboard's 33.9% is explained rather than merely reduced, and no fourth mechanism is still hiding. That is a stronger position than it sounds, because a remainder is the only thing that tells you to keep looking — an investigation that stops while the named causes still fall short of the gap has found some of the answer and has no way to know how much.

The residual matters as much as the licence, because it is the argument for the reconciliation later in this chapter. The sum closes for April, at the granularity of a monthly total, for the tools this agent had in April. It says nothing about which mechanisms can exist, and a fourth ships the moment somebody adds a tool with a model call inside it. That is why the reconciliation is a standing daily check rather than a one-off audit, and why the alert fires on a change in the ratio rather than on the gap being anything other than zero.

Emit the span where the call is, not where the loop is#

That every billed request should leave a record is a requirement, and it is the only part of this that is not negotiable. Where the span is opened is a design decision, and it is the one that decides whether the requirement can be met at all.

The structural fix is one line of design: an llm.completion span is emitted by the client wrapper, not by the agent loop. Once every model call in the process goes through one function that opens a span, tool-internal calls land inside the enclosing tool span automatically, because the tracer's context is ambient and the tool span is already current when the tool's own code runs.

That last clause is doing more work than its length suggests, and it is why one line of design fixes the largest of the three leaks without anybody having to remember anything. The tracer keeps its reference to the currently open span in context-local storage, and every new span reads that reference to fill in its own parent pointer. When the loop opens a tool.search_policy span and then calls the tool, that span stays current for the whole duration of the tool's body. So a model call made three functions deep inside the tool opens a correctly parented span without the tool's author doing anything, and without the loop knowing the call happened at all.

PRICES = {  # USD per million tokens, effective 2026-03-01
    "claude-opus-5":    {"in": 5.00, "out": 25.00, "cache_read": 0.50, "cache_write": 6.25},
    "claude-haiku-4-5": {"in": 1.00, "out":  5.00, "cache_read": 0.10, "cache_write": 1.25},
}


def price_micros(model: str, usage) -> int:
    p = PRICES[model]
    return round(
        usage.input_tokens * p["in"]
        + usage.output_tokens * p["out"]
        + (usage.cache_read_input_tokens or 0) * p["cache_read"]
        + (usage.cache_creation_input_tokens or 0) * p["cache_write"]
    )


def traced_completion(**kwargs):
    """The only place in the process that calls the provider."""
    with tracer.start_as_current_span("llm.completion") as span:
        span.set_attribute("gen_ai.request.model", kwargs["model"])
        try:
            response = client.messages.create(**kwargs)
        except anthropic.APIStatusError as exc:
            span.set_attribute("gen_ai.error.status", exc.status_code)
            span.set_attribute("cost.micros", 0)      # nothing produced, nothing billed
            span.set_status(trace.StatusCode.ERROR)
            raise
        u = response.usage
        span.set_attribute("gen_ai.response.model", u_model := response.model)
        span.set_attribute("gen_ai.usage.input_tokens", u.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", u.output_tokens)
        span.set_attribute("gen_ai.usage.cache_read_input_tokens", u.cache_read_input_tokens or 0)
        span.set_attribute("gen_ai.usage.cache_creation_input_tokens", u.cache_creation_input_tokens or 0)
        span.set_attribute("cost.micros", price_micros(u_model, u))
        span.set_attribute("cost.price_version", "2026-03-01")
        return response

Three details in that function are load-bearing. It prices on response.model rather than on the requested model, because an alias can resolve to something other than what you asked for and a server-side fallback can serve the response from a different model entirely at a different rate. It writes cost.micros on the error path too, so a failed call produces a span whose cost is explicitly zero rather than a span with no cost attribute, which is the shape that later gets silently excluded from a sum. And it stamps the price table version, because the rate that was in force is a property of the row and not a lookup to be redone later — the argument for that was made in 2.16 and applies unchanged here.

The middle of those three reads as pedantry and is not. A summing query skips a row whose attribute is absent and adds a row whose attribute is zero, and both give the same total — indistinguishable in the number, completely different in the record. Only the explicit zero leaves evidence that a call happened at all, which is what lets anybody later count the calls that contributed nothing and ask why there were so many. The first detail has the same shape: pricing on the request rather than the response computes correct arithmetic on the wrong rate precisely on the days something unusual happened upstream, which are the days somebody comes looking.

The SDK's own retries still happen underneath this. Setting max_retries=0 on the client and implementing the backoff in your own wrapper, one span per attempt as in chapter one, is the version that leaves a record. The alternative is to accept that a small fraction of billed requests are invisible and to size the reconciliation tolerance accordingly, which is a defensible choice as long as it is a choice.

Two costs per span, and only one of them sums#

Once model calls appear inside tool spans, every non-leaf span in the trace has two defensible cost figures and confusing them produces totals that are wrong by a factor of two. The exclusive cost of a span is what its own model calls billed, which for a tool.* span is zero and for an llm.completion span is the whole of it. The inclusive cost is the exclusive cost plus everything its descendants billed, which is what makes tool.search_policy a 5.9¢ span and run.claims_triage a 41.6¢ one.

Both are useful and they answer different questions. Inclusive cost tells you which tool to optimise; exclusive cost is the only one you can sum without double-counting, because adding a parent's inclusive cost to its children's re-counts the children. The reconciliation query in the next section sums exclusive cost by filtering to name = 'llm.completion', which is the leaf-only version of the same rule and the reason it is safe.

The useful way to hold the pair is as a gap rather than as two levels. The gap between a span's inclusive and exclusive cost is exactly what its descendants billed, so a tool.* span whose two figures differ is a tool with model calls inside it, and one whose figures are both zero is a tool that only moved data around. Under the old instrumentation every tool span had a zero inclusive cost, which is to say the gap was identically zero everywhere and the one quantity that would have exposed the problem had been defined out of existence.

Write the exclusive figure and derive the inclusive one at query time with a recursive rollup, rather than trying to maintain a running total on parent spans. Spans are exported as they close and children close before parents, so a parent could in principle accumulate its children's costs — until a child is a sub-agent in another process that finishes forty seconds after the parent, at which point the maintained total is silently short. Anything derived at query time is correct whenever it is asked; anything accumulated at write time is correct only if nothing arrives late, and in an agent something always arrives late.

Reconciliation is the part that makes it true#

Instrumentation is a hypothesis about where the money goes. The reconciliation is the test, and without it there is no reason to believe the number. Run it daily against the provider's usage API rather than monthly against the invoice, so that a divergence is a Tuesday problem rather than a quarter-end one.

WITH spans AS (
  SELECT date_trunc('day', start_time)          AS day,
         attributes->>'gen_ai.response.model'   AS model,
         SUM((attributes->>'cost.micros')::bigint) / 1e6 AS instrumented_usd
  FROM otel_spans
  WHERE name = 'llm.completion'
    AND start_time >= now() - interval '30 days'
  GROUP BY 1, 2
)
SELECT p.day,
       p.model,
       p.provider_usd,
       COALESCE(s.instrumented_usd, 0)                       AS instrumented_usd,
       p.provider_usd - COALESCE(s.instrumented_usd, 0)      AS unattributed_usd,
       ROUND(100 * (1 - COALESCE(s.instrumented_usd, 0) / NULLIF(p.provider_usd, 0)), 1) AS gap_pct
FROM provider_usage_daily p
LEFT JOIN spans s USING (day, model)
ORDER BY p.day DESC, p.model;

The unattributed_usd column is the deliverable, not a diagnostic. Publish it on the dashboard next to the attributed spend, because a chart that shows $25,400 attributed and $49,500 unattributed tells the truth about what you know, while a chart that shows $25,400 and stops does not. Alert when gap_pct moves outside a band rather than when it is non-zero — some gap is permanent, since negotiated rates mean your list-price sum sits systematically above the invoice while unlogged SDK retries pull it below, and the useful signal is a change in the ratio rather than its absolute value. The claims team settled on ±4 percentage points day over day, which fires roughly once a quarter and has twice caught a new tool shipping with an uninstrumented model call inside it.

Alerting on a change rather than on a level is the design decision the permanent gap forces, and it is worth seeing why no threshold on the level would do. The two standing distortions push in opposite directions and neither is knowable in advance: discounts pull the ratio one way by an amount that depends on a contract, unlogged retries pull it the other by an amount that depends on how flaky the provider was that week. Their net is a number nobody can derive, which rules out asserting what it ought to be. What can be asserted is that it should not move much overnight, since neither contract terms nor retry rates do — so the day-over-day difference is the quantity with a defensible expected value, and a new uninstrumented call site appears in it as a step.

Reconciling against the usage API and reconciling against the invoice are different exercises and you want both at different cadences. The usage API reports consumption at list price and is available the next morning; the invoice reports money after discounts, committed-use credits and any fallback repricing, and arrives weeks later. Treating the invoice as the daily source of truth means finding out about a regression a month after it shipped, and treating the usage API as the final word means being surprised at the end of the quarter by a credit nobody modelled.

Worth and enforcement are different questions#

A per-step cost is not a measure of what a step was worth. The most expensive span in the median run above is the last model call at 3.82¢, and it is the one that writes the decision — cheap for what it does. The cheapest is step two at 0.70¢, which fetched a claim record the agent could have been handed in the initial prompt for nothing. Cost attribution tells you where the money is, and the question of whether that money bought anything is answered by evaluation, not by tracing. A team that optimises the largest number on the cost dashboard without asking what it produced will reliably cut the wrong step.

It is also not a budget control, and the reason is when the attribute is written rather than what it contains. A cost.micros attribute is written after the call has been made and the money has been spent; nothing about having the number prevents a run from consuming forty dollars. Enforcement needs a running total consulted before each request, which is a different mechanism with a different failure mode, and the trace's role in it is to explain afterwards why the ceiling was reached rather than to stop it happening.

A mean that reconciles is the strongest statement this chapter can make, and it is a statement about a total rather than about any run. Multiplying the per-run mean across April's 180,000 runs returns the invoice, to within the rounding on the mean itself, so the per-run figure can now be multiplied out to the bill and lands on it. That licenses arithmetic over the whole population and nothing about any member of it, because a total is compatible with every distribution that sums to it.

With the sums reconciling, the aggregate is finally trustworthy, and the first thing it shows is that the aggregate was never the interesting part. The median run costs 19¢ once tool-internal calls are counted. The mean costs 41.6¢. That gap is the whole story, and finding out what lives inside it is not a matter of looking harder at prices.