The calls nobody counts are the gap in the invoice

A second team in the same organisation runs a retrieval assistant on a mid-tier model: $0.60 per million input tokens, $3.00 per million output. Two million requests a month, the same 6,000-token stable prefix, about 2,000 tokens of retrieved context, about 900 tokens out. Prompt caching is on and the dashboard reports an 85% hit rate. A cascade escalates roughly 15% of requests to a stronger model at $1.50 and $7.50, and an LLM judge — a second model that scores the assistant's answers against a rubric — runs over a 5% sample for regression monitoring.

Their model, built the way chapter 1 built one, says $14,000 a month. The invoice says $23,000, and it has said something within $600 of $23,000 for four consecutive months. Nobody has found a billing error, nobody has found a runaway loop, and the request count in the application logs matches the request count everyone agrees on. The consistency is what makes this interesting: a model that is wrong by 64% every single month is not noisy, it is missing terms.

The model as written#

Per request, the terms they wrote down come to this. The prefix, at a claimed 85% hit rate with reads at a tenth of list, is 0.85 × $0.00036 + 0.15 × $0.00360 = $0.000846. The variable context is 2,000 × $0.60 per million = $0.0012. Output is 900 × $3.00 per million = $0.0027. That is $0.004746 for a base call. Escalation was folded in as a blend — 85% of requests at the base cost and 15% at the stronger model's $0.01875 — giving $0.0068466. The judge adds a 5% sample of a small-model call at $0.00063, or $0.0000315 amortised. Total $0.0068781, times two million, is $13,756.

Two of those steps contain the error, and neither is arithmetic. The escalation was modelled as a replacement: 15% of requests priced at the strong model instead of the base model. What the cascade actually does is run the cheap model, inspect the result, and then run the strong model when the check fails, which means an escalated request pays for both calls. And the 85% figure came from the provider's dashboard aggregated across all traffic, including a high-volume internal health check whose prefix is trivially cacheable, so the hit rate on real user traffic is lower.

The general shape of the mistake is treating a user request as a provider call. In a system with a router, a validator, a retry policy and an evaluation sample, one is not one.

Counting the solid and dotted edges gives 1.35 provider calls per user request. The formula assumed 1.05.

Where the $9,000 is#

Reconciling term by term against the provider's usage export — not the dashboard, the export, which is the artefact the invoice is computed from — produces the following.

The escalated second call is the largest single line, and it is worth being clear about why it is so large relative to its traffic share. An escalated request re-sends the entire 8,000-token prompt to a model priced at 2.5× the base, and the strong model's cache is a separate namespace, so none of that prefix is discounted on the second pass. A 24% escalation rate on those terms costs $9,000 a month against $11,666 for every base call in the system combined. The cascade is not a 15% adjustment to the cost model; it is a second cost model.

The prefix line moves for two reasons at once. The real hit rate on user traffic is 62%, not 85%, and the 38% that miss pay the write premium at 1.25× rather than list at 1.0×, which the model had ignored on the grounds that it looked like a rounding detail. It is a rounding detail per request — $0.0009 — and $2,174 a month.

The retry line is small here and is worth dissecting because it is the term most often assumed to be zero. It has three distinct sources with different costs. A 429 or a 529 — the status codes a provider returns for a rate limit and for an overloaded fleet — that the SDK retries with exponential backoff usually costs nothing, because the request was rejected before any tokens were processed. A request that times out client-side after 60 seconds while the provider completes it costs the full call, and the application never sees the response it paid for. Most expensive is the generation that dies mid-stream — a dropped connection at token 400 of an expected 900 — which is billed for the input in full plus the output produced up to the failure, and is then retried in full. In this system 6% of calls fall into one of those categories, weighted toward the cheap kind, which is why $900 rather than $3,000.

The two terms the team had actually remembered to include, the judge and the embeddings, together come to $103 a month, which is 1.1% of the gap. This is the trap the whole exercise exists to avoid. When a model is short by $9,000 and someone says "it must be the judge and the embeddings", they are naming the terms that were salient rather than the terms that were large, and because those terms are small the explanation cannot be checked by anyone who has not already done the reconciliation. A cost model that absorbs its own error into whichever line the author happened to think of stops being falsifiable, and an unfalsifiable model will keep predicting $14,000 next month too.

The one number worth measuring first#

Given a single measurement — one number, from a system with no cost dashboard — the one that recovers the most accuracy is provider calls per user request. Not tokens, not the hit rate, not the escalation rate individually. The multiplier absorbs escalations, retries, dead generations, judge samples, validators and any agentic loop that runs more than one turn, and it is computable from data you already have: count rows in the provider usage export for a window, divide by the request count in your application logs for the same window.

For this system the multiplier is 1.35 against an assumed 1.05, and multiplying the base model by that ratio alone moves $13,756 to roughly $17,700, closing 43% of the gap without knowing anything about which calls they were. Following it up by asking why it is 1.35 is what finds the escalation rate, and the escalation rate is what finds the additive-versus-replacement error.

Knowing what the multiplier ought to be for a given architecture makes it a diagnostic rather than just a correction factor. A single-shot completion with no router and no validator sits between 1.0 and 1.1, the excess being retries and whatever evaluation sampling runs against production. A cascade of the kind 15.8 described sits between 1.2 and 1.5 depending on the escalation rate, and anything above 1.5 means the router is escalating so often that the cheap tier is mostly a tax. A tool-using agent loop is a different regime entirely, typically 3 to 12 calls per user request, and it is the architecture where the per-request formula is least useful because the multiplier's variance is enormous — a task that resolves in two turns and a task that resolves in eleven cost the same to a formula that does not count turns. Measuring the distribution of the multiplier rather than only its mean is worth doing there, since the p95 request is what sets your rate limits and your worst-case unit economics.

Running the reconciliation itself has a few practical constraints. Use a complete window — a full calendar month or a full week, aligned to the timezone the invoice uses, since a partial day at either end moves the total by several percent on traffic with a daily cycle. Compare against the usage export rather than the console dashboard, which aggregates and rounds differently. And do it more than once: a model that matches October and misses November by 30% is telling you something changed in November, which is a finding, whereas a model checked against a single month can be right by coincidence.

Getting the multiplier honestly requires that every provider call be attributable to a user request, which means passing a correlation identifier through and writing one ledger row per call at the point where the response comes back:

def log_call(span, response, *, request_id: str, purpose: str, model: str):
    u = response.usage
    ledger.write({
        "request_id": request_id,          # the user request, not the API call
        "call_id": response.id,
        "purpose": purpose,                # base | escalation | judge | retry | embed
        "model": model,
        "input_tokens": u.input_tokens,
        "cache_read_tokens": u.cache_read_input_tokens,
        "cache_write_tokens": u.cache_creation_input_tokens,
        "output_tokens": u.output_tokens,
        "stop_reason": response.stop_reason,
        "attempt": span.attempt,
        "route": span.route,
        "ts": span.started_at,
    })

Three fields there do disproportionate work. purpose is what lets you separate the escalation term from the base term without guessing. attempt is what makes retries countable, including the expensive case where attempt 1 streamed 400 tokens before the connection dropped and was billed in full. And splitting input_tokens from cache_read_tokens and cache_write_tokens is what turns the hit rate from a dashboard claim into a per-route measurement — the dashboard's 85% and the user traffic's 62% are both true, of different populations.

The ledger has one failure mode worth pre-empting. A streamed response that the client abandons may never deliver its final usage block, so those calls silently vanish from the ledger while remaining on the invoice. Writing a row on the error path with estimated tokens, flagged as estimated, keeps the count right even when the numbers are approximate, and a count that is right is what the multiplier needs.

What a reconciled model does not give you#

Closing the gap to within a few percent does not make the model predictive of next month. Every term in it is conditional on a traffic mix that moves: the escalation rate is a function of the difficulty distribution of user queries, the hit rate is a function of how many distinct prefixes are live and how they interleave, and the retry rate is a function of the provider's capacity on a given Tuesday. A model reconciled against four months of invoices is a description of those four months. It tells you where the money is now, which is what you need to choose a lever, and it does not tell you what the bill will be after a product launch changes the mix.

Nor does reconciliation identify waste. Every one of the nine thousand dollars above is money the system genuinely spent to do something it was configured to do; there is no error to fix and no refund to claim. The escalation rate might be correct behaviour for the quality bar the product needs. The point of the exercise is that the terms are now visible and ranked, so an argument about whether 24% escalation is worth $9,000 a month is an argument about a real trade rather than a preference.

There is also a scope limit hiding in the framing. Everything so far has treated the feature as the unit of analysis, and a real product does not have one feature. It has a dozen routes, each with its own prefix, its own escalation behaviour and its own volume, and the reconciled model for one of them says nothing about which of the twelve deserves the next engineer-week. Choosing that requires looking at how spend is distributed across routes, which turns out to be far more lopsided than most teams expect before they measure it.