Spend reconciliation and the residual

Everything in this topic so far assumes a call returns a usage object your wrapper reads and writes down. Reconciliation is the practice of checking that assumption against the only independent source available: the provider's own reported totals, pulled on a schedule for a closed day and one model, with the difference recorded as a number rather than left as a suspicion. That difference is the residual, and on a working system it is small, stable and never zero.

A reconciliation is a subtraction, and a subtraction between two numbers means something only once both are known to be measuring the same quantity. Both sides here are meant to be the money spent on one model on one day. Making them genuinely that takes four agreements no comparison enforces on its own: the same set of calls on each side, the same day boundary in the same timezone, the same price basis, and the same currency. Every one of the four can be violated in silence, and the residual is then the sum of the real instrumentation gap and whatever the violations contributed to it.

The reason reconciliation needs its own chapter is the direction of the error. A row that was never written contributes nothing to any aggregate, so instrumentation gaps are always understatements — the dashboard is not wrong about the requests it knows about, it is wrong about which requests exist. That is the failure mode where a cost-per-request tile sits flat for a month and finance arrives with an invoice up 35%.

That direction follows from what an aggregate is. A sum() is defined over the rows that are present, and a row nobody wrote has no representation in the table at all — not a null, not a zero, nothing a query could count. So the arithmetic over a ledger can never come out too high on account of an absence, and the only symptom an absence has is a total that is too low. The same fact says the gap cannot be found from inside, because every query you are able to write is a question about rows that exist. Detecting it needs a number produced by something that was not your own instrumentation, and that requirement is the whole justification for the machinery below.

Four ways a billed call leaves nothing behind#

A client-side timeout abandons a generation that keeps going. An HTTP client configured with a 60-second timeout raises when a long synthesis at effort: "xhigh" runs past it, the request is abandoned, and the server keeps decoding until it reaches a stop condition. Those tokens are produced and billed. On one system, 0.4% of research requests exceeded the client timeout — small enough to look like noise on an error dashboard, and misleading, because the requests that time out are by definition the longest ones. That 0.4% of requests accounted for 3.1% of output tokens on the feature, and output tokens are the expensive leg.

Those two figures are the same fact stated twice, and dividing one by the other is what makes it usable: a class of requests that is 0.4% of the count and 3.1% of the output carries roughly eight times the output of an average request. The ratio is not a quirk of that system. Timing out is a consequence of running long, running long on a model call means producing more tokens, so the selection is on precisely the variable the cost is proportional to. No error rate in this chapter can be called small until somebody has said what it selects for.

A cancelled stream bills what was already produced. Streaming delivers a response token by token over a connection held open for the whole generation, which is what lets text appear as it is written and also what lets the caller walk away halfway through. A user navigates away mid-answer and the connection closes. Aborting the stream stops future tokens; it does not refund the ones already decoded. If the wrapper writes its cost row from the return value of the streaming call, a cancelled stream has no return value and writes nothing, while the provider has a perfectly good record of 900 output tokens.

The SDK retries without telling your wrapper. The Anthropic SDKs default to max_retries=2, retrying 408, 409, 429 and 5xx responses plus connection errors, with exponential backoff. An instrumented wrapper calls messages.create once and gets one response back; the API may have served three attempts. Most retried attempts were rejections that produced no output and cost nothing — but timeouts are retried too, and a timeout is precisely the case where the first attempt was generating tokens when the client gave up. Wall-clock on such a call can reach timeout × (max_retries + 1), which is three full generations billed against one row.

A stream reports usage only at the end. Token counts arrive in the terminal message_delta and message_stop events, not incrementally. A stream that errors after 40 seconds of output has emitted no usage object at all, so a wrapper reading usage from the final message writes either nothing or a row with null counts.

Three of the six terminal states are billed and carry no usage object. A wrapper written as resp = await client.messages.create(...) followed by write_row(resp.usage) covers exactly two of them.

Writing a row for a call that produced no usage#

The shape that covers the rest is to write the row from a finally block, with an explicit status column, and to count output events locally while the stream is running so an interrupted stream still has a floor estimate.

async def call_model_streamed(client, ctx, **kwargs):
    status, usage, produced = "unknown", None, 0
    try:
        async with client.messages.stream(**kwargs) as stream:
            async for event in stream:
                if event.type == "content_block_delta":
                    produced += 1          # local floor on output tokens
            usage = (await stream.get_final_message()).usage
            status = "complete"
    except asyncio.TimeoutError:
        status = "client_timeout"          # server likely kept decoding
    except asyncio.CancelledError:
        status = "cancelled"
        raise
    except Exception:
        status = "stream_error"
    finally:
        write_cost_row(
            ctx, kwargs["model"],
            usage=usage,
            usage_status=status,
            output_tokens_observed=produced,   # used when usage is null
        )

Choosing finally over a longer list of except clauses is the load-bearing part of that shape, and it is a claim about enumeration rather than about style. An except block runs for the exits you thought to name, and the reason rows go missing is the exits nobody thought to name. A finally block runs on all of them: a clean return, a named exception, an unnamed one, and a cancellation that is being re-raised so it can carry on upward. The raise in the CancelledError branch is worth reading twice for that reason. The row is still written, because finally runs before the exception resumes its journey, so a cancelled call leaves a record behind and an intact cancellation both.

The usage_status column is what makes the gap countable instead of merely present. A row with usage_status = 'client_timeout' and a null usage object is an honest statement — this call happened, it was billed, and we do not know for how much — and a query counting those rows per feature per day is the fastest diagnostic anyone will run when the residual moves. The locally counted delta events give a lower bound rather than a true count, since a delta is not a token and the server may have continued past the client's last received event, but a floor that is 70% right beats a null that is 0% right.

The SDK's internal retries are the one class this does not catch, because they happen below the wrapper. The fix is to move retry control up: set max_retries=0 on the client and let the instrumented wrapper own the loop, so every attempt writes its own row with its own attempt number. That is a small loss of convenience and the only way the attempt count on a row can be true.

The residual as a first-class number#

None of the above gets a ledger to exact, which is why the comparison against the provider runs on a schedule and stores its answer.

-- Nightly: compare our summed rows against the provider's per-model total
INSERT INTO spend_reconciliation
  (day, model, ledger_usd, provider_usd, residual_usd, residual_pct)
SELECT
  d.day,
  d.model,
  d.ledger_usd,
  p.provider_usd,
  p.provider_usd - d.ledger_usd,
  (p.provider_usd - d.ledger_usd) / nullif(p.provider_usd, 0)
FROM (
  SELECT date_trunc('day', occurred_at) AS day, model, sum(cost_usd) AS ledger_usd
  FROM   llm_calls
  GROUP  BY 1, 2
) d
JOIN provider_daily_usage p USING (day, model);

The residual is then carried as an explicit unattributed_spend line on every dashboard reporting spend by feature or tenant, sitting alongside the named buckets rather than being quietly dropped. Showing it changes the conversation in a specific way: a chart where "unattributed" is 1.2% invites nobody to argue, and the same chart where it has grown to 18% over two weeks makes the instrumentation gap a visible problem with an owner, at the moment it appears rather than at month end.

A healthy residual on a well-instrumented system runs roughly 0.5% to 3% of provider spend. It is never zero, for reasons that are structural rather than fixable. Provider usage reporting lags — same-day figures are provisional and settle over 24 to 48 hours, so reconciling today against today produces noise, and the job should compare against a day that closed at least 48 hours ago. Rounding differs: a ledger computes per call at full precision while the provider aggregates, and Managed Agents' own list-cost figure is documented as rounded to the nearest cent while enforcement compares exact amounts. And the timezone boundary between your day and the provider's is worth checking once rather than debugging quarterly.

The timezone deserves that check because date_trunc('day', occurred_at) silently uses whatever timezone the database session happens to be set to, while the provider's day is whatever its own reporting uses. Nothing errors when the two disagree. Both sides go on producing clean-looking daily totals, and the difference between two adjacent hours of traffic lands in the residual instead of anywhere visible. On flat overnight traffic that difference is small enough to pass for rounding. On a workload with a business-hours peak it is not.

Two properties of a stored residual carry more diagnostic weight than its size, and both exist only because it is stored rather than eyeballed. The first is its sign. Positive, with the provider reporting more than the ledger, is the expected direction, because an understatement is the only error a missing row can cause. Negative is a different animal: the ledger is claiming spend the provider does not report, and no absence of rows can produce that. Something has to have been added rather than lost — a row written twice by two layers of instrumentation, a row priced against the wrong rate, or the two sides not describing the same quantity at all. A ledger priced at list against a figure net of a negotiated discount comes out negative by roughly the discount, and it does so every single day without ever moving.

The second property is stability, and it separates three situations that share a number. A residual holding at much the same percentage day after day is a constant of the system rather than a fault in it — rounding, reporting lag, the same few unmeasured call shapes recurring at the same rate. One drifting upward over a fortnight is instrumentation coming apart, which is a different investigation with a different owner. One alternating in sign from day to day while averaging close to nothing is neither, and it is the signature of the boundary problem above: a misaligned day takes a slice of traffic off one day's residual and adds it to the next day's, so the error has to reverse itself every time. That pattern is unmistakable on ninety days of stored rows and completely invisible in any single day's figure.

Alert on the ratio rather than the dollars, with a floor so a quiet Sunday does not page anyone: residual above 5% of provider spend for two consecutive settled days, on any model with more than $200 of daily spend. Both thresholds matter — the percentage catches instrumentation breakage during growth, and the dollar floor stops a $3 test model generating noise.

Getting provider-side totals to compare against is the one piece of this that is not code you write. The usage and cost reporting in a provider's console, or its administrative API where one exists, gives spend broken down by day, model and API key or workspace, and that is the full extent of the dimensionality available. There is no feature dimension on an invoice and no tenant dimension, which is why "just filter the invoice down to draft-reply" is a reconciliation that cannot be performed: the proposal describes an operation the data does not support, and pursuing it ends the investigation at precisely the point where the gap in your own accounting was about to be found. Reconcile at the granularity the provider offers — day and model, per key — and let your own ledger carry every dimension below that.

Comparing a sum of priced rows against an invoice is the version of this that feels most conclusive and is the least fair comparison on offer. The ledger holds list-priced token spend for the calls it made. An invoice is a billing document, and it can carry a negotiated discount, credits applied at the account level, tax, and non-token lines such as per-search metering or session running time, none of which any token counter will ever produce. Those differences are real money and none of them is an instrumentation gap. The scheduled job should therefore run against the provider's usage and cost reporting for a closed day and one model, where both sides are the same quantity, and the invoice is best treated as the thing those reconciled figures eventually have to be explained to.

A residual near zero is the reading that most deserves suspicion, and it is tempting for an honest reason: two independent measurements agreeing is normally strong evidence that both are right. The word carrying the weight there is independent. Errors also cancel, and an overcount on one model against an undercount on another vanishes in a company-wide total — which is why the comparison is run per model and per day rather than once a month against a single figure. A residual that sits at nothing and never moves at all usually means the two sides were never independent to begin with, because the provider-side number came from an export your own ledger feeds. That check compares a number against itself and will agree forever.

Reconciliation detects and does not attribute#

A residual of $27,720 says the ledger is missing that much and says nothing about which tenant, feature or user incurred it, because the whole reason the money is in the residual is that the row identifying those things was never written. It cannot be allocated proportionally across features either, however tempting the arithmetic looks — the missing spend is systematically concentrated in the slow, long, failure-prone calls, which are not distributed like ordinary traffic. Allocating it pro rata would smear the expensive tail evenly across cheap features and produce a chart that is wrong everywhere instead of honest in one place.

It is also not a substitute for per-call instrumentation. A team that reconciles well and instruments badly gets an accurate total and no ability to act on it, which is the same position as reading the invoice. Reconciliation exists to bound the error in the attribution, and the attribution is where the decisions get made.

One more nuance belongs on the row while you are here. When a request runs under server-side fallbacks, usage.iterations is the per-attempt source of truth and the top-level usage covers only the attempt that produced the returned message — a declined-before-output attempt is reported there and not billed, while a mid-stream decline bills the streamed partial at the original model's rates before the fallback model's attempt begins. A wrapper reading only the top-level usage under-reports a fallback turn by exactly the partial it discarded.

The pairing to take from this chapter is a status column and a stored residual. The status column turns silence into a countable state, so a call that was billed and not measured is still a row somebody can group by. The residual turns the remaining error into a number with a trend, which is what makes drift visible on the day it starts rather than in a quarterly surprise — a spend_reconciliation table holding ninety days of daily residuals answers "when did this begin", which is the question that identifies the deploy responsible, and a console page you open during an incident does not.

At this point the rows exist, they are priced, they are grouped, they are split by category, and the gap between them and the invoice is measured. What does not yet exist is a place to put them where a question about spend is a query somebody writes in thirty seconds rather than an afternoon of joins across application logs.