Model spend is one line item, not the cost per task

The finance business partner asked one question about the model: "Is $0.31 the cost of serving a session, or the cost of the API calls in a session that worked?" The engineering answer was that it was the second thing, measured accurately against the provider invoice for sessions that reached a final answer. Her follow-up was whether the company had any sessions that did not reach a final answer, and whether anyone looked at them, and what happened when a customer complained about one.

That conversation is the difference between a metered number and a unit cost. Chapters 1 and 2 produced a good metered number by decomposing input and pricing a transcript. It is the denominator that is wrong: $0.31 is divided by successful sessions while the cost of running the product is divided by everything that happened, and the two differ by more than most teams guess before they measure.

Everything the invoice covers is not everything the session costs#

The agent product handles about 190,000 completed sessions a month. Working per thousand completed sessions makes the arithmetic legible, because the outcomes that are not completions have to be expressed as a rate against the ones that are.

For every 1,000 completed sessions, the platform starts about 1,280. Roughly 195 are abandoned, typically after two or three turns, which still costs about $0.063 each in model spend before the user closes the tab. Another 85 fail — a tool that never returns, a loop the runtime kills at turn 20, a malformed response the parser rejects — and those average $0.14 because failures skew towards long sessions rather than short ones. On top of that, 4.2% of individual turns are retried after a transient error, which adds about 3.1% to the spend of the sessions that succeed.

Model spend is 39% of the real number. The single largest non-model line is human review: 4.1% of sessions escalate to an operations reviewer who spends about six and a half minutes on each, and at a fully loaded $46 an hour that is $4.98 per escalation, or $0.204 spread across every session. Support contacts run second at $0.120, from a 1.6% contact rate and an eleven-minute average handle time. Neither of those appears on any invoice from a model provider, and both scale linearly with sessions, which is what makes them cost of goods — spend incurred only because a session was served, as opposed to spend the company would carry anyway — rather than overhead.

Embedding cost is on the table mostly to be dismissed. It comes to $0.002 per session, because the corpus is embedded once on ingest and query embeddings are a few hundred tokens at rates two orders of magnitude below generation. Teams routinely spend a planning cycle on embedding cost and none on escalation rate, and the ratio between those two lines — roughly 100 to 1 — is the argument against doing that again.

The path from an event to a unit cost#

Every line in that table is a join between an event stream you already have and a rate you have to establish once. Drawing the shape makes clear where the estimates enter, which matters because those are the lines a reviewer will push on.

The metered branches on the left need no estimation, because each request carries usage fields and a session identifier. The two conditional branches need a rate and a unit cost, both of which are measurable but neither of which updates itself; escalation rate in particular moves whenever the model, the prompt or the customer mix changes, so a unit cost quoted from a rate measured six months ago is a number about a product that no longer exists.

Making those branches self-updating is mostly a matter of putting the session identifier in two places it is usually missing. The reviewer tool has to record which session it opened and how long it stayed open, which turns escalation cost from a headcount estimate into a metered quantity. The support desk has to carry the session identifier on tickets raised from the product, which most helpdesk integrations support and almost nobody configures, and which replaces the text-matching heuristic that currently attributes tickets to the feature. Both are a few days of work, and both convert the two largest non-model lines from estimates into meters, which matters disproportionately when someone starts stress-testing the margin.

There is a subtler failure hiding in the abandoned branch. Sessions that end without a final answer are not uniformly cheap: a user who abandons because the assistant is slow leaves after two turns, and a user who abandons because the assistant is confidently wrong often leaves after twelve, having tried repeatedly to correct it. Averaging them at $0.063 is fine for the total and misleading if you use it to reason about the cost of quality problems, because the expensive abandonments are exactly the ones a better model would eliminate.

The denominator is a product decision#

"Cost per task" needs a task, and on an agent product the obvious candidate — a session — is not always the unit anyone cares about. A user with a shipment exception to resolve may take three sessions to get there, because the first one hit a tool error, the second returned an answer they did not trust, and the third worked. Divided by sessions, the product costs $0.875. Divided by resolved exceptions, at 1.34 sessions per resolution, it costs $1.17.

Both numbers are correct and they answer different questions. The per-session number is the one that reconciles against infrastructure and is the right basis for capacity planning, because sessions are what the system serves. The per-outcome number is the one that belongs in any comparison with the alternative, because the alternative — a human resolving the exception in nine minutes — is priced per outcome, not per attempt. A team that quotes $0.875 against a $6.90 manual cost is claiming a better ratio than it has, and the correction is not small: 1.34 sessions per outcome is a 34% understatement.

The choice of denominator also decides what improvements look like. Reducing retries improves cost per outcome and leaves cost per session roughly unchanged, so a team measured on the session number will not be credited for the work and may not do it. Pick the denominator that matches the decision being made, write it into the metric name so nobody has to ask, and keep both if the product has both audiences.

Splitting shared cost evenly is how tenants get subsidised#

The $31,000 a month of shared platform cost — the orchestration cluster, the API gateway, observability, and the on-call rotation that exists because of this product — has to land on somebody. The default is to divide it by tenant count, which for 42 tenants gives $738 each, and that number is wrong for every tenant simultaneously because tenant sizes span a factor of 400 in sessions.

Allocation needs a driver that correlates with the cost being allocated. Orchestration compute and gateway traffic scale with sessions, vector storage scales with indexed documents, and on-call load scales with incidents, which in practice correlates with sessions too. Allocating each pool by its own driver takes about twenty lines of SQL and turns a per-tenant cost from a fiction into something you can act on.

with tenant_usage as (
  select tenant_id,
         count(*) filter (where outcome = 'completed')      as completed,
         count(*)                                            as started,
         sum(model_cost_usd)                                 as model_cost,
         sum(escalation_minutes) / 60.0 * 46.0               as review_cost,
         sum(support_minutes)    / 60.0 * 41.0               as support_cost
  from sessions
  where started_at >= date_trunc('month', now())
  group by tenant_id
),
storage as (
  select tenant_id, sum(chunk_count) * 0.0000174 as vector_cost  -- $/chunk/month
  from document_chunks group by tenant_id
),
totals as (select sum(started) as all_started from tenant_usage)
select u.tenant_id,
       u.completed,
       u.model_cost,
       u.review_cost,
       u.support_cost,
       s.vector_cost,
       31000.0 * u.started / t.all_started                  as platform_alloc,
       (u.model_cost + u.review_cost + u.support_cost
        + s.vector_cost
        + 31000.0 * u.started / t.all_started)              as total_cost,
       (u.model_cost + u.review_cost + u.support_cost
        + s.vector_cost
        + 31000.0 * u.started / t.all_started) / u.completed as cost_per_session
from tenant_usage u
join storage s using (tenant_id)
cross join totals t
order by cost_per_session desc;

Running that query for the first time on this product produced a fleet average of $0.875 and a spread from $0.41 to $6.20 per completed session. The tenant at $6.20 had a 19% escalation rate against a fleet average of 4.1%, because its users were asking the agent to do something adjacent to what it was built for and a reviewer was quietly fixing the output every time. Under an even split of shared cost and an average escalation rate, that tenant appeared to cost $0.88 like everybody else, which is the precise mechanism by which a healthy-looking average conceals a subsidised account.

Which costs belong, and which do not#

Not everything spent on the feature belongs in its unit cost. The engineers who built it, the eval experiments that chose the model, the one-off migration that reindexed the corpus, and the design work on the next version are all real spend and none of them scale with a session, so putting them in cost per task makes the number move for reasons unrelated to serving anyone. They belong in operating expense, and if the company wants them in a margin figure it belongs in the contribution-to-fixed-cost analysis, not in COGS — cost of goods sold, the accounting line that holds only what scales with units served. The test is mechanical: if serving one more session tomorrow changes the amount, it is a unit cost; if it does not, it is overhead being amortised, and amortised overhead should be labelled as such.

That test also produces two legitimate numbers rather than one, and confusing them causes arguments. The marginal cost of a session here is about $0.71 — everything except the allocated platform pool and the annotation programme, which do not move if volume changes by 10%. The fully loaded cost is $0.875. The marginal number is the right input to a question about whether to accept an extra 50,000 sessions from an existing customer; the fully loaded number is the right input to a question about whether the product line makes money. A team that carries only one of them will use it for both and be wrong in one of the two.

The last thing this per-session number is not is a margin. It is a cost, computed at current volume, current rates, current escalation behaviour and list prices, against a denominator of completed sessions. What happens to it when volume triples, when a committed-spend discount lands, when reserved capacity sits idle overnight, and when a finance team asks which parts of it are metered and which are guesses, is a different exercise, and it is where the number stops being an engineering artefact.