Comparing hosted and owned state

Two quantities get conflated whenever a hosted-state migration is proposed: the number of network hops your service makes, and the number of tokens the provider processes. They are independent, and a migration onto a conversation handle only touches the first. The turn that used to be a database read followed by a model call becomes one round trip carrying a handle and the new user message, and the model still processes the entire conversation as input, because that is what a stateless model does.

This chapter prices that properly and then states a method for measuring it on your own workload, because the size of every effect below depends on conversation length, tool trigger rate and document size. Three numbers come out of the comparison — latency, cost, and replay fidelity, meaning the share of past runs whose exact input you can reconstruct — and the third is the one that does not respond to tuning.

One round trip fewer, the same tokens#

Consider what the database read was actually costing. It fetched a dozen rows from an indexed table on a Postgres instance in the same availability zone as the service, and the p50 on that query is around 4 milliseconds, with a p99 near 20. Removing it removes 4 milliseconds from a request whose total is 2,400. That is the entire latency prize, and the improvement teams observe — on the order of 130 milliseconds — is mostly the disappearance of JSON serialisation of a growing message array on the client side plus a smaller request body on the wire, not the query itself.

Those three figures say more against each other than in sequence. Four milliseconds removed from a request whose total is 2,400 is about one part in six hundred, which is a change no dashboard would ever resolve. The observed saving is more than thirty times the query it supposedly removed, and that ratio rules the query out as the cause on its own, before anything else about the system is known. Whatever produces the saving has to be work that grows with the size of the message array, since that is the only quantity the migration changed on the client side. Serialising a growing array of message objects into JSON is such work, and an indexed read of a dozen rows by conversation id is not — the read's cost is set by the depth of the index rather than by how much text the rows contain.

Now consider what the model call was costing. On turn 12, the model processes the whole conversation — eleven previous exchanges plus the system prompt — as input, and prices it as input. Chapter 2's trace showed this directly: moving to a conversation handle shrank the request payload from about 34 kilobytes to 312 bytes and left input_tokens unchanged. The conversation still grows linearly, turn 12 still re-reads everything before it, and the total input across a twelve-turn conversation is still quadratic in turn count. You removed a database read. You did not remove a token.

The reason the token count did not move is better stated as a chain than as a fact, because everything else in this chapter follows from it. A model retains nothing between calls, so anything it is expected to take into account has to be present in the input of the call it is answering. The full conversation is therefore assembled somewhere before inference begins, whatever the request body happened to contain. Under application-owned state your process does the assembling and ships the result; under hosted state the provider does it from the copy in their own store. The bytes you stopped sending did not stop existing. They stopped travelling, and token counts are taken from the assembled input rather than from the payload.

The quadratic total is a short calculation on this chapter's own turn count. Turn one presents one turn's worth of conversation, turn two presents two, and the pattern continues until turn twelve presents twelve. Adding the whole series gives twelve times thirteen halved, which is seventy-eight turn-loads of text processed in order to deliver twelve replies. What makes the total quadratic rather than linear is that each turn re-presents everything before it, so lengthening a conversation adds a term and enlarges every term after it. Doubling the number of turns therefore multiplies the total input by roughly four rather than by two, and no change in who assembles the array touches that.

The belief this blocks is a natural one, and it is natural because the visible evidence points the wrong way. The two payload figures from chapter 2's trace, about 34 kilobytes down to 312 bytes, are a reduction of roughly a hundredfold, and a hundredfold reduction in anything usually shows up on a bill somewhere. What shrank there was the transport. What is priced is the input to the model, and the provider rebuilt that from material it already held before charging for it.

That accounts for a bill not falling. It does not account for one rising, and there are three contributors, in descending order of size.

Hosted tool invocations are billed on top of the model. Provider-executed search is metered per call, at rates in the region of ten dollars per thousand searches; hosted code execution is billed per container-hour after a free allowance; hosted file search adds both a per-call charge and a storage charge on the vector store. None of these appear in the token accounting at all. An agent that runs one search on 40% of turns, across twelve turns and 30,000 conversations a month, is issuing roughly 144,000 searches — about $1,440 — which on its own can be most of an increase.

That figure is three multiplications, and each one is worth pausing on, because the same shape prices any tool metered per call. One search on 40% of turns, over a twelve-turn conversation, is 40% of twelve, or four point eight searches per conversation. Multiplying a per-turn rate by a turn count is allowed here only because the trigger rate is already an average over turns rather than a property of any particular turn. The result can be checked against the measurements later in this chapter, where the hosted arm records 4.8 hosted tool calls per conversation — the same quantity reached by counting rather than by estimating. Four point eight searches across 30,000 conversations a month is 144,000 searches, and a hundred and forty-four thousand searches priced at ten dollars per thousand is a hundred and forty-four lots of ten dollars, or $1,440. None of that passes through the token accounting, so a team watching only input_tokens and output_tokens meets this line as an unexplained gap between their own model of the spend and the invoice.

Retrieved text becomes permanent context, and this is the compounding one. Chapter 2 measured a hosted search adding about 4,400 tokens to the conversation, and those tokens are re-read as input on every subsequent turn. A search on turn 3 of a twelve-turn conversation is paid for nine more times. The cost of a hosted tool is not the tool call; it is the tool call plus its output multiplied by the number of turns remaining.

Why that output cannot simply be read once and discarded is a property of the transcript rather than a policy anyone chose. A hosted tool result arrives as another entry appended to the conversation, and the conversation is what the provider replays as input on the next call. There is no position in that arrangement from which one entry could be read and thereafter skipped, short of the provider rewriting the transcript. Rewriting the transcript is exactly what server-side compaction does, and it is the same mechanism that later costs this chapter its replay fidelity. The compounding bill and the unreproducible run are two consequences of one property.

The compounding is unconditional in absolute terms and conditional as a comparison, and the two get run together. Those nine repeat charges are real under any arrangement that keeps the retrieved text in the thread, hosted or not. Whether the hosted arm therefore costs more than the owned arm depends on whether the owned arm was retrieving as well. In the workload measured below it was, with the retrieval step client-side rather than hosted, which is why the two input-token figures end up as close as they do. Where a team adds retrieval for the first time in the same commit that adopts hosted tools, all of the retrieved text is new context, and the comparison looks nothing like the one in the table.

Prompt caching moves out of your hands, and this one cuts both ways honestly. Prompt caching is the provider keeping the processed form of a stable prefix — a system prompt, tool schemas, the early turns of a thread — so that re-sending those same tokens on the next call costs a fraction of processing them fresh. Hosted conversations are typically cached server-side by default, and that is a real saving you get without writing anything, since cached input is priced around a tenth of fresh input. What you lose is placement: with an application-owned message array you decide where the cache breakpoints go, keep the volatile parts after them, and can audit cache_read_input_tokens to prove the prefix is stable. With hosted state you get whatever policy the provider applies, and a conversation that compacts mid-thread rewrites its own prefix, which invalidates the cache exactly when the conversation is longest and the invalidation is most expensive.

Measuring it instead of arguing about it#

What settles the question is running both variants on the same inputs, and the method matters more than the numbers because the numbers are yours and the method is transferable.

Take 40 real conversations from production logs, sampled to match the live distribution of turn counts rather than picked for being interesting. Replay each one through both variants: variant A assembles context from the application's own store, variant B passes a conversation handle. Pin the model identifier explicitly in both — not an alias — because an alias moving mid-experiment invalidates everything. Use identical system prompts and identical tool schemas, with the only difference being where state lives and whether the retrieval step is hosted or client-side. Run each conversation three times in each variant, interleaved rather than in blocks, so a slow ten minutes on the provider's side does not land entirely on one arm. Measure latency at the caller, wall-clock, including your own database time for variant A, because that is the number a user experiences. Sum cost from the response's usage fields rather than from the invoice, so you can attribute it.

def cost_cents(resp, rates) -> float:
    u = resp.usage
    fresh  = u.input_tokens - getattr(u, "cached_input_tokens", 0)
    total  = fresh * rates.input + u.output_tokens * rates.output
    total += getattr(u, "cached_input_tokens", 0) * rates.cached_input
    tools  = getattr(u, "server_tool_use", None)
    if tools:                                  # hosted calls are not tokens
        total += tools.web_search_requests * rates.per_search
        total += tools.file_search_requests * rates.per_file_search
    return total / 1_000.0

The server_tool_use block is the line that makes attribution possible, and it is worth checking early whether your provider populates it for the tools you have enabled — some counters exist and are documented as always zero. Where the counter is missing, count the tool-call items in the response yourself; the trace from chapter 2 already has them.

The controls in that method each defend against a specific way the comparison comes out wrong, and the list reads as fussy until the failures are named. Sampling to the live distribution of turn counts matters because every cost in this chapter grows with conversation length, so a sample of unusually interesting conversations — which in practice means unusually long ones — inflates both arms and does not inflate them equally. Interleaving defends against the provider's own capacity moving during the run, which it does on a timescale of minutes. Identical prompts and schemas leave the location of state as the only free variable, and that variable is the whole claim under test. Measuring at the caller rather than at the provider is what keeps variant A honest, since its database time is time a user waits and a provider-side number would not contain it.

Pinning the model identifier deserves the separate warning it gets, and it also deserves a limit on what it buys. It is the one term in this system with an identifier you can write down, and that is precisely what makes it feel like control of the experiment. Everything else that can move underneath a hosted run — how the provider assembles the input, what the retrieval corpus held this week, what the execution environment contained — has no identifier you hold. Pinning the model pins one term and leaves the rest free, so the question to put to a result like this is which of the remaining terms was recorded rather than assumed.

Three runs of 40 conversations is 120 samples per arm, and that number is not arbitrary. Provider latency on a workflow like this has a standard deviation around 600 milliseconds per turn, so a single pass would put a 130-millisecond difference well inside the noise; at 120 samples per arm the standard error on the median is small enough that the direction of the difference is stable across re-runs, which is all the claim needs to be. The same reasoning is why the p95 belongs in the table and a mean does not — a distribution with a heavy right tail has a mean that mostly reports how many slow calls happened to land in the sample. Report the median and a tail percentile, state the sample size, and say which hours of which day the runs happened in, because provider latency has a time-of-day shape and an experiment run at 04:00 is not a claim about 14:00.

The step underneath that sample-size claim is the relationship between a spread and an average, and it is worth writing out, because it is what fixes the sample size on any comparison of this kind. The difference being hunted is 130 milliseconds and the spread of a single turn's latency is around 600 milliseconds, so the effect is roughly a fifth of the variation in one observation. No pair of single observations can show a difference that small; the two arms trade places from run to run purely on which requests happened to be slow. What averaging buys is that the spread of an average falls with the square root of the number of observations averaged, rather than with the number itself. Tripling 40 conversations to 120 samples per arm therefore shrinks the uncertainty by the square root of three, which is a little under a factor of two. The count is the only lever you hold and it is a weak one, which is why the defensible claim from an experiment like this is the direction of the difference rather than its size.

Replay fidelity is the third measurement and needs a definition to be a measurement at all. Use this one: take the recorded run, attempt to reconstruct the exact input to its final turn, and hash both. A run replays if the reconstructed input is byte-identical to the captured one. Score it as a percentage of the 40 conversations, and score it twice — once immediately, once at day 35, after the shortest retention window in the system has expired.

Three choices inside that definition are doing the work. Comparing hashes rather than reading the two inputs side by side is what makes the test cheap and what makes it impossible to pass by accident, since one differing byte anywhere changes the digest and there is no partial credit to argue over. Taking the final turn rather than the first is what makes it strict, because the final turn's input contains every earlier turn plus anything a tool wrote into the thread, so it is the reconstruction with the most ways to fail. Scoring it twice separates two failures that are indistinguishable in a single reading: material the provider never exposed verbatim in the first place, and material that was exposed and has since aged out of a retention window.

Results from a twelve-turn onboarding workflow, 40 conversations, three runs each:

The p95 row is the one that changes how people feel about the p50 row. Variant B's median improved by 130 milliseconds and its tail more than doubled, because every turn that triggers a hosted search carries a retrieval round trip inside the model call — a system whose queue depth you cannot see, whose timeout you did not set, and whose slow days land entirely inside the single latency number your service reports. If your service level objective is written on the p95, this migration made it worse while making the dashboard headline better.

Two rows moving in opposite directions from one change is less strange than it looks once the trigger rate is set beside the percentiles. A hosted search runs on 40% of turns, which is under half of them, so the turn sitting at the median is a turn with no search in it. That turn did get faster, because it lost an application round trip and gained nothing. The turn sitting at the 95th percentile is well inside the minority that did trigger a search, and carries the retrieval wait described above. One change therefore improves the statistic computed over turns that never touch the new capability and degrades the statistic computed over turns that do, and reporting only the first is incomplete rather than untrue.

The input token rows are close, and the 3% difference is not the interesting part; the interesting part is that they are close. Everything the provider assembles on their side, you were assembling on yours, and the arithmetic is identical. The cached share improving from 61% to 74% is a genuine win from server-side caching, and it is the reason the cost gap is 27% rather than the 40% the tool charges alone would imply.

What that cached share is worth can be worked out from the two figures in the table and the pricing ratio stated earlier, and the working is short. Cached input is priced at around a tenth of fresh input, so any block of input can be scored in fresh-token equivalents by counting the fresh tokens at their own weight and the cached ones at a tenth of it. In variant A, sixty-one of every hundred input tokens are cached and thirty-nine are fresh, giving thirty-nine plus six and a tenth, or about forty-five fresh-token equivalents per hundred. In variant B, seventy-four are cached and twenty-six fresh, giving twenty-six plus seven and four tenths, or about thirty-three. Variant B is paying roughly three quarters as much per input token processed as variant A is.

That conversion is legitimate only because both arms are pinned to the same model identifier. Rates differ between models, and a ratio of equivalents is a ratio of money only when the price of a fresh token is the same on both sides of it. Set against variant B processing about 3% more input tokens, a per-token cost of roughly three quarters puts its model-token spend below variant A's rather than above it. A hosted tool charge can therefore be larger than the increase it appears inside, which is the arithmetic the scenario below ran into.

What the measurement does not settle#

It is a specific result about a specific workflow and it does not generalise upward or downward on its own. Three short conversational turns with no hosted tools and heavy prefix reuse can come out cheaper on hosted state, because the caching is automatic and there is no retrieved text to compound. A workflow whose retrieval you were doing yourself, badly, with a hand-rolled vector store you also pay to run, can come out cheaper still once that operational cost is in the comparison — and it is not in the table above, because variant A's retrieval was already free in this workload. Comparing only the model bill flatters whichever side has more infrastructure hidden behind it.

There is one more caveat about the cost column, which is that summing usage fields and reading the invoice are two different exercises and they will not agree exactly. The invoice aggregates across every workload sharing the account, applies whatever contracted discounts you have, and prices tool usage on its own line with its own rounding. Usage fields are list-priced and per-request. Use the fields for attribution — they are the only way to say which turn or which tool caused what — and reconcile against the invoice at the account level, monthly, rather than expecting the two to line up per conversation.

The same care is needed one level up, when an increase is being explained rather than measured. A migration usually adds a line to the bill and changes several existing ones at the same time, and the instinct is to attribute the whole rise to the new line, because the new line is the visible thing that was not there before. That reasoning fails whenever the change also moved an existing line, which server-side caching does by construction. A new charge can exceed the total increase without anything being wrong, and an explanation built by summing only the additions will overshoot and send somebody looking for a bug that is not there. The discipline is that attributed components have to reconcile to the net figure, with reductions carried at their own sign.

Nor does the measurement decide anything by itself. A 27% cost increase and a doubled p95, traded for four hundred fewer lines and a retrieval system nobody has to operate, is a reasonable trade for plenty of teams. It stops being reasonable when the replay row is a compliance obligation rather than an inconvenience, and that is the point of measuring all three together: the cost and latency numbers are recoverable positions you can tune, and the fidelity number is a property of the architecture that no amount of tuning moves.

The one thing the measurement genuinely rules out is the original expectation. Nobody should port a workflow onto hosted conversation state on the theory that fewer application round trips means a smaller bill. The bill is a function of tokens and tool calls; the tokens are unchanged, and the tool calls are new. If the case for the migration is deleted code, an operational burden removed, or a retrieval system nobody wants to run, it can be an excellent case — it is simply a different one, and stating it accurately is what lets the next team decide whether their workload shares the properties that made it true here.

There is also a quiet assumption running through this entire comparison that needs to be made explicit before the topic goes further. Both variants issue refunds, update addresses, and cancel orders through code in your process, under credentials you chose, at a call site you own — the hosted tools only search and retrieve. That assumption is doing an enormous amount of work, and the next chapter walks the other kind of tool loop once, slowly, with a privileged action at the end of it, because the identity question that arrangement answers is the one that chapter 6 will show hosted retrieval attacking directly.