The cost multiplier lives on the input side

A design review that splits one tool-using agent into five almost always carries a cost estimate, and the estimate is almost always built out of the call count: five model calls where there was one, so five times the money at the very worst. A payables service that made exactly that move measured something else on its next invoice — about eleven times its previous bill, against a predicted five. Nothing had gone wrong with the multiplication on the slide. The call count is simply not the thing a model charges for.

Both of the designs in that story have names, and the whole topic turns on the difference between them. An agent, here, is a language model running in a loop: it is given a task and a set of tools, it reads the task, decides to call a tool, receives the tool's result back into the same conversation, and keeps deciding and calling until it judges the work finished. A single-agent loop does the entire job that way, inside one conversation that grows turn by turn. A multi-agent system splits the same job across several agents, each with its own instructions, its own tools and its own separate conversation, handing work to one another under an orchestrator — the controller that decides which agent runs next and what to give it. The payables service moved from the first shape to the second, and the invoice is the record of what that move cost.

Teams build the second shape for reasons worth taking seriously. A single conversation that has to plan, extract, validate, price and write carries the instructions for all five jobs at once, and past a certain size a model follows five sets of instructions less reliably than five narrow agents each follow one. Separate agents can also be owned by separate teams, deployed on their own schedule, given their own credentials, and watched separately when something goes wrong. Most people meet the design in one of three places: a framework tutorial whose opening example is a supervisor with two workers, a design review that proposes splitting a working loop into stages, or a document pipeline that grew one stage at a time until it had six. What follows is the arithmetic that decides whether any of that was worth the money, starting with the bill.

The number that matters here is not the call count. Both designs make many calls; what differs is how much material each call has to re-read, and how much of that re-reading the model is allowed to bill at the cached rate. Once you measure those two things instead of counting invocations, the 11.3x on the invoice stops being a surprise and becomes a prediction you could have made in the design review.

Why counting calls is the natural mistake#

The slide's arithmetic came from a habit that is correct in the system most backend engineers have spent their careers on. In a request-scoped microservice architecture, the unit of cost really is the call: each service is stateless, receives a small payload, does its work, and returns. Splitting one service into five gives you five payloads that are each smaller than the original, and the marginal cost of the split is dominated by network hops and serialisation, both of which are cheap and both of which scale with the number of calls. Counting calls is the right instinct there, and it transfers badly.

A model call is not stateless and its payload is not small. The cost is dominated by how much text the model has to read, and splitting one agent into five does not divide that text between them — it multiplies it, because each of the five needs enough context to do its job, and "enough context" overlaps heavily across the five. The single conversation is the case where the overlap is free, because material read on turn two does not have to be re-presented on turn three; it is already there, and after the first request it is cached.

The chain from that property to the bill is short enough to check link by link. A model carries nothing between conversations, so everything an agent needs to know has to be present in the request it is sent. The second stage's job overlaps with the first stage's: it needs the schema the first stage worked from, the document the first stage read, and now the first stage's answer as well. The overlapping part therefore has to be sent twice, once into each conversation, and there is no mechanism by which sending it the first time could satisfy the second. Repeat that across five agents and the shared material is not split between them, it is copied into each of them, so the total input grows with the number of stages rather than staying fixed and being divided among them. A tool call can fetch a fact mid-run, and the way it delivers that fact is by putting it into the same conversation, which is another way of paying for it in tokens.

Put differently: in a microservice split, the shared state lives in a database and each service reads the slice it needs. In an agent split, the shared state lives in the prompt and each agent reads most of it. The database has an index; the prompt does not.

That reframing has a limit, and the limit is where the wrong conclusion forms. A microservice split genuinely does shrink each service's payload, and the reason it can is that the shared state sits in a store any service may query at the moment it needs a row. What a model reads has to be assembled before the call is made, so the question "which slice does this component need" is answered by a human writing a prompt rather than by a query at run time, and the safe answer to that question is always more than the minimum. Where a design does give its agents a shared store to read from on demand, the microservice arithmetic starts to apply again in proportion to how much of the context moves into it. The payables pipeline has no such store.

Neither design makes one call#

The loop is a single conversation that runs seven turns on a typical 22-page invoice with 340 line items. Turn one reads the schema and plans; turns two through four extract line items in batches, each one appending its tool results to the same conversation; turn five checks the extracted lines against the supplier's contract terms; turn six recomputes totals with a calculator tool; turn seven writes the payable record. Seven model calls, one growing context.

The pipeline makes thirty. The orchestrator makes six (one to kick off, one after each agent reports), and the five agents make twenty-four between them — the extractor alone runs eight turns, because it works page by page and each page's tool result lands in its own conversation. So the honest call-count ratio is 30 to 7, or 4.3x, which is roughly what the slide claimed and is still not the number on the invoice.

The material each call carries is where the two designs diverge. Four blocks of text are involved in every run:

• the system prompt and tool definitions, which differ per agent but are fixed per deployment
• the extraction schema and the validation policy — 5,600 and 8,200 tokens, identical for every invoice
• the supplier's contract terms, 5,900 tokens, identical for every invoice from that supplier
• the invoice text itself, 11,200 tokens, new every run

The loop reads all four once, at turn one, and every subsequent turn is a continuation of that same conversation. The pipeline reads the schema in three agents, the contract terms in four, and the invoice text in four, because each agent is a separate conversation with a separate system prompt and no memory of what the previous one was given.

Counting those blocks gives the shape of the input ratio before any measurement is taken. Call the schema and policy block S, the contract terms C, the invoice text D, and an agent's own system prompt with its tool definitions P. The loop's opening turn carries S plus C plus D plus P, one of each, and every later turn continues that same conversation rather than restating it. The pipeline's opening turns carry S in three of the agents, C in four, D in four and P in all five, so its total before any agent has taken a second turn is a sum in which S appears three times, C four times, D four times and P five times. Nothing in that expression is divided by the number of agents. Each term is either unchanged or multiplied by however many agents need it, and that is the whole of the mechanism the slide missed.

Where the tokens actually go#

Measured over 200 real invoices, holding the model at claude-opus-5 and the effort setting — the dial that says how much reasoning the model may spend before it answers — at high on both arms:

That table is easier to read across than down, because the quantities that mean something are the gaps between the columns rather than the levels inside them. The gap between the first and second columns is what the split cost. The gap between the second and third is what a fortnight of billing work recovered: identical tokens, identical reading, charged differently. The gap that is still open between the third column and the first is the part no billing work can close, because it is a difference in how many times the same material is read rather than in the rate it is read at.

Raw input is 2.8x, output is 2.7x, and the bill is 10.8x. The gap between those ratios is the whole chapter. At Opus 5 rates — $5 per million input tokens, $25 per million output — a cache read costs a tenth of a fresh input token and a five-minute cache write costs 1.25x. The loop pays the read rate on 89% of its input. The pipeline, as shipped, paid the full rate on all of it.

Those three ratios collapse into one identity, and the identity is worth having because it says which of them to attack. Cost per successful document is the input tokens times an effective input rate, plus the output tokens times the output rate, all divided by the share of runs that deliver a document. The effective input rate is not the list rate: it is the list rate weighted by how the tokens split between fresh input at full price, cache reads at a tenth, and cache writes at 1.25 times. Divide one arm's expression by the other's and the headline multiplier factors into four terms that can each be measured on their own — how many more tokens, at what worse mix of rates, with what worse ratio of output to input, over what worse success rate.

What that identity licenses is a diagnosis. It says which factor is carrying the multiplier, and on this workload the rate mix carries more of it than the token count does, which is why the second and third columns of the table differ so much while reading the same material. What it does not license is any statement about value. Every term in it is a cost term, so the identity is silent on whether the extra tokens bought anything at all, and a pipeline whose rate mix has been repaired is still a pipeline that reads the same document four times.

The "runs billed per successful doc" row is the other multiplier people forget. The pipeline retries individual agents when their output fails a schema check, which happens on 23% of runs, and its end-to-end success rate is 84%, so a document that eventually gets processed correctly has consumed 1.23 / 0.84 = 1.46 pipeline executions on average. The loop retries on 6% of runs and succeeds end to end 93% of the time, giving 1.14. Cost per run flatters the pipeline by about 28% relative to cost per successful document, and the second number is the one the business pays.

Splitting the agent forfeits the cache, and getting it back costs a week#

Prompt caching is a prefix match. The request is rendered as tools, then system, then messages, and the cache key is the exact bytes up to each breakpoint. That mechanic is why the pipeline's cross-agent duplication is so expensive: the extractor and the validator both carry the same 5,600-token schema and the same 5,900 tokens of contract terms, but each sits behind a different system prompt, so the shared block starts at a different byte offset in each agent's prefix and the two never share a cache entry. Duplicated text is not duplicated prefix.

A confident and wrong sentence usually gets written here, and it is worth setting it down so that it can be checked. The sentence is: the duplicated text is byte-for-byte identical, so caching will absorb it. It is tempting because the quantity anyone would think to measure — how many duplicated tokens a run contains — really is unchanged by the split, and that quantity holds still while the property the cache actually keys on moves underneath it. What moves is where the duplicated block begins and what sits in front of it. Two agents carrying the same schema behind two different system prompts have an identical duplicate count and no shared prefix at all, so the cache treats them as two unrelated requests and bills both at full price. A constant token count sitting beside a varying prefix is the exact shape of a defence that looks like it works.

Within a single agent the picture is better, and that is what the third column bought. Adding a cache_control breakpoint after each agent's stable block — system prompt, tool definitions, and the schema and policy fragments that agent needs — lets its own turns two through eight read a warm prefix, and lets the block stay warm across runs as long as invoices keep arriving inside the five-minute TTL. That work took an engineer about a week across six prompts and moved the bill from $7.21 to $2.85 per successful document, which is still 4.3x the loop.

It cannot go lower than that, because the residue is structural. The pipeline still writes 148,200 tokens to cache per run against the loop's 33,100, and writes cost 1.25x. Every agent's first turn writes its own copy of the invoice text. Every handoff writes the upstream agent's output into a new conversation. The loop writes the invoice once and appends 16,000 tokens of its own work across six turns; the pipeline writes the equivalent material six times over, once per conversation.

Counting it in code#

The measurement is not hard, and doing it wrong is mostly a matter of summing the wrong fields. Every response carries a usage object with the three input categories separated, and the total prompt size is their sum — a common mistake is to read input_tokens alone and conclude the run was cheap, when input_tokens is only the uncached remainder.

from dataclasses import dataclass, field

# Opus 5 list rates, USD per token
IN_RATE = 5.0 / 1_000_000
OUT_RATE = 25.0 / 1_000_000
CACHE_READ = 0.1      # multiplier on input rate
CACHE_WRITE_5M = 1.25 # multiplier on input rate


@dataclass
class RunLedger:
    """Accumulates one end-to-end attempt, across every agent and retry."""
    calls: int = 0
    fresh_in: int = 0
    cache_read: int = 0
    cache_write: int = 0
    out: int = 0
    per_agent: dict[str, int] = field(default_factory=dict)

    def record(self, agent: str, usage) -> None:
        self.calls += 1
        self.fresh_in += usage.input_tokens
        self.cache_read += usage.cache_read_input_tokens
        self.cache_write += usage.cache_creation_input_tokens
        self.out += usage.output_tokens
        prompt = (usage.input_tokens
                  + usage.cache_read_input_tokens
                  + usage.cache_creation_input_tokens)
        self.per_agent[agent] = self.per_agent.get(agent, 0) + prompt

    @property
    def cost(self) -> float:
        return (
            self.fresh_in * IN_RATE
            + self.cache_read * IN_RATE * CACHE_READ
            + self.cache_write * IN_RATE * CACHE_WRITE_5M
            + self.out * OUT_RATE
        )


def cost_per_success(ledgers: list[RunLedger], succeeded: list[bool]) -> float:
    """The figure the business pays: total spend over documents actually delivered."""
    wins = sum(succeeded)
    if wins == 0:
        raise ValueError("no successful runs in this sample")
    return sum(l.cost for l in ledgers) / wins

The cost property is the identity above, written out one term at a time. Fresh input is billed at the list rate, cache reads at a tenth of it, cache writes at 1.25 times it, and output at its own rate, and nothing in it depends on how many calls produced those tokens — which is why the same function scores a seven-turn loop and a thirty-call pipeline without knowing which it is looking at.

record is called on every response from every agent, including the ones that failed a schema check and were retried, and including the orchestrator's own calls. Attributing a retried agent's tokens to nothing, or leaving the orchestrator out because "it barely does anything", is how a measurement comes back at 5x. The orchestrator in this pipeline consumed 122,500 input tokens per run — more than the intake and write agents combined — because it accumulated every agent's full report in its own conversation so it could decide what to do next.

The per_agent dictionary is what turns the total into an argument. On this workload it came back as extract 340,800, validate 157,900, price 122,400, orchestrator 122,500, intake 54,700, write 53,900. Two of those numbers are load-bearing for later chapters: the orchestrator costs as much as a working agent while producing no output anyone consumes, and the extractor's 340,800 is 40% of the whole pipeline's input because it re-reads a growing conversation eight times.

The orchestrator grows faster than the pipeline does#

The 122,500 figure deserves its own paragraph, because it is the term that gets worse as designs get bigger and it is almost never in anyone's estimate. An orchestrator that carries state across stages accumulates each stage's report into its own conversation, so its context on call k holds the sum of the first k−1 reports. Summed across all its calls, its total input is quadratic in the number of stages rather than linear.

Why that comes out quadratic is worth doing slowly, because it is the step everyone skips. Each report the orchestrator receives stays in its conversation and is re-sent with every later call. So the first call carries the system prompt alone, the second carries the system prompt plus one report, the third the system prompt plus two, and so on to the last. Adding those up means adding one report, then two, then three, and the running total of the whole numbers up to some count grows with the square of that count rather than in step with it. Linear growth would mean each new stage adds one call's worth of tokens and leaves the earlier calls alone. Quadratic growth means each new stage adds a call and lengthens every call that follows it, which is why the term is invisible in a design review and unmissable on an invoice.

With five stages producing reports of roughly 4,000 tokens each and a 2,100-token system prompt, the orchestrator reads about 2,100 + 6,100 + 20,300 + 26,100 + 31,600 + 35,200 tokens across six calls. Add a sixth stage and you do not add one call's worth; you add one call plus 4,000 extra tokens on every call after it. Going from five stages to seven takes the orchestrator from 122,500 tokens per run to roughly 205,000, a 67% increase on a component nobody thinks of as a workload.

There is a version of the orchestrator that does not have this property — one that holds no state, routes on the last stage's output alone, and forgets everything else. That version is cheap and it cannot make any decision that depends on what happened three stages ago, which is most of the decisions people build orchestrators to make. The choice is real, and the point here is that it has a price attached to it that grows with the stage count.

That is a design decision rather than a requirement, and the two are worth keeping apart. The requirement is that the run reaches a correct payable record. Deciding that the component which sequences the stages should also remember what they all said is a way of meeting that requirement, and the quadratic term is the price of that particular way. Naming it as a decision is what makes it possible to price the alternatives — a stateless router, a shared store the orchestrator reads on demand, a running summary that is rewritten rather than appended — instead of treating the growth as something inherent in having stages at all.

Latency needs the tail, not the average#

The p50 row, the median run that half of all runs finish faster than, makes the pipeline look 2.3x slower. The p95 row — the duration that only one run in twenty exceeds — makes it 4x slower, and p95 is where the operational pain lives, because the payables service has a 5-minute SLA with the AP team and a run that blows through it lands in a manual queue. The tail is worse than the median for a reason specific to staged designs: a retry at stage four does not add a fraction of a run, it adds a whole stage plus a handoff plus the orchestrator's re-read of the new output. The loop's retries are cheaper in wall-clock because a failed turn is re-attempted inside a conversation that is already warm.

Reporting the mean would have hidden this entirely. The pipeline's mean was 112 seconds against the loop's 47, a ratio of 2.4x that sits comfortably between the two honest numbers and describes neither the typical run nor the bad one.

The mean fails here in the same shape the call count failed in earlier. It is a single figure that moves smoothly while the property that costs the AP team their afternoon — how far past the SLA the worst runs land — moves independently of it. Two latency distributions can share a mean and put either none of their runs or a large share of them into the manual queue, and nothing in the mean separates the two cases. The p95 row does separate them, because it is defined by where the tail sits rather than by where the bulk sits.

Observability, iteration and reliability are separate bills#

It is not an argument that the pipeline is wrong. It is a price tag, and a price tag only becomes a decision when you know what is on the other side of it. The design review claimed two things in exchange for the cost: per-stage observability, and the ability to iterate on one stage without touching the others. Both are real claims and neither has been priced here. Chapter 6 takes the observability one apart specifically, because it is the argument that survives longest and is the easiest to satisfy another way.

The arithmetic also has conditions, and they are visible in the block algebra above. The multiplier is large here because S, C and D are large and almost every agent needs them. A decomposition whose stages read genuinely different material — where each block appears in one agent rather than four — has an input sum close to the single-conversation total, and its cost ratio collapses towards the call ratio the slide assumed. The way to tell which case a proposed design is in, before building it, is to write out the blocks and count how many stages each one appears in. That count is the multiplier, and it can be estimated in an afternoon from a design document.

Nothing in the arithmetic says the loop wins on the other side of the ledger either. A single conversation carrying five sets of instructions is the condition under which a model follows each of them less reliably, which is the reason the five agents were proposed. This chapter prices the tokens, and the cost model has no term at all for whether the answer was right.

The way to keep from being surprised by an invoice is to measure the two things the invoice is made of. Input and output are billed at different rates and move independently, so they are two columns rather than one. Fresh input, cache reads and cache writes are three prices for the same tokens, so the mix is a column too. Retries and the orchestrator's own calls belong to whichever document eventually shipped, which is why the denominator is successful documents rather than runs. Any measurement that keeps those separate would have produced the 11.3x before the migration instead of after it.

It is also not a reliability measurement, and the two get conflated constantly. The "1.46 runs per successful document" figure in the table came from an 84% end-to-end success rate that appears here only as a divisor. That 84% sits next to per-agent success rates that are all above 96%, and the gap between those two numbers is not a cost question at all — it is an arithmetic question about what "96% per stage" means when the stages are not independent of one another. That is where the next chapter starts.