Retrieved context and system prompt, not the question, set input cost

The relevance experiment shipped on a Tuesday and looked like a clean win. Answer quality scores rose from 3.9 to 4.3 on the internal rubric, the support team stopped complaining that the assistant missed obvious documents, and the change was two lines in a config file: top_k, the number of passages retrieval attaches to each request, from 5 to 8, and chunk size from 400 tokens to 600. Nineteen days later the finance analyst who reconciles the model invoice asked why cost per request had gone from $0.021 to $0.033, a 57% increase, in a month where request volume was flat and the median user question was the same length it had always been — thirty-four tokens, the length of "how do I change the billing email on a team account".

Nobody in the room had connected the two events, because the retrieval config lived in the search team's repository and the invoice lived in a finance dashboard, and the only shared vocabulary between them was "cost per request", a single number that hides its own causes. The arithmetic is not subtle once you look: eight chunks of 600 tokens is 4,800 tokens of retrieved context where five chunks of 400 was 2,000. At $3.00 per million input tokens that is $0.0084 more per request before the model has read a word the user wrote, and at 1.4 million requests a month it is an extra $11,760.

The vocabulary those two teams were missing has a name. Unit economics is the arithmetic of a single unit of the thing you sell — one task, one session, one seat, one customer — instead of the month as a whole: what serving that one unit costs you, what it earns, and therefore whether having more of them makes the business better or worse. The figure at its centre is cost per task, the all-in cost of one completed piece of work, and the ratio built on top of it is gross margin. Conventional software mostly gets away without computing any of this, because the marginal cost of one more user is close to zero and any sensible price clears it. Inference behaves differently, in a way that shows up on an invoice: every request carries a variable cost that scales with how hard a customer chooses to use the product, so a price set from a comfortable average can lose money on the customers who use it most, and nothing in the contract stops them. You meet the exercise wherever a per-unit figure gets quoted — the cost-per-request line on a finance dashboard, the margin claimed for an AI feature in a board deck, the pricing page, the renewal where a customer's usage has tripled since the last one — and it is what turns "cost per request went up 57%" from a fact into a decision. Building it starts with tokens, because tokens are where the money first goes.

The question is a rounding error in its own request#

Topic 15.9 decomposed a single request into its terms to nominate a cost lever. The same decomposition serves a different purpose here, which is attribution: for every token you pay for, who decided it would be there? Run that question over a production request from the support assistant and the answer is uncomfortable, because almost none of it was decided by the user.

The user's question is 0.37% of the input tokens on a median request. Even at the 99th percentile of question length — 180 tokens, someone pasting a stack trace into the box — it is under 2%. Every other line in that table is a choice made by someone on your team, usually in a pull request that mentioned quality and never mentioned cost, and that is the practical meaning of the claim that retrieved context and the system prompt set input cost. Input spend is a function of your configuration, sampled by user traffic rather than caused by it.

This has a pleasant consequence and an unpleasant one. The pleasant one is that input cost is controllable, because you own every large term. The unpleasant one is that input cost drifts upward on its own: tool definitions accumulate as features ship, few-shot examples get added whenever someone finds a failure mode, and retrieval breadth gets raised whenever recall looks weak. None of those changes is individually large enough to argue about, and all of them are priced per request, forever.

Counting by component instead of by request#

Provider APIs report usage per request — input_tokens, output_tokens, and cache fields — which is enough to reconcile the invoice and useless for attribution, because it is one number where you need six. The fix is to count the pieces yourself at assembly time and log them alongside the request identifier, so that the same row can be joined to the invoice on one side and to a config version on the other.

from dataclasses import dataclass, asdict
from anthropic import Anthropic

client = Anthropic()

@dataclass
class InputBreakdown:
    system: int
    tools: int
    examples: int
    retrieved: int
    history: int
    question: int

    @property
    def total(self) -> int:
        return sum(asdict(self).values())

def count(text: str) -> int:
    return client.messages.count_tokens(
        model="claude-sonnet-4-6",
        messages=[{"role": "user", "content": text}],
    ).input_tokens

def build_request(system: str, tools: list[dict], examples: str,
                  passages: list[str], history: list[dict], question: str):
    retrieved_text = "\n\n".join(passages)
    breakdown = InputBreakdown(
        system=count(system),
        tools=sum(count(str(t)) for t in tools),
        examples=count(examples),
        retrieved=count(retrieved_text),
        history=sum(count(m["content"]) for m in history),
        question=count(question),
    )
    return breakdown

Calling count_tokens on every component of every production request would double your API call volume, so in practice you count exactly once per stable artifact and cache the result. The system prompt, the tool schemas and the few-shot block change only when someone deploys, so their token counts belong in the build: compute them at startup, attach them to the config version, and emit them as constants on every request log. Only the retrieved passages and the user's message need counting at request time, and for passages you can usually skip the API entirely because the chunker already knows how long each chunk is — it produced them by splitting on a token budget.

Approximating tokens with a character heuristic is tempting and is worth resisting for anything that feeds a cost model. The familiar four-characters-per-token rule holds reasonably for English prose and breaks badly on the content that dominates these requests: JSON tool schemas run closer to 2.8 characters per token because of punctuation and quoting, code snippets in retrieved passages run around 3.1, and non-English text can be twice as expensive per character as the rule predicts. A German-language tenant on this product showed 41% more input tokens than the heuristic estimated for the same documents, which is enough to make a per-tenant margin calculation wrong in the direction that matters.

What you want at the end is one row per request carrying the six component counts, the config version, the tenant, and the provider-reported input_tokens. The gap between your sum and the provider's number tells you how much of the request is envelope you did not model — message framing, role markers, the serialisation overhead of tool schemas — and in the support assistant that gap ran about 4%, which is small enough to carry as a fixed multiplier and large enough that ignoring it would have made every downstream estimate 4% optimistic.

Retrieval knobs are pricing knobs#

Once component counts exist, the retrieval configuration can be evaluated the way any other pricing decision is, with the cost consequence next to the quality consequence. The grid below is from the support assistant, holding the 4,330-token fixed block constant and varying only the two parameters the relevance experiment touched. Cost per request assumes $3.00 per million input tokens and excludes output.

The shipped configuration is the fourth row, and the third row is interesting because it buys 0.3 of the 0.4-point quality gain for $6,700 a month less. The fifth row is the one worth internalising: fifty percent more retrieved context bought exactly zero measurable quality, which is the normal shape of retrieval — recall saturates while cost stays linear. A team that cannot see this table will keep raising top_k whenever a bad answer gets escalated, because raising it is free at the moment of the decision and expensive only on a bill that arrives a month later, attributed to no one.

The same logic runs over the fixed block. Nine tool definitions at 2,400 tokens are 26% of the median request, and if three of those tools are called in under 1% of sessions, they are being paid for in the other 99%. Splitting the assistant into two entry points with four tools each, or moving rare tools behind a routing step, is a cost decision priced in the same units as the retrieval decision, and it belongs in the same table.

Turning the count into a guardrail#

A measurement that is read when someone remembers to read it will drift, because the changes that grow the fixed block arrive one at a time and each is defensible on its own. Over the eleven months before this instrumentation existed, the support assistant's system prompt grew from 380 tokens to 1,150 across nineteen pull requests, none of which added more than 120 tokens, and the tool block grew from four schemas to nine. Nobody made a decision to triple the fixed cost of every request; nineteen people made small decisions and the sum was never presented to anyone.

The mechanism that stops this is a budget checked in continuous integration, in the same place tests run. Compute the fixed block's token count at build time, compare it against a committed baseline, and fail the check when it grows by more than a set fraction, forcing an explicit update of the baseline and a line in the pull request describing the cost.

BUDGET = {"system": 1_200, "tools": 2_500, "examples": 800}
MONTHLY_REQUESTS = 1_400_000
INPUT_PRICE = 3.00 / 1_000_000

def check(counts: dict[str, int]) -> int:
    failures = 0
    for name, limit in BUDGET.items():
        actual = counts[name]
        if actual > limit:
            delta = actual - limit
            cost = delta * INPUT_PRICE * MONTHLY_REQUESTS
            print(f"FAIL {name}: {actual} > {limit} "
                  f"(+{delta} tokens = ${cost:,.0f}/month)")
            failures += 1
    return failures

The value of expressing the failure in dollars a month rather than in tokens is that it makes the trade-off legible to the person making it. An engineer adding 140 tokens of instruction to fix a formatting complaint sees "+$588/month" in the failing check and can decide whether the formatting complaint is worth that, which is a reasonable question with a reasonable answer either way. Without the number, the same engineer sees a lint failure and raises the budget.

The equivalent guardrail on retrieval is a cap on retrieved tokens rather than on chunk count, because top_k and chunk size interact and a limit on either alone can be evaded by moving the other. Enforcing a ceiling of 5,000 retrieved tokens per request, and truncating the lowest-scoring chunks when the ceiling binds, converts an unbounded configuration surface into a bounded one and makes the worst-case input tokens per request a known quantity rather than something you find out from an invoice.

What a per-component count does not give you#

This is a measurement of input on a single turn, and three things sit outside it. Output tokens are typically priced at four to five times input and are generated fresh every time, so a request whose input you have carefully modelled can still have most of its cost in the completion; the component table above stops before the model speaks. Multi-turn behaviour is absent entirely, because history is zero on a single-turn request and becomes the dominant term the moment the product has a conversation in it. And the arithmetic assumes list prices, which is where a per-request number and a per-month number start to diverge — committed-use discounts, batch rates and cache pricing all apply after the tokens are counted.

There is also a failure mode worth naming, which is treating the component table as the cost model. It is an input to one. A per-request token breakdown tells you what a successful call costs; it says nothing about calls that failed and were retried, sessions the user abandoned halfway, or the embedding and storage cost of having those 4,800 tokens of passages available to retrieve at all. Teams that stop here get a number that is precise, defensible against the provider invoice, and roughly half of the real unit cost.

The immediate gap, though, is the one in the history row. Every number in this chapter came from a request where history was zero, and the moment the support assistant became a multi-turn agent, that row stopped being zero and started growing with every exchange in a way that makes single-turn arithmetic misleading rather than merely incomplete.