Token cost of a tool result

A tool result is the one part of a prompt whose size you choose, row by row, and the arithmetic that decides how many rows to send is a per-row token cost. The number most teams use is the byte size of the payload divided by four, which is a rule of thumb calibrated on English prose. Topic 4.13, on budgeting the context window, established that characters per token is a property of the text rather than a constant, so the general shape of the error is not new. What is specific to a tool result is that its content is systematically the least prose-like text in the whole request — identifiers, timestamps, hashes and repeated field names — which puts it close to the worst case for that heuristic.

The consequence is not a rounding error but a mis-sized page, and the direction is the dangerous one: the estimate reads low, so a page that was scored as comfortable arrives at a fifth over budget or, on the largest accounts, does not fit at all. This chapter is about getting the number right and about what the number is then worth, which is less than it looks.

Every field in an order is the kind of string BPE handles badly#

Byte-pair encoding, or BPE, is the algorithm that cuts text into the tokens you are billed for, and it builds its vocabulary by merging frequent character sequences in a training corpus that is overwhelmingly natural language and code. Common English words become single tokens; so do common code fragments like def or return. A UUID is by construction a string that appears nowhere in that corpus, so it decomposes into whatever short fragments happen to exist, and the same applies to SKUs, content hashes and machine-formatted timestamps. Measured on one production order from a list_orders tool, field by field:

Each figure in the last column is that row's characters divided by its tokens, which is the direction that keeps the numbers comparable: a larger number means more text carried per unit billed. order_id is the extreme case and shows the mechanism without any interpretation. Thirty-six characters come back as twenty-three pieces, so the tokenizer found almost nothing in that string longer than a character and a half. That is what "appears nowhere in the training corpus" means in practice. A merge exists in the vocabulary only if the pair it joins was frequent while the vocabulary was being built; the sequences inside a random hex string were not frequent, because they are random; and a tokenizer with no long merge available has nothing to fall back on but the short merges it does have.

Only the free-text note behaves like the prose the heuristic was calibrated on. Everything else runs between 1.6 and 2.2 characters per token, so the whole row lands at 2.5 rather than 4.0, and a fifty-order page that was budgeted at 9,500 tokens measures 14,880. That is 57 percent over the estimate, and the error compounds with page size in exact proportion, so a 200-row page scored at 38,000 tokens is really about 59,600.

Those figures are reproducible from two numbers, and walking them is what turns the claim into something a reader can check. The 38 KB payload is 38,000 characters, and 38,000 divided by four is 9,500, which is the estimate. The measurement is what the tokenizer returned for the same bytes: 14,880. Fifty rows at the median 298 tokens each comes to 14,900, so the per-row figure and the whole-page figure agree to within a fraction of a percent, which is the check worth doing before trusting either. Dividing the measurement by the estimate gives a little over one and a half; the one is the estimate itself and the remainder is the overshoot, which is the 57 percent. It stays 57 percent at any page size, because both quantities are linear in the number of rows — the estimate is characters-per-row over four, the measurement is tokens-per-row — so quadrupling the page multiplies numerator and denominator alike and the ratio between them does not move. That is why the error does not wash out on a bigger page and why it does not have to be re-derived for each one.

The 2.5 characters per token for a whole row is also not the average of that last column, and reaching for the average is the commonest way to get this wrong. Averaging seven ratios treats a fourteen-character SKU as counting for as much as a two-hundred-and-ten-character scaffold, which it plainly does not, and gives about two and a third. Adding the character column and the token column and dividing one total by the other gives a little over two and a fifth for the same seven fields. They are different numbers off the same table and only the second answers the question, because the ratio that describes a row is the row's characters over the row's tokens, which weights each field by how much of the row it actually is. Do it that way over all fourteen fields — the seven above and the seven the table does not list, which are more text-like and tokenize better — and roughly seven hundred and sixty characters over 298 tokens is the 2.5.

The last row of that table is the part people leave out of estimates entirely. Key names are serialized once per row, not once per response, so a fourteen-field order pays for the strings "order_id", "placed_at", "customer_name" and the rest fifty times over. In this payload that scaffolding is 96 of the 298 tokens in an average row, roughly a third of the cost, spent on labels the model has already seen forty-nine times.

The reason it works out that way is a property of the format rather than an oversight by whoever wrote the serialiser. A JSON object carries its keys inline and has no channel for declaring a shape once, so there is nowhere to say "every element of this array has these fourteen fields" and then omit them. Each row therefore restates every key it has. What follows from that is the multiplication people miss: the cost of a field name is the length of the name times the number of rows, not times the number of fields. Adding one modestly named field to an order row is a small change to the schema and fifty repetitions of it in the payload, which is why a page's token count can move noticeably on a migration nobody thought of as a context change.

Bytes over four is not a bad rule; it is a rule with a stated domain that has been carried outside it. The intuition that keeps it there is worth naming, because it is the one that survives review. JSON looks token-heavy — dense with braces, quotes and colons, and punctuation feels expensive — so an estimate made on a JSON payload feels conservative. The mechanism above says the opposite. Punctuation in JSON is not rare. It is one of the most frequent sequences in any corpus that contains code, so "," and ":" and {" are exactly the pairs that got merged early and are cheap. The expensive part of a JSON payload is not its syntax; it is the identifiers the syntax is wrapped around, and those are the part nobody looks at when forming the impression that the estimate has headroom in it.

Count the payload the way the model will receive it#

The reliable way to get the number is to serialize a real payload and hand it to the tokenizer inside a real tool-result block, because the framing around the result — the tool_use id, the block type, the role structure — is tokens too, and because you want the count for the model you actually deploy on rather than a generic BPE library from a blog post.

import json
import anthropic

client = anthropic.Anthropic()
MODEL = "claude-sonnet-4-5"

def result_tokens(payload: object) -> int:
    """Tokens the tool result adds, isolated from the surrounding request."""
    def count(body: str) -> int:
        return client.messages.count_tokens(
            model=MODEL,
            tools=[LIST_ORDERS_SCHEMA],
            messages=[
                {"role": "user", "content": "show me recent orders"},
                {"role": "assistant", "content": [{
                    "type": "tool_use", "id": "toolu_01",
                    "name": "list_orders", "input": {"page_size": 50},
                }]},
                {"role": "user", "content": [{
                    "type": "tool_result", "tool_use_id": "toolu_01", "content": body,
                }]},
            ],
        ).input_tokens

    baseline = count("{}")
    return count(json.dumps(payload, separators=(",", ":"))) - baseline

Subtracting the baseline is what makes the result comparable across payload shapes, since otherwise the schema and the message framing — 1,100 tokens or so in this request — sit inside every measurement and flatten the differences you are trying to see. The subtraction is legitimate because that framing is a constant: the same tools, the same user turn and the same block structure are present whether the body is {} or fifty orders, so counting the empty case measures the constant and taking it away leaves the part that varies with the payload. Skip it and a payload that doubles in size looks like it grew by rather less than double, because a fixed 1,100 tokens is sitting in both readings and shrinking the apparent difference between them.

The same reasoning gives the per-row cost without ever isolating a row. Count a page of 10 and a page of 50, and both counts contain one wrapper and one set of framing; subtracting one from the other cancels everything the two pages have in common and leaves exactly the 40 rows by which they differ, so the difference over 40 is the marginal cost of a row. That is the number page-size decisions are actually made with, because a page is a fixed overhead plus a multiple of a row, and only the multiple is under your control.

The fixture is not the worst case#

Running that harness against the repository's test fixtures gives 121 tokens per order, which is less than half the production p50 of 298 — the median order — and the gap is not noise. Fixtures are written by hand to be readable: two line items, a customer called Test Customer, no note, no gift message, no split shipment. Production has a long tail of orders that are structurally different rather than merely larger. The p99 order in this system, meaning the size only one order in a hundred exceeds, is a wholesale order with 40 line items, a delivery instruction in Vietnamese, and three partial refunds, and it measures 1,240 tokens on its own.

Every cell in that table is one multiplication, which is what makes it checkable rather than authoritative. The estimate column is the page size times one hundred and ninety, because 9,500 tokens over 50 rows is what bytes-over-four claims a row costs. The middle column is the page size times 298 and the right-hand column is the page size times 1,240. Reading across a row compares three per-row costs at one page size; reading down a column confirms that each of the three is linear in the number of rows, which is the property that lets a cap be chosen by dividing a budget at all. Three per-row figures are the entire table, and everything else in it is arithmetic.

The right-hand column is where the interesting failures come from. A wholesale customer's second page of 200 orders does not fit in a 200,000-token window at all, and the request is rejected outright before the model has a chance to do anything sensible with it. Note also that the spread between the p50 and p99 columns — a factor of four — is larger than the estimator error that started this chapter. Correcting the heuristic without also choosing the right payloads to measure would have replaced a wrong number with a differently wrong number, so "representative worst case" means the largest real account, the row with the most children, the locale with the least Latin text, and the record that has accumulated the most history.

Fitting the window is a weaker claim than it sounds#

Suppose the measurement had come back at 59,600 tokens for 200 p50 orders and every account behaved like the median. The page fits, in the narrow sense that the request will be accepted. What it does not do is leave the session in a workable state, for three reasons that are easy to check and rarely checked.

The result stays in the conversation for the rest of the session. A tool result returned on turn three is re-sent as input on turn four, turn five, and every turn after that, so a 59,600-token page in an eight-turn session is read as input roughly nine times — about 536,000 input tokens, or $1.61 at $3 per million, from a single tool call. Prompt caching, which keeps the unchanged front of the request on the provider's side so that later turns are billed at roughly a tenth for that portion, reclaims most of that when the prefix is stable, at the cost of a 25 percent premium on the cache write and a discipline about where the cache breakpoint sits, but the first read is paid in full either way and the cache does nothing about the other two problems.

The nine is the step that makes that figure surprising, so it is worth pinning down. The result lands on turn three, which means it is part of the request the model reads on turn four; it is still in the request on turn five, and on every turn to the end of the session. An eight-turn session therefore reads it once on arrival and once for each turn that follows, which is why the multiplier tracks the number of remaining turns rather than sitting at one. Nine readings of 59,600 tokens is about 536,000 input tokens, and 536,000 at $3 per million is $1.61. None of that arithmetic is peculiar to tool results — it is how every token already in a conversation is billed — but a tool result is the one part of the prompt whose size you chose row by row, which is where the multiplier lands hardest.

Prefill is not free in latency terms. Sixty thousand tokens of input adds on the order of one to two seconds before the first output token, on every turn that carries it, which turns a conversational agent into one that feels like it is thinking hard about nothing.

The window is also not yours alone. Topic 4.13's budget divides it between system instructions, tool definitions, retrieval, history and an output reserve, and a 59,600-token result is claiming thirty percent of a 200,000-token window for one call in a session that will make more calls. Two such results and a moderately long conversation and you are inside the region where the agent starts dropping its early turns. Beyond the accounting there is a quality question that no token count answers: a model given 200 orders is measurably worse at finding the relevant one than a model given the eight orders that match the user's filter, and no amount of headroom in the window fixes that.

What a token count measures#

A token count is a measurement of size, and it is easy to slide from there into treating it as a measurement of cost or of quality. It is neither. Cost depends on cache state, on how many turns the result survives, and on whether the payload is read once or twenty times, so the same 14,880-token page can cost four cents or a dollar depending on the shape of the session around it. Quality depends on whether the right rows are in there at all, which a count cannot see.

The count is also perishable in the same way a context budget is: tokenizer vocabularies ship with model versions, so a family change can move these numbers by ten to thirty percent, and a schema migration that adds two fields to the order row moves them without anyone thinking of it as a context change. Measuring once and writing the answer in a design document gives you a number with a shelf life of about a quarter. The version that survives is an assertion in the test suite — count the p99 fixture, fail the build above a threshold — which costs one API call per CI run and catches the field that someone adds in six months.

What you have at the end of this is a defensible per-row cost, measured on the records that actually exist rather than on the ones in the fixtures directory, and taken through the same tool-result framing the model will receive. That turns page size from a preference into arithmetic: if a row is 298 tokens and you are willing to spend 8,000 tokens on this tool, the cap is about 25 rows.

Division is the right operation there because a page is one fixed wrapper plus a per-row cost repeated, and the budget is what you are willing to have repeated into. Dividing 8,000 by 298 gives a shade under twenty-seven, and the 25 comes from rounding that down and leaving the wrapper somewhere to sit. What the quotient cannot tell you is which per-row cost belongs in the denominator. The median order gives the twenty-seven above; the p99 order at 1,240 tokens gives six. A cap divided out of the median is a cap that holds for most accounts and fails on the ones with the most history — the fixtures lesson from two sections earlier, arriving the second time as a number rather than as an argument.

Choosing that cap is the easy half. The moment a cap exists, some queries match more rows than the cap allows, and the result the model receives is no longer the answer to the question it asked — it is a fragment of the answer, and unless the payload says so, the model has no way to tell the difference.