Quadratic growth of input tokens

Every request in this topic has carried the entire conversation, because the API holds nothing between calls and the array is the only thing the model reads. That is a correctness requirement, and it has a price attached that is easy to mis-estimate: turn forty is not a small request, because it contains turns one through thirty-nine, and so did turn thirty-nine, and turn thirty-eight, all the way down. The tokens you send are billed input, priced per million and reported on every response as usage.input_tokens, so the cost of a conversation is the sum over its turns of an array that keeps growing. Resending full history therefore grows the input bill as n squared, and the rest of this chapter is that sentence turned into a number you can put in a forecast.

The closed form, on the shape of a real support bot#

Take a user message and the assistant's reply as one pair, and call the pair's size t. On turn one you send t. On turn two you send the first pair plus the new question, so roughly 2t. On turn n you send roughly n·t. The cumulative billed input across a conversation of n turns is therefore the sum of t, 2t, 3t, up to n·t — the arithmetic series that closes to:

cumulative input tokens  =  t · n(n + 1) / 2

That closing step is the whole chapter, so it is worth doing rather than accepting, and it is easier on a small case than in symbols. Take a conversation of five turns. The amounts sent are t, then twice t, three times t, four times t, five times t. Add them from both ends at once. The first and the fifth come to six t. The second and the fourth also come to six t, because the one turn gained on the left is exactly the one lost on the right. That leaves the middle amount, three t, which is half of six t. So the total is two and a half lots of six t, or fifteen t. With t at 600 tokens, fifteen lots of six hundred is 9,000 — which is what the table below reports as the running total at turn five.

The general case is the same move with n in place of five, and each step is allowed for the same reason. The first and last amounts are t and n·t, summing to (n + 1)·t. The second and second-to-last are twice t and (n − 1)·t, which also sum to (n + 1)·t. Working inwards, every pair sums to (n + 1)·t, because each step inwards adds one turn's worth on one side and removes one turn's worth on the other. There are n/2 such pairs. Multiply and the total is t · n(n + 1)/2, which is the line above. When n is odd there is an unpaired middle amount, and it is exactly half of (n + 1)·t — so counting it as half a pair leaves the expression unchanged, which is why the formula needs no separate case for odd conversations.

The intuition to keep, once the algebra is done, is that the multiplier is not mysterious. Dividing the quadratic total by the linear one leaves (n + 1)/2. The factor by which a linear forecast is wrong is therefore just half the conversation length, plus a bit — a number you can work out in your head before anyone opens a spreadsheet, and the source of the twenty-times error this chapter keeps returning to.

The instructive way to see the gap is turn by turn rather than as a formula, because the per-turn number stays comfortable long after the total has stopped being comfortable:

At 600 tokens per pair and forty turns, one conversation costs 492,000 billed input tokens. Across 50,000 conversations a month that is 24.6 billion input tokens, and at $3 per million the monthly input bill is $73,800. The linear estimate — 40 × 600 × 50,000 = 1.2 billion tokens, $3,600 — is off by twenty times.

The same shape at smaller scale is worth having in your head as a sanity check. A bot that resends everything with 500 tokens per pair, run for twenty turns, bills 500 × 20 × 21 / 2 = 105,000 input tokens for that single conversation, where the naive count says 10,000.

def cumulative_input_tokens(turns, tokens_per_pair):
    return tokens_per_pair * turns * (turns + 1) // 2

def monthly_input_cost(turns, tokens_per_pair, conversations, usd_per_mtok):
    per_conv = cumulative_input_tokens(turns, tokens_per_pair)
    return per_conv * conversations * usd_per_mtok / 1_000_000

monthly_input_cost(40, 600, 50_000, 3.0)   # 73800.0

Two things this arithmetic deliberately ignores. Output tokens are excluded, and they are priced several times higher per token — but they grow linearly, because turn forty's reply is no longer than turn one's, so they are not where the surprise lives. And it assumes uniform turns, which real traffic never is; a single pasted stack trace or a retrieved document in the middle of a conversation lifts t for every subsequent turn, which is the quadratic term getting a larger coefficient rather than a different curve.

The other ceiling the curve runs into#

Cost is the effect that shows up on a monthly boundary. The one that shows up mid-conversation is the context window — the maximum number of tokens a model will accept in one request — and the arithmetic there is the per-request figure rather than the cumulative one. Turn n sends n·t tokens, and when that exceeds the limit the request fails outright or has to be truncated on the spot, mid-incident, by code written under pressure.

At 600 tokens per pair the wall is a long way off: a 200,000-token window is reached somewhere past turn 330, and no support conversation gets there. Each of those numbers comes from one division rather than from the series, and the difference is the point. The cumulative figure is what you pay; the per-request figure is what has to fit. Turn n sends n·t, so the turn at which the window fills is the window divided by t — two hundred thousand over six hundred, which is the turn number above.

The number that matters is therefore not the average pair but the pair size of the conversations that involve documents. A turn that injects two retrieved pages of 3,000 tokens each pushes t to around 6,600, and the same division now gives a wall at turn 30, which is an ordinary length for a conversation about a complicated case. Note what changed and what did not: the window is the same, the formula is the same, and t went up by roughly a factor of eleven, so the turn at which you hit the ceiling came down by roughly a factor of eleven. The cost figure moves by that same factor for the same reason — t is a plain multiplier in front of n(n + 1)/2 — which is why the two ceilings tend to be approached together, by the same conversations, on the same days.

Measuring t on your own traffic takes one query and is worth more than any published average. For each conversation in a week of request logs, sort the calls by time and take the difference between consecutive usage.input_tokens values: that delta is the size of the user/assistant pair added since the previous turn. Two statistics come out of it, and both are load-bearing. The median tells you what the bill will look like. The 95th percentile tells you when you will hit the context ceiling and how badly a fixed message-count window will overshoot a token budget — a workload with a median pair of 480 tokens and a 95th percentile of 2,900, because some users paste logs, is one where the average-based forecast is fine and the tail conversations are the ones that break.

What a bounded window actually buys#

Cap the array at the last ten messages — five user/assistant pairs, 3,000 tokens — and the per-turn payload stops growing after turn five. Conversations still ramp up (600, 1,200, 1,800, 2,400, 3,000) and then flatten, so a forty-turn conversation bills about 9,000 tokens for the ramp plus 35 × 3,000 for the rest, or 114,000 input tokens against the unbounded 492,000. Across 50,000 conversations that is 5.7 billion tokens and $17,100 a month, a saving of about $56,700, or 77%.

Both halves of that sum come from somewhere visible. The ramp is the first five turns of the unbounded table, unchanged, because a window that holds five pairs has not discarded anything yet — which is the 9,000 running total already read off the table above. After that the payload is pinned at the window size, and the remaining thirty-five turns each send it, so the tail is the window size multiplied by the turns that are left. The two added together are the whole conversation, and the shape is worth noticing: one term that curves and stops, and one that is a straight line for as long as the conversation runs.

The curve matters more than the number. Full resend is O(n²) in conversation length; a bounded window is O(n), with the window size as the constant. Doubling the average conversation length quadruples the first bill and merely doubles the second, which is why the unbounded version tends to break in the same quarter that engagement improves.

The saving is real and the conclusion "cap the array at ten messages, done" is still wrong, because a fixed message count is measured in the wrong unit. Ten messages is not a token budget: nine short turns and one pasted 8,000-token stack trace is a window that blows through any per-request ceiling you thought you had set, and there is no upper bound on it because there is no upper bound on what a user can paste. Ten messages is also not a structural boundary — the previous chapters' tool-call pairs occupy two array positions bound by an id, and counting backwards ten entries has no idea it might be landing between them. That failure has its own chapter, immediately after this one, because it produces a hard 400 rather than a large invoice.

The options are best compared along the dimensions you will actually be asked about:

That last row is worth understanding properly, because it is the honest answer to "when would you still resend everything?" Prompt caching lets the provider retain the computed prefix of an array you have already sent, so a subsequent request whose leading tokens are byte-identical is billed for those tokens at roughly a tenth of the input price. Applied to a conversation, where every turn's array is the previous turn's array plus two entries, the hit rate is excellent and the effective bill on the support bot above drops from $73,800 to something in the region of $10,000 — better than the windowed version, with no context discarded at all.

The reason that estimate is a region rather than a figure is worth a sentence, because it shows where the discount comes from. On turn n the array is the turn n − 1 array plus one new pair, so the cached share is (n − 1)/n and the uncached share is one turn's worth. Early in a conversation that fraction is poor; by turn forty almost everything is a hit. Divide the whole bill by ten and you get a figure a little under eight thousand dollars, which would be the answer if every token were cached; add back the pair that is new on each of the forty turns, at full price, and you land near the ten thousand quoted above. The discount is large because the resend is almost entirely repetition, which is the same property that made the resend expensive.

The catch is in "byte-identical". Caching is a prefix match, so anything that edits the front of the array invalidates everything behind it: rewriting an early turn, inserting a summary, changing the instruction, reordering the tool declarations, or interpolating a timestamp into the system field. It also does nothing for the context window, which is a hard ceiling that a long enough conversation reaches whatever you are paying per token. Caching changes the price of the resend, not the size of it.

Where full resend survives the arithmetic#

The defensible cases are narrower than habit but wider than zero. Conversations short enough that n² never gets large are simply not worth engineering for — an eight-turn checkout assistant costs 36t, and a window for it is wasted effort. Legal, medical and financial transcripts, where dropping an earlier turn is a correctness or compliance problem, pay the quadratic deliberately and account for it. And cached high-volume workloads with a genuinely stable prefix get the tenth-price read on almost everything, which makes the quadratic term affordable without discarding a single turn. In each case you are choosing the cost with a number in front of you.

One proposal fails this test in a way worth pricing explicitly, because it looks free: the reinforcement proposal. The chapter on the system role set aside an argument for injecting a fresh block of restated rules before every user turn — reinforcement, on the theory that a rule repeated near the end of the array is obeyed more reliably — and left the cost here. Suppose the block is 150 tokens. It does not add 150 tokens per turn; it adds 150 tokens to t, and t is the thing being multiplied by n(n+1)/2. The forty-turn conversation goes from 600 to 750 tokens per pair, and from 492,000 to 615,000 cumulative input tokens — 123,000 extra tokens per conversation, 6.15 billion a month, $18,450 a year-round line item for repeating a paragraph the model already has at the front of every request. It is also the shape of change that defeats caching, because inserting fresh text into the middle of the array on every turn moves the prefix and forfeits the reads that were making the resend affordable. A proposal that costs $18,450 a month directly and quietly removes a $60,000 discount is worth pricing before it ships, and the pricing takes five minutes with the function above.

Where to put the meter#

Cost per conversation, not cost per request, is the metric that catches this class of error. Log usage.input_tokens and usage.output_tokens on every call along with the conversation id and the turn index, and plot the per-conversation total against turn count. A linear plot means your window is working. A parabola means you are resending everything, whatever the code comments say. Watching the per-request number alone shows a figure that grows slowly and looks fine right up until the month closes.

The whole of this chapter is one substitution: replace n with n(n+1)/2 wherever you were about to estimate the cost of a conversation, and the numbers stop surprising you. At forty turns that is a factor of twenty, which is large enough that no amount of care about per-token prices or model selection compensates for getting it wrong, and small enough to fix with a window measured in tokens or a cache on a stable prefix. The two ceilings move independently: a window moves both, caching moves only the price, and a summary moves both at the cost of fidelity and of the cache.

That plot is also the first place a truncation bug shows up, which brings us to the version of this that fails loudly. A window has to decide where to cut, and the array is not a flat list of interchangeable strings — some positions in it are bound to other positions by identifiers the provider validates. Cut in the wrong place and you do not get a slightly worse answer; you get a 400 on some conversations and not others, and the ones that fail are the ones where the assistant did the most work.