Input tokens per session, not per request, decide the cost

Two engineers bring competing designs for the same feature to a review. The feature answers questions about a single master services agreement — around 180 pages, which tokenises to roughly 250,000 tokens — and the usage logs from the prototype say a user asks about twelve questions before closing the tab. The first design puts the whole agreement in the model's context window on every request and asks the question at the end. The second chunks the agreement, embeds the chunks, retrieves the eight most relevant ones per question, and sends about five thousand tokens instead of two hundred and fifty thousand.

Both designs have names, and the choice between them is what this topic is about. The first is long context: put the whole document into the model's context window, which is the fixed span of text a model can read in a single request, measured in tokens — the word fragments that models read and that providers bill by, at roughly four characters each, which is why 180 pages comes to about 250,000 of them. The second is retrieval-augmented generation, written almost everywhere as RAG: cut the document into passages called chunks, turn each chunk into a list of numbers called an embedding that places it near other passages of similar meaning, keep those numbers in a searchable index, and then, when a question arrives, embed the question too, pull back the handful of chunks sitting nearest to it — the top-k — and send only those to the model alongside the question. The model answers from the passages it was handed rather than from the whole document, which is where the "augmented" comes from.

RAG exists because until recently the window was too small to hold anything substantial. A 4,000-token window could not read a contract, so answering questions about one meant finding the relevant pages first and paying a search system to do it. Windows have since grown by three orders of magnitude, and the practical question has moved from whether a document fits to whether putting it all in is the better way to spend money, seconds and accuracy. You meet the decision in any feature that answers questions over material the model was never trained on: a contract assistant, a support bot over a help centre, an assistant reading a code diff, an analyst tool over a data room. Which way a team goes is usually decided by whichever approach somebody read about most recently, and deciding it properly means pricing the two designs against each other for the particular feature in front of you.

Somebody in the room does the division out loud: fifty times cheaper, so the decision is made. That division is correct and the decision it produces is not, because the number being divided describes one request and the bill describes a session. Getting the comparison right is mostly a matter of writing down what you are assuming before you multiply, and this chapter does that carefully enough that the next three chapters can show which of those assumptions actually moves the answer.

Write the assumptions down before you multiply#

A cost comparison that lives in someone's head compares two numbers. A cost comparison you can defend compares two line-item tables, and the line items are where the argument actually happens. Here is the full set of assumptions for the document Q&A feature, chosen to be plausible for a mid-tier frontier model in 2026 and stated so that anyone can substitute their own.

Pricing is $3.00 per million input tokens and $15.00 per million output tokens, with an embedding model at $0.02 per million tokens. The document is 250,000 tokens. A session is twelve questions asked as one continuous conversation. Each answer runs about 400 output tokens. The system instructions plus the user's question come to about 300 tokens. The retrieval design chunks at 600 tokens with 15% overlap, which produces 479 chunks and 287,000 embedded tokens for one pass over the document, and retrieves the top eight chunks, so a retrieved prompt carries roughly 4,800 tokens of evidence plus the same 300 tokens of instructions and question.

Only two of those lines are requirements, and keeping them apart from the rest is what makes the table arguable rather than merely stated. The document's size and the number of questions in a session come from the product: a user brings the agreement they have, and asks what they need to ask. Everything else was chosen by somebody. Chunking at 600 tokens with 15% overlap is a decision, and the 479 chunks and 287,000 embedded tokens are consequences of that decision rather than facts about the document. Retrieving eight chunks is a decision, and the 4,800 tokens of evidence follow from it. Sending the whole agreement on every request is a decision too, and most of what follows is an account of what that one costs.

What a turn actually sends is three things stacked into one prompt. There is the evidence, whichever form it takes. There is the system instructions and the user's question, about 300 tokens of them. And there is the transcript of every exchange already completed in this session, which has to travel with the request because a model holds no memory between calls — a conversation continues only by being resent in full, every time. Just one of those three differs between the two designs, and it is the one everybody looks at.

One assumption is easy to skip and changes the retrieval column substantially: the twelve questions are a conversation, so each turn resends the previous turns. At roughly 700 tokens per completed exchange, turn twelve carries about 7,700 tokens of history, and across the session the conversation contributes 46,200 input tokens. That is 1.5% of the stuffed design's input and 43% of the retrieved design's input, which is the first sign that the two columns are sensitive to completely different things.

Where the 46,200 comes from is worth doing slowly, because it is the shape of the whole problem. One completed exchange is a question and its answer together: the 300 tokens of instructions and question, plus the 400 tokens of the answer, which is the 700. The first turn of a session carries no history at all. The second carries one exchange, the third carries two, and turn twelve carries eleven of them, which is the 7,700 above. The session's history bill is therefore 700 multiplied by the sum of the whole numbers from zero up to eleven.

That sum needs no lookup. Pair the smallest with the largest, the next-smallest with the next-largest, and so on: zero with eleven makes eleven, one with ten makes eleven, two with nine makes eleven, and the twelve terms are exhausted after six such pairs. Six elevens is sixty-six, and 700 multiplied by sixty-six is 46,200. The pairing trick works here for a reason that will hold at any session length — the terms run in an unbroken sequence, so every pair taken from opposite ends sums to the same total.

Writing the whole thing out once, in words, gives a relation that survives changing every number in it:

session input  =  turns × (evidence + question)
                  +  exchange × turns × (turns − 1) / 2

The first term grows in step with the number of turns. The second grows roughly with the square of it, because each new turn both carries the history accumulated so far and lengthens the history that every later turn will carry. That is the sentence to hold on to: evidence is linear in session length and transcript is quadratic in it, and the two columns below sit at very different points on that trade.

Instantiating the relation for the stuffed design puts 250,000 tokens of agreement plus 300 tokens of instructions and question into the first bracket, twelve times over, which is the 3,003,600 in the table's first row — a row that reads as evidence and in fact carries the instruction block along with it. For the retrieved design the same bracket holds 4,800 tokens of chunks plus the identical 300, twelve times over, which is 61,200. The second term is the same in both, because a conversation is a conversation whichever way its evidence arrived.

Read that table across the rows rather than down the columns. Two of its rows are the same number written twice — conversation history, and output tokens. One row exists in a single column and is blank in the other. Exactly one row differs by a large factor. Every argument anyone will have about which design is cheaper is an argument about that one row, and every correction to the naive fifty-fold figure comes out of the rows that do not differ at all.

The input rows also close, which is worth more than tidiness. 3,003,600 and 46,200 add to 3,049,800 with nothing left over, and 61,200 and 46,200 add to 107,400 the same way. An exact sum with no residual says the input model is complete: there is no fourth thing being billed that the table forgot to name. If a real invoice came back above those totals, the gap would be something outside the model — a system prompt longer than assumed, a retried request, a tool definition attached to every call — rather than an arithmetic slip inside it.

The ratio is 23×, not 50×. The fifty-fold figure came from dividing 250,000 by 5,000, which compares the evidence payloads and silently assumes everything else is either zero or identical. Two things break that: output is a floor both designs pay, and it is 18% of the retrieved design's bill while being under 1% of the stuffed design's; and conversation history is a fixed overhead that the stuffed design absorbs invisibly and the retrieved design cannot, because the retrieved design has no large number for it to disappear into. Neither correction changes the direction, and both change the magnitude by enough that quoting fifty in a design review is quoting a number you have not computed.

The collapse from fifty to twenty-three is a property of ratios rather than anything about language models, and seeing that makes it predictable instead of surprising. The stuffed design's bill is its evidence term plus a quantity both designs pay. The retrieved design's bill is a far smaller evidence term plus that same shared quantity, plus its index build. Adding the same positive amount to the top and the bottom of a fraction always drags the fraction toward one. Here it barely touches the numerator, where the shared quantity is a couple of percent of the total, and it dominates the denominator, where output alone is 18% of the bill and history is 43% of the input. Fifty falls to twenty-three because the divisor grew, and the divisor grew by an amount that has nothing to do with retrieval.

Subtraction behaves better than division here, and it is the operation the break-even below actually needs. Take the two session totals: $9.22 against $0.40 leaves $8.82. Now reach the same quantity without forming either total. History cancels, because both designs carry the identical 46,200 tokens of it. Output cancels, because both emit 4,800 tokens. The 300-token instruction block cancels, because both send it on all twelve turns. What survives is the difference between a whole agreement and eight retrieved chunks, carried twelve times and priced at $3.00 per million, less the single embedding pass that only the retrieved design pays for. That lands on $8.82 as well, and the residual between the two routes is a fraction of a cent — the rounding already sitting in the table's own cells.

The agreement between those two routes licenses a habit worth keeping. When two designs share most of their cost, compare them on the term that differs, and compute the ratio afterwards for presentation only. The ratio moves whenever anything common to both moves — a longer system prompt, a wordier model, an extra instruction block — while the difference does not move at all. A team that quotes the ratio will find it drifting for reasons unrelated to the decision it was supposed to inform.

The costs that sit outside the token counters#

Retrieval's index has to be built, and for this feature it has to be built per document rather than once for the product, because each session is a different agreement. The embedding pass is $0.005 per document, which is not the interesting part. The interesting part is that a per-document index is a piece of infrastructure with a lifecycle: something has to decide when to build it, where it lives, how long it is kept, what happens when a user uploads a revised agreement, and what the feature does during the seconds or minutes between upload and index-ready. On a 250,000-token document the embedding pass at typical hosted throughput takes somewhere between fifteen and sixty seconds, which is either a background job with a UI state or a blocking wait, and either way it is a thing that exists in the stuffed design and does not exist at all.

The embedding line is worth re-deriving rather than accepting, because it is the only figure in the table that is not a plain multiple of something already given. Multiply the 479 chunks by the 600 tokens each of them holds and the product is the 287,000 in the table, to the precision the table quotes it at. Now divide that product by the 250,000 tokens of the agreement, and what comes back is the overlap itself: 15% of the document is embedded a second time, because every chunk repeats the tail of the chunk before it. So the index build costs the document plus its overlap, and that holds whatever chunk size is chosen. Two engineers arguing about 600 tokens against some other size are not arguing about this line.

The chunk count is the rougher half of that, and it is worth knowing which half is soft. 479 is the document divided by the chunk size and then scaled up by the overlap fraction. Counting the chunks by their true stride instead — each one starting a little short of a full chunk past the last, since the overlap eats into the advance — gives about a dozen more of them and a slightly larger embedding pass. At $0.02 per million tokens the two answers differ by a small fraction of a cent, so nothing in the table moves. It matters only so that somebody re-deriving the line does not conclude the table is wrong when their own count comes out higher.

There is a second-order effect worth pricing while you are here. A per-document index is dead weight the moment the session ends, so either you keep it — storage, a vector store bill, a retention policy, and a deletion path for a customer who asks you to remove their contract — or you throw it away and rebuild it if the user comes back tomorrow. For a corpus that is shared across all users, the index is built once and amortised across every session forever, and its cost genuinely rounds to zero. For a corpus that is one customer's uploaded document, the amortisation window is a single session, which is the shortest it can possibly be. That distinction between shared and per-session corpora recurs throughout this topic, and it is usually the fact that decides an argument that looks like it is about token prices.

The belief worth naming here is that an index is a one-off. It is a true statement about a shared corpus and a false one about an uploaded document, and the sentence sounds identical either way, which is what makes it dangerous rather than merely wrong. What makes it true when it is true is amortisation across many sessions. So the property to check is never the build cost on its own — it is the number of sessions one index survives, and for a per-document index that number is one.

The same asymmetry shows up in the failure surface. The stuffed design has one way to be wrong about evidence: the model had everything and still answered badly. The retrieved design adds chunk boundaries that split a clause across two chunks, an embedding model that scores a paraphrase lower than a lexical near-match, a top-k that was tuned on a different corpus, and a reranker if you added one, meaning a second and slower model that re-scores the retrieved candidates before they are placed in the prompt. None of that appears on an invoice. It appears as engineering weeks, and the fair way to include it in a cost comparison is to convert the token saving into a break-even.

At $8.82 saved per session, a retrieval pipeline that costs one engineer six weeks to build and a day a month to keep alive — call it $45,000 fully loaded in the first year — pays for itself at about 5,100 sessions in that year, roughly fourteen sessions a day. Any real product clears that easily, which is why the break-even framing looks like a formality here. It stops being a formality in the next chapter, when the saving per session drops by a factor of five and the threshold moves into the range where actual traffic decides it.

The break-even is one division, and writing it out shows what it is sensitive to. The one-off build cost goes on top, the saving per session goes underneath, and the quotient is the number of sessions before the pipeline has paid for itself: $45,000 over $8.82 is the 5,100, and dividing that by the days in a year gives the fourteen. Because the saving sits in the denominator, halving it doubles the threshold. And the saving is a difference, which means it is the one quantity the previous section showed to be immune to history, output and instruction length, and fully exposed to just two things: how large the document is, and how large a retrieved prompt is.

Scale it before you argue about it#

Quality, latency, and the missing middle#

The table is not a quality comparison, and nothing in it says the two designs answer the same questions equally well. Retrieval that returns the wrong eight chunks produces a cheap wrong answer, and a cost table has no column for that. Chapter 4 measures the quality axis directly, and it does not point the way this table does.

It is not a latency comparison either. Sending 250,000 tokens means the model has to read 250,000 tokens before it can emit the first one, and that shows up as time to first token in a way that no amount of cost optimisation removes. That axis also gets its own chapter, and it is the axis on which the stuffed design most often loses in production even when it wins on accuracy.

It is also not, despite appearances, a comparison of two architectures. It is a comparison of two points on a spectrum whose middle is missing: select a subset of the corpus, then put that subset in a large window. Chapter 5 puts that branch back, and it turns out to be the design that most production systems converge on once both endpoints have been measured.

The per-request number that never appears on an invoice#

The framing error worth naming, because it is the one that leads teams to skip the arithmetic entirely, is thinking per request. At $3 per million tokens, one 250,000-token request costs 75 cents, which sounds like nothing next to an engineer's hourly rate, and a prototype that a team tests by hand — a few dozen requests over an afternoon — produces an invoice of about twenty dollars that confirms the impression.

That invoice is the defence, and it is a defence offered in good faith: somebody did check the cost against real usage, and the number that came back really was small. What makes it useless is the shape of the quantity it measures. Cost per request in the stuffed design is the agreement, plus the instruction block, plus however much history that particular turn happens to carry — and history is 1.5% of the session's input, so the per-request average barely stirs however long sessions run. A dashboard reading dollars per request sits at almost the same value for a session of one question and a session of forty, while the bills for those two sessions differ by a factor of forty or more.

That is the shape to look for whenever a cost check comes back reassuring: a quantity holding still in front of a quantity that is free to move. The metric here is genuinely measured, genuinely correct, and blind by construction to session length, which is the one input that decides the bill. The unit that bills you is the session, and the unit that bills you at scale is the session multiplied by daily volume, so the per-request figure is the one number in this whole analysis that never appears on an invoice and it is the one people quote.

Twelve full reads of the same agreement#

Everything in the table rests on a single load-bearing line: total input tokens per session, 3,049,800. That number assumes the model reads the agreement from scratch twelve times and charges full input price for each read. It is the correct assumption for a naive implementation, and it is exactly what a naive implementation does.

Trace where that assumption enters and it turns out to sit in one place. The first term of the session-input relation multiplies the evidence by the number of turns, and that multiplication is honest only if every read is a fresh one. Nothing in the arithmetic demands it. Twelve identical reads of a byte-identical prefix is a fact about how each request was assembled rather than about anything the model requires, and no other line in the table has that character: the document's size is given, the session length is measured, the chunking was chosen deliberately, and this one was chosen by nobody.

It is not what the design being proposed does, or at least not what it should do. The agreement is byte-identical across all twelve requests within a session, it sits at the front of the prompt, and prompt caching exists precisely so that a stable prefix is paid for once. Chapter 2 works out what happens to the 3,049,800 when eleven of the twelve reads are served from cache at a fraction of input price, why that requires the agreement to sit ahead of everything that varies, and how a prefix that looks stable can measure a 4% hit rate in production. The 23× gap in the table above survives that chapter at about a fifth of its size, and the break-even that looked like a formality here stops being one.