Cost optimization portfolio

Read these first

• Model routing and cascades
• Prompt cache operations

What you will know

• Cut the cost of a real feature by half and say what it cost in quality

  Use a fixed evaluation set and production-like traffic so the savings and regression are measured rather than estimated.

• Rank the available levers by expected saving

  Base the ranking on request volume, token distribution, prices, hit rates, and implementation effort instead of headline percentage claims.

• Build a cost model on a whiteboard from first principles

  Include input and output tokens, retries, model mix, cache hits, batch utilization, and any fixed endpoint charges.

• Identify the dominant cost term before optimising anything

  A small number of high-volume paths usually dominate spend, while attractive per-request savings can be irrelevant at portfolio scale.

• Compare model tiering, prompt compression, output limits, batching, and caching on a fixed workload

  Measure savings alongside quality, latency, cache invalidation risk, queueing effects, and the traffic thresholds where each lever pays off.

Chapters

1. 1

   Half the request is tokens that never change

   Two million requests a month at 8,000 input tokens — 6,000 of them a stable instruction and tool block — plus 900 output tokens, priced at $3 and $15 per million, comes to roughly $0.0375 a request with about half of it in bytes that are identical every time. That arithmetic, not a vendor claim, is what nominates prompt caching first and sets its ceiling near 39% of the bill.

   Read chapter
2. 2

   The calls nobody counts are the gap in the invoice

   Requests times tokens times price predicts $14,000 against a consistent $23,000, and the whole gap is calls the formula does not know about: retries, generations that died after emitting tokens, the cache write premium, the second call on every escalation, and every request that is not the user's — judge scoring, verifiers, embeddings. Assigning the shortfall to whichever terms you happened to remember is how a cost model stops being falsifiable.

   Read chapter
3. 3

   Eleven routes, and the spend lives in two of them

   Spend concentrates severely across routes, so the first thing to compute — before any lever is evaluated — is which paths carry the bill. A per-request saving that looks excellent on a low-volume route is irrelevant at portfolio scale, and an hour spent there is an hour not spent where the money is. This also disarms the instinct to start with the route whose inefficiency is most irritating rather than most expensive.

   Read chapter
4. 4

   Each lever discounts a different base and none compound

   Ninety percent from caching, fifty from batch endpoints and eighty from a smaller model cannot be multiplied into a ceiling: caching touches only the repeated prefix, batching discounts the whole request and forfeits interactivity, and a cheaper model reprices everything at a quality cost. Prompt compression is the sharpest case, since the compressor is itself a model call and its signature failure is discarding the one fact the answer needed.

   Read chapter
5. 5

   Shortening an answer is not the same as truncating one

   Output is priced several times above input and is the one large term caching cannot touch — 900 tokens is roughly 36% of chapter 1's request, which is why the lever the vendor blogs never headline is often the largest one left. Two different things get called an output limit. Instructing brevity, narrowing the answer's scope, or moving to structured output genuinely shortens the response; max_tokens is a hard stop that cuts it off. The saving looks similar and the failure does not: a cap does its damage in the long tail, where the answer that needed 2,000 tokens ends mid-sentence while every median case scores exactly as before. Deduplicating overlapping retrieved chunks belongs beside it, as the compression that costs no model call.

   Read chapter
6. 6

   A day and reversible beats three weeks for the same money

   Capping max_tokens on the two chattiest routes and moving a reporting workflow to a batch endpoint are each worth about $4,000 a month, but one is a day's work undone by a config change and the other is three weeks that rewrites the product's latency contract from seconds to hours. Ship the cheap reversible one and re-measure, since the dominant term may have moved. Four levers at once forfeits attribution for savings and regressions alike.

   Read chapter
7. 7

   Fifty-two percent is half a result on its own

   Spend per request down 52% through caching, an output limit and cheaper routing is not accepted until the writeup carries the quality delta on a fixed evaluation set over production-like traffic, attributed per lever and sliced by task. The three levers carry different risks, and one blended score cannot distinguish paying 0.2 points across the board from breaking a single workflow completely. State what you gave up alongside what you saved.

   Read chapter
8. 8

   Dashboards that disagree cannot rank anything

   Model spend growing 6% a month, six teams shipping independently, and two homegrown cost dashboards that contradict each other and the invoice: nothing gets assigned until those reconcile, because a ranking built on numbers nobody trusts gets argued rather than executed. A uniform 30% target per team ignores how concentrated the spend is and invites six private quality trades. Keeping the reduction means the accounting outlives the project that produced it.

   Read chapter