Best-of-five costs more than five times one call

The monthly invoice for the contract reviewer went from $4,820 to $41,500 on flat volume. The team had signed off on an eight-way fan-out expecting eight times the cost, had told finance to expect roughly $38,500, and were 8.6 times over the old number rather than 8. On a per-review basis that is $0.1578 before and $1.3593 after, and the gap of $0.097 per review is not large enough to be interesting on its own. It is interesting because nobody could account for it, and an unaccountable 8% on a design change is the same thing as not knowing what the design change costs.

The estimate that produced $38,500 was one multiplication: eight calls where there was one, so eight times the price. The same reasoning appears in a smaller form every time somebody proposes best-of-five and estimates it as five times the per-request cost. Both estimates leave out the same four things, and it is worth going through them on a task small enough to hold in your head before returning to the contract bill.

What five samples of a small task actually cost#

The natural-language-to-SQL feature sends a 2,900-token schema and instruction block that is identical on every request, plus a question averaging 300 tokens, and gets back a query averaging 260 tokens.

The prices below assume prompt caching, which is the provider holding on to the processed form of a repeated prefix for a few minutes so that a later request beginning with the same text is billed at a low cache-read rate, with the request that establishes the entry paying a slightly raised cache-write rate. Four rates are in play: $3.00 per million input tokens, $15.00 per million output, $0.30 per million for a cached input read and $3.75 per million for a cache write. Two ratios among them do most of the work in this chapter. A cached read is priced at a tenth of an ordinary input read, and a cache write is priced above an ordinary input read rather than below it — $3.75 against $3.00, which is a factor of 1.25 — so the gap between the cheapest and dearest way to send the same input token is a factor of 12.5.

With the schema block warm in the prompt cache because every request in the service shares it, one call is three products summed: the 2,900-token prefix at the cache-read rate, the 300-token question at the full input rate, and the 260-token query at the output rate. That comes to $0.00567.

Five samples with the winner chosen by counting normalised query results in application code costs exactly $0.02835, which is 5.00 times the baseline. The naive estimate is correct here, and saying so matters: the multiplication is not wrong in principle, it is unaudited. Three conditions happen to hold in this case — the schema prefix stays cached across all five branches, the output length does not change, and nothing else was added — and each of those is a condition that can fail rather than a property of fan-outs.

Add a judge that reads the question and the five candidates and picks one, and the picture moves. That call sends the cached 2,900-token prefix, the 300-token question and 1,300 tokens of candidate queries, and returns a 120-token verdict, for $0.00747 — thirteen percent of the entire five-sample spend, spent on the step that is easiest to forget because it is not part of "the model call". Total $0.03582, or 6.32 times baseline.

Then measure retries rather than assuming them. In production 2.1% of branches hit the 20-second client timeout, on average after generating about 180 output tokens, and were re-run from scratch. Which of those failures cost money is worth being precise about, because the intuition that failed requests are free is only half right. Provider-side refusals — the 429s from the previous chapter — are not billed, and neither are 5xx responses, because in both cases the provider did no work on your behalf. A stream that your client abandons is different: the tokens were generated, and you are billed for everything produced before you hung up. So an abandoned attempt costs its input plus its partial output, and then the replacement costs a full call on top. Those aborted partials plus their replacements add $0.00107 per request, taking the total to $0.03689 and the multiple to 6.5.

The size of that figure is worth reconstructing, because it is the term people assume is negligible and it is not obviously so. The expected number of abandoned attempts per user request is the branch failure rate multiplied by the number of branches, so a 2.1% failure rate over five branches means roughly a tenth of a request-worth of wasted work every time. Each of those tenths costs the abandoned attempt's own input and the 180 output tokens it managed, and then a replacement at full price on top. A small failure rate multiplied by a branch count and by two payments per failure is how a rounding error becomes a line item, and none of the three factors is visible in "five times one call".

The fourth row has no dollar figure because its cost is conditional. Five concurrent requests per user request quintuples the peak submission rate without changing daily volume at all, which is the mechanism from the previous chapter seen from the billing side. It can push you into a higher usage tier, which is usually cheaper per token; it can push you past a limit, which converts spend into latency; and it removes the option of the asynchronous batch endpoint, where work is submitted for completion within hours rather than seconds and whose roughly 50% discount is unavailable to anything that has to answer a user synchronously. Whichever of those applies, the per-token price you multiplied by five is not the price you will pay.

The eight-way bill, line by line#

The contract fan-out has a different shape from the SQL example in one decisive way: every branch is sent the whole 30,000-token agreement, because a branch reviewing indemnity has to read all of it. The agreement is different on every request, so unlike the SQL schema it is not warm in anyone's cache when the fan-out starts.

Follow that through the cache mechanism and the consequence is the opposite of what caching is for. A cache entry exists only after some request has written it, and a write is only committed once that request has been processed. Eight branches submitted within 40 milliseconds all arrive before any of them has committed anything, so all eight miss, and a miss on a prefix marked cacheable is not billed as an ordinary read — it is billed as a cache write, at the raised rate. Eight branches therefore pay eight writes for a prefix that was meant to be paid for once.

Two of those rows move in opposite directions and roughly cancel, which is why the total landed so close to the naive estimate while being composed of completely different quantities.

Output got cheaper. The single pass wrote 4,400 tokens because it covered eight dimensions in one report, and the eight branches write 5,600 between them, so the per-branch saving is real but small.

Input got much more expensive, and not by the factor you would guess. Compute the counterfactual first: eight copies of the baseline input, priced as the baseline priced them, would be $0.7344. The actual input line is $1.0800. The difference, $0.3456, is entirely the price gap between reading 36,000 tokens at cache-read and at cache-write rates, and it is a gap the design created rather than one the volume created.

That is the belief this chapter exists to break. The team had prompt caching enabled and believed it was helping, and the belief was well founded on the baseline, where the rubric prefix was warm from the previous request. Under a simultaneous fan-out the same feature, with the same configuration flag set the same way, inverted: a cache write costs 1.25 times a normal read and every branch paid one. The setting is the quantity that stayed constant; whether it saved money or cost money is the property that varied, and nothing in the configuration reports which way it is currently pointing.

That leaves $0.1953 of genuinely new spend, in two rows. The merge call that reconciles the eight branch outputs is $0.0315 and was in nobody's estimate. The abandoned stream is $0.1638, and it is the largest single surprise on the table: one branch per review, on average, hit the 90-second client timeout while queued behind the quota, was billed for the 500 tokens it had generated, and was re-run. So the queueing from the previous chapter does have a cost consequence, though not the one people assume — the refusals themselves are free, and it is the client timeouts sitting on top of them that cost money.

The change that takes 8.6x down to 2.3x#

Firing the first branch, waiting for it to establish the cache, and then releasing the other seven changes the input line from eight writes to one write and seven reads. At $0.30 per million, a branch's 36,000 input tokens cost $0.0108 instead of $0.1350.

async def fan_out(dimensions: list[str], shared_prefix: list[dict]) -> list[str]:
    # Branch one populates the cache for the shared 36,000-token prefix;
    # the rest read it at 1/12.5 the price. Costs ~1.5 s of wall clock.
    first = asyncio.create_task(run_branch(dimensions[0], shared_prefix))
    await asyncio.sleep(1.5)
    rest = [asyncio.create_task(run_branch(d, shared_prefix)) for d in dimensions[1:]]
    return await asyncio.gather(first, *rest)

Rebuild the input line from those two figures and the arithmetic is short: one write at $0.1350 plus seven reads at $0.0108 each is $0.2106, against $1.0800 before. Add $0.0840 of branch output, $0.0315 for the merge and $0.0396 for the retry — the retry falls too, because the abandoned attempt and its replacement now read a warm cache — and the total is $0.3657 per review, which is 2.32 times the single pass rather than 8.61.

The sleep has to satisfy two constraints from opposite directions. It has to be long enough for the write to be committed, or the seven followers miss and you have paid for the delay and nothing else. And it has to be short enough that the cache entry has not expired by the time the last branch arrives. With a five-minute cache lifetime and a 66-second fan-out, that second constraint is not binding, which is worth checking rather than assuming for a longer fan-out. It also softens the burst that was tripping the rate limit, so it partially addresses the latency problem as a side effect, which is a rare thing in this area and worth taking when it is available.

The accounting worth producing on demand#

The estimate that is worth defending is built from measured traces rather than from a price list, and it is short enough to live in the same module as the orchestrator.

def cost_of(traces: list[BranchTrace], p: Prices) -> dict[str, float]:
    fresh = sum(t.input_tokens for t in traces if not t.cache_hit)
    cached = sum(t.input_tokens for t in traces if t.cache_hit)
    return {
        "input_uncached": fresh * p.cache_write / 1e6,
        "input_cached": cached * p.cache_read / 1e6,
        "output": sum(t.output_tokens for t in traces) * p.output / 1e6,
        "aggregation": aggregation_cost(traces, p),
        "abandoned": sum(
            t.output_tokens * p.output / 1e6 + t.input_tokens * p.cache_write / 1e6
            for t in traces if t.abandoned
        ),
        "peak_concurrent": max_in_flight(traces),  # not a cost; a risk flag
    }

Three details in that function are the ones people leave out. Input tokens are separated by cache status rather than summed, because the two are priced 12.5 times apart and a fan-out changes which side of that line you are on — summing them first would average a rate that varies by more than an order of magnitude. Abandoned attempts are counted, which requires the orchestrator to record them rather than discard them on retry, and a retry loop that overwrites its own trace has already destroyed the evidence. And peak concurrency is reported alongside the money even though it is not money, because it is the quantity that determines whether the next tenfold in volume behaves linearly.

Report the result per successful user request, not per model call. A design where 4% of requests fail the aggregation step and are retried end to end costs 4% more than its per-call arithmetic says, and it is the per-request figure that finance and capacity planning both need.

Where staggering buys nothing, and the step this leaves standing#

The stagger works on the contract reviewer because every branch shares an identical 36,000-token prefix, so there is a large object whose processing can be paid for once and read seven times. In a design where each branch reads a different slice of the document, there is nothing large to share. The input line is small to begin with, since the slices together are the document, and staggering buys nothing while costing 1.5 seconds. The property that decides is the one the first chapter named: whether the input splits or the subtask splits. Slice-sectioning has no shared prefix worth caching; subtask-sectioning has one and is where the whole of this section's saving lives.

The same qualification applies to voting, and more strongly. Voting always re-sends the whole prompt, so it always has a shared prefix, which means the stagger is always available — and it is also the arrangement where the shared prefix is most likely to be warm already from the previous user request, in which case there is nothing to stagger for.

That leaves the aggregation step, which this chapter has priced and not examined. The merge call is $0.0315 on the contract review and the judge call is $0.00747 on the SQL feature, which makes the aggregation step somewhere between 2% and 13% of the total in these two designs. Having a price for it is useful and does not tell you whether it works.

Counting votes over five SQL queries assumes you can tell when two queries are the same answer, and five correct queries written five different ways are five different strings. The next chapter builds the aggregation step properly: the normalisation that makes counting possible, the equivalence rule for outputs that cannot be normalised, and what the code should do when five samples produce five different answers, or two answers each with two votes.