Marginal cost per accuracy point

A cost multiplier travels well and decides nothing. Five times the calls sounds expensive next to five points of accuracy, or cheap, depending on who is speaking, and neither reading is a number anyone can act on. What converts it into one is two divisions: marginal spend divided by accuracy points, which prices the improvement, and marginal spend divided by errors avoided, which prices it against the thing being improved.

The second division is the one that gets skipped, and it is where most sampling proposals are decided in practice without anyone noticing the decision was made. An error's cost is not a property of the model; it is a property of what happens to the error after it leaves the service. The same five points are worth three hundred dollars a day or three hundred and sixty thousand depending entirely on whether a downstream check catches wrong answers, a reviewer was going to read every record anyway, or the answer goes to a customer.

The marginal cost of a point, computed end to end#

Start from tokens rather than from the multiplier, because the multiplier hides where the money goes. Take a document service where each call carries roughly 1,100 input tokens — the document, the instructions and three few-shot examples — and produces about 220 output tokens of short reasoning plus a structured answer. At $0.80 per million input tokens and $4.00 per million output, one call costs $0.00088 in input and $0.00088 in output, or $0.00176 all in. Those two figures come off a rate card, the vendor's published price per million tokens in and per million tokens out.

Each of those three figures is one multiplication, and doing them in the open is worth the four lines because a rate card is quoted in a unit no request ever uses. A price of $0.80 per million input tokens means one input token costs that price divided by a million, so the input side of a call is 1,100 tokens at that per-token price, which comes to $0.00088. The output side is 220 tokens at $4.00 per million, which comes to $0.00088 as well. Adding the two sides gives $0.00176 for one call, and adding rather than taking the larger of the two is the right operation because the vendor bills both directions of the same call.

The two halves landing on the same figure is arithmetic rather than coincidence, and seeing why is worth more than the number itself. Input runs five times as many tokens as output here, 1,100 against 220, while the output rate is five times the input rate, $4.00 against $0.80. Five times as many tokens at a fifth of the price per token is the same money. That balance is a property of this workload's shape rather than a general fact: a service that ships long generations off a short prompt has an output side that dominates the bill outright, and the same rate card produces a completely different split.

RATE_IN, RATE_OUT = 0.80 / 1e6, 4.00 / 1e6

def daily_cost(requests: int, n: int, in_tok: int, out_tok: int,
               retry_rate: float = 0.0, discard_rate: float = 0.0) -> float:
    """Generation spend per day, counting retried and discarded samples."""
    calls = requests * n * (1 + retry_rate) * (1 + discard_rate)
    return calls * (in_tok * RATE_IN + out_tok * RATE_OUT)

base = daily_cost(600_000, 1, 1_100, 220)                     # 1,056.00
voted = daily_cost(600_000, 5, 1_100, 220, retry_rate=0.018,
                   discard_rate=0.024)                        # 5,509.94
points = 89.0 - 84.0
print(f"${(voted - base) / points:,.0f} per point per day")   # 890.79

The function multiplies the two loss rates rather than adding them, and that ordering is most of its accuracy. Start from the calls a fixed-N design intends to make, which is requests times N. Retries are a proportion of those intended calls, so 1 + retry_rate scales the count up to include them. Discards are a proportion of everything generated so far, retries included, which is why 1 + discard_rate multiplies the already-inflated figure instead of being added to the first rate. Multiplying the resulting call count by the per-call price then converts calls into money in a single step.

The last two lines are the divisions the whole chapter turns on. voted - base isolates the marginal spend — what the fan-out costs over what the service was paying already — because the baseline was going to be spent whether or not anybody approved N=5, and charging it to the proposal would make every proposal look worse than it is. Dividing that difference by points, the accuracy at N=5 minus the accuracy at N=1, converts a sum of money into a price for the thing the money bought.

That comes to about $891 a day per accuracy point, or $325,000 a year per point, and $1.63 million a year for the five. The retry and discard rates are not decoration. A service that retries on 5xx and on responses failing schema validation runs those together at some rate — 1.8% here — and it discards samples whose output is truncated at the token limit before they reach the vote, which ran at 2.4% and had been invisible because discarded samples were logged at debug level. Those two lines add about $220 a day that nobody had counted, and the discard rate has a second effect worth noticing: a case where two of five samples are discarded is voted at N=3 without anything in the metrics saying so.

That second effect is worth following through, because it is a correctness problem wearing a cost label. A truncated sample is dropped before the aggregator ever sees it. The aggregator counts whatever does arrive and takes the plurality of that. So a case that lost two samples is decided by three votes on a service whose accuracy was measured at five, so whatever gap the accuracy curve showed between three and five is being paid on those cases without appearing in any metric. The cases that lose samples are not a random selection either: truncation happens on the longest outputs, which are the ones where the model had the most to say, which skews towards the harder documents. The setting drops silently on exactly the traffic that needed it most.

The full accounting, expressed per thousand requests so it can be compared against other line items:

Most of that table is a call count times a per-call price, and the rows add. Retries are 1.8% of the 5,000 generation calls, which is 90 of them; discards are 2.4% of the same 5,000, which is 120; the embedding step makes one batched call per request, which is 1,000. Those three added to the 5,000 give the 6,210 total. The generation-derived cost cells are each their own call count at the $0.00176 worked out above, so 90 calls come to $0.16 and 120 come to $0.21 — while the embedding row is priced off a different and far cheaper rate card, which is why 1,000 calls there cost $0.02 rather than $1.76. The marginal row is the total less the N=1 baseline in both columns, 6,210 less 1,000 calls and $9.19 less $1.76, and dividing the resulting $7.43 by the five accuracy points gives the $1.49 in the last cell.

That $1.49 and the $891 a day from the code block are the same price in different units, and checking that they agree is the cheapest available audit of the whole accounting. There are six hundred thousand-request blocks in a day at this volume, so multiplying the per-thousand marginal by that many blocks recovers the daily figure to within the rounding in both. Two accountings built from different starting points landing on the same number is the evidence that neither has dropped a line item.

Four things get left out of a fan-out's price and all four are recoverable from logs you already have. Retried calls, because the retry policy sits in a different layer from the voting code. Discarded samples, because a sample that never reaches the aggregator was still generated and still billed. Any embedding or normalisation call the aggregation step makes, which is small here and is not always. And the judge call, where the aggregation is a selection rather than a count — a judge prompt carries the original input, the rubric and all N candidates, so it is routinely the longest single prompt in the system rather than a sixth of the bill. A cost model that multiplies one call's price by N is understating every one of those.

The judge call is the one that catches people out, and the reason is a fact about what has to be in its prompt. A vote reads only the answers and needs nothing else in front of it. A judge has to be handed the original input, the rubric it is scoring against, and all N candidate answers, because it cannot compare things it has not been shown. Its input length is therefore the original prompt plus roughly N times the length of an answer, which makes it the longest single prompt in the system rather than one more call of average size. Pricing it as a sixth of a five-way fan-out is not slightly wrong. It is wrong in the direction that grows with N.

Latency deserves the same treatment, and it is the cheaper measurement to make. A p95 moving from 1.4 seconds at N=1 to 2.1 seconds at N=5 means the five points cost 0.7 seconds of tail latency, or 140 milliseconds per point. Whether that is affordable depends entirely on who is waiting, which is a question with a factual answer rather than an opinion: batch traffic does not care, and an interactive screen with a 3-second budget cares a great deal.

Both figures there come from a subtraction and a division. Tail latency at N=5 less tail latency at N=1 is 2.1 seconds less 1.4, giving the 0.7 seconds; sharing 0.7 seconds across the five points bought gives 140 milliseconds per point. The division means something only because both ends are p95s measured the same way — subtracting a p95 from a median, or from a mean, produces a number with no interpretation at all. For the interactive path the level matters more than the increment: 2.1 seconds against a 3-second budget sits uncomfortably close to the edge, and the tail is the part of a distribution that crosses an edge first.

The question that comes before the arithmetic#

The costing above prices the improvement and not the thing being improved. Five points of accuracy on 600,000 requests a day means about 30,000 fewer wrong answers per day: 16% of 600,000 is 96,000, and 11% is 66,000. Divide the marginal spend of about $4,450 a day by that, and the service is paying about $0.148 per error avoided.

Two steps in that are easy to run past. The error rates are one minus the accuracies: a hundred per cent less 84.0% is 16%, and a hundred per cent less 89.0% is 11%. Both rates are then applied to the same 600,000 requests a day, which turns rates into counts of wrong answers, 96,000 and 66,000, and subtracting the counts gives the 30,000 errors a day that the five points remove. Dividing the marginal spend by that count rather than by the five points is what makes the answer comparable against anything outside the eval, because the world outside the eval has opinions about errors and none at all about accuracy points.

The tempting shortcut is to read five points of accuracy as five per cent fewer errors. It is not, and the gap matters at exactly the moment somebody is deciding whether to spend. Five points is five per cent of the requests, and against a base of 96,000 wrong answers it removes just under a third of the errors — so reading it the other way understates the win badly here. On a service whose errors already run below five per cent of traffic the same misreading fails in the opposite direction, because a five-point gain has nowhere to come from.

Now the number is comparable against something.

Every row is the same two multiplications, with one extra factor in the third. The cost of one error, multiplied by the 30,000 errors a day the five points remove, gives the value of avoiding them, which is then set against the $4,450 a day that avoidance costs. The third row needs the escape rate as a separate factor, because an error that reaches a human only when something flags it costs $0.90 when it lands but lands on 15% of occasions, so its expected cost is $0.90 × 0.15 = $0.135. What changes down the column is never the model and never the five points. It is what the system downstream does with a wrong answer: reprocess it automatically, absorb it into a review somebody was doing anyway, or hand it to a customer as a wrong invoice.

The rows differ by three orders of magnitude, and nothing in the eval distinguishes them. The general lesson is in the spread rather than in any row: if only a slice of traffic carries a real error cost, N belongs on that slice and not on the rest.

Adaptive N, and the samples you do not have to buy#

A fixed-N design pays for five samples even on cases where the first three agree unanimously, which on this service was 71% of them. An early-exit rule stops as soon as a decision is arithmetically settled, since once three of five samples agree, no further sample can change the plurality.

The arithmetic behind that quorum is worth checking rather than accepting. With a ceiling of five samples, three agreeing on one answer leaves two still undrawn. Even if both of those two land on the same rival answer, that rival finishes on two votes against three, so the leader cannot be caught. Three is the smallest count with that property at a ceiling of five: at two votes, the three unseen samples could all agree on something else and win. The rule generalises as more than half the ceiling, rounded up, and it is a fact about counting rather than anything about documents.

async def adaptive_vote(prompt, max_n: int = 5, quorum: int = 3) -> dict:
    counts, used = Counter(), 0
    for batch in ([0, 1, 2], [3], [4]):          # 3 in parallel, then top up
        results = await gather(*(call(prompt, seed=i) for i in batch))
        used += len(results)
        counts.update(r.answer for r in results)
        top, votes = counts.most_common(1)[0]
        if votes >= quorum:
            return {"answer": top, "calls": used, "agreement": votes / used}
    return {"answer": counts.most_common(1)[0][0], "calls": used, "agreement": None}

Expected calls per request drop from 5.0 to about 3.6, since 0.71 × 3 + 0.29 × 5 = 3.58, which is a 28% reduction in generation spend. Measured accuracy fell by 0.1 points, inside the repeat spread that the previous chapter's methodology — resampling one stored collection of draws to get a distribution at every setting — reports, because the cases that exit early are the ones where the additional samples would almost certainly have agreed anyway.

That expected-call figure is a weighted average, and it is also a ceiling. The 71% of cases whose first three samples agree unanimously cost three calls each, and the remainder are charged the full five, which is where 0.71 × 3 + 0.29 × 5 = 3.58 comes from; the saving is that figure against the flat 5.0, which is the 28%. But the code tops up one sample at a time, [3] before [4], so a case that reaches quorum on the fourth sample stops at four rather than five. The true expected count is therefore somewhat below 3.58 and the saving somewhat above 28%. Estimating the pessimistic side of a cost saving is the right direction in which to be wrong.

The catch is on the latency side: the design is now two sequential rounds instead of one parallel fan-out, so p95 rises for the 29% of cases that go to the second round. On this service that moved p95 from 2.1 to 2.6 seconds, which was free for the 78% of traffic arriving from an overnight batch pipeline and unacceptable for the 22% arriving from an interactive review screen with a 3-second budget — so the batch path got adaptive N and the interactive path kept the flat fan-out. Cost and latency optimisations pull in opposite directions here, and there is no setting that is best for both.

Why a minority of slow cases moves the headline figure is a question about what a p95 is. The 29% is the complement of the 71% that reach quorum on the first three samples, and every one of those cases now pays for two sequential rounds instead of one. A p95 is the line above which the slowest twentieth of requests sit, and once nearly a third of traffic is slower than the rest, that line falls inside the slow group rather than inside the fast one. The p95 therefore reports the two-round path, which is how it reaches 2.6 seconds while the majority of requests never enter the second round at all.

What a marginal cost per point leaves out#

A marginal cost per point is a comparison between two designs you happen to be considering, and it says nothing about the designs you did not put on the list. The clearest omission is usually the prompt. A two-line edit that moves a bad slice 23 points with zero additional calls is an infinite return by this arithmetic, and no cost model will surface it because prompt fixes do not appear as line items. Before any spend on N is approved, the comparison should include at least one option that costs nothing per request: fixing an ambiguous instruction, adding two examples covering the worst slice, or tightening the output schema so that malformed answers fail loudly instead of being scored as wrong.

The pricing is also static, and the quantity it prices moves. Model prices have fallen substantially and repeatedly, so a marginal cost that is unaffordable this year may be trivial next year, while the latency figures are far stickier because they are set by sequence length and network round trips rather than by the rate card. Reviewing an N decision after a price change or a model upgrade is cheap if the sampling harness is still runnable, and expensive if the numbers in the original proposal were computed by hand in a spreadsheet nobody can find.

Two numbers, then, and they are both divisions. Marginal spend per accuracy point, computed from tokens at your volume with retries and discarded samples counted in, which tells you what the next increment costs. And marginal spend per error avoided, held against what an error does after it leaves the service, which tells you whether any increment is worth buying at all. The first is arithmetic anyone can do in an afternoon. The second requires asking a question outside the team — what happens to a wrong answer — and it is the one that reorders the options, because it is the only one of the two that can return the answer "spend this on 4% of traffic and nothing on the rest".

There is one more assumption underneath all of this. Where voting has nothing to count, the fallback is a judge, and the cost model above treats that judge as one more call with a price. It is also a model with opinions.