Three passes or three samples: the same budget, different bets

The write-up that went to the platform review had two numbers in it: the gated loop scored 74.6 against a single-pass baseline of 70.6 on a fresh 400-ticket set, at an average of 3.1 model calls per request. Four points for triple the calls, on a system serving 40,000 tickets a day, is a real question rather than an obvious yes, and the first person to read it asked what else three calls buys.

The gap is smaller than the 300-ticket set from chapter 2 suggested, and the reason belongs in the write-up rather than in a footnote. That set had been frozen for months while the critic prompt, the finding taxonomy, the gate margin and the veto list were all tuned against it, so the 10.8-point figure includes an unknown amount of fitting to 300 particular tickets. Drawing 400 fresh tickets from the same traffic and rerunning both arms without touching anything gives 4.0 points, and that is the number a decision should be made on. Reporting both, with the explanation, is more persuasive than reporting the larger one.

The mechanism behind that shrinkage is worth a sentence, since it is the commonest way a good measurement turns into a bad one. Every tuning decision made while looking at a set is a choice among options, and the option chosen is the one that scored best on those particular cases. Some of that advantage is real and transfers; some of it is the set's own idiosyncrasies being fitted, and that part transfers to nothing. Nobody can tell the two apart from inside the set, because inside the set they look identical. A fresh draw from the same traffic separates them, and the difference between the two figures is a lower bound on how much fitting happened — a lower bound because the fresh set is one draw too, and it has its own idiosyncrasies pulling in an unknown direction.

The same budget spent in parallel#

Three model calls can be arranged as one draft plus one critique plus one revision, or as three independent drafts with a step that picks one. The second arrangement uses the same selector machinery from chapter 3, applied to choose among candidates rather than to gate a replacement.

The shapes differ in more than topology. The serial arm's calls are dependent, so latency is the sum of them and a slow critique delays everything after it. The parallel arm's calls are independent, so latency is the maximum of the three plus the selector, which on the payments drafter meant 5.4 seconds at p50 against 13.6 for the loop. The serial arm keeps one line of reasoning and improves it; the parallel arm draws three lines of reasoning and discards two, which is wasteful in tokens and is the source of its advantage, because the three drafts differ in ways that a revision of a single draft cannot reproduce.

Sum against maximum is the whole of the latency story, and it has a consequence people usually get backwards. Adding a fourth pass to the serial arm adds that pass's full duration to every request. Adding a fourth sample to the parallel arm adds nothing at the median and a little at the tail, because a fourth draw only shows up in the answer time when it happens to be the slowest of the four. So the two arms scale differently in the direction anyone would want to scale them. The catch sits in the same arithmetic read carefully: the maximum of three draws is slower than one draw, so the parallel arm's 5.4 seconds against the single-pass 4.2 is not the selector's cost. Part of that difference of just over a second is the cost of waiting for the slowest of three, and the table cannot say how it divides, which makes the whole difference an upper bound on what the selector adds rather than a measurement of it.

The parallel arm is also the smaller piece of code, which is part of its appeal when there is no evidence yet about which arm suits a task.

async def sample_and_select(ticket, n: int = 3, temperature: float = 0.9):
    drafts = await asyncio.gather(*[
        generate(ticket, temperature=temperature) for _ in range(n)
    ])
    survivors = [d for d in drafts if not [c for c in HARD_CHECKS if not c.ok(ticket, d)]]
    if not survivors:
        survivors = drafts                       # nothing passed; judge them all
    scored = [(cross_family_judge(ticket, d), d) for d in survivors]
    return max(scored, key=lambda pair: pair[0])[1]

The hard checks from chapter 3 do double duty here as a filter before scoring rather than as a veto on a replacement, which removes candidates that dropped a currency figure or a required section without spending a judge call on them. Everything else in the parallel arm is the selector, and the selector is the same component the gate uses, so a team that has built one arm has most of the other.

The selector admits two arrangements that look alike and cost differently, and the difference is worth pricing before reading any call count. One arrangement scores each candidate on its own and takes the highest, which is one judge call per surviving candidate. The other shows all the candidates to a single call and asks which is best, which is one judge call whatever the candidate count. The first gives an absolute score per candidate, which is comparable across requests and is what a gate needs. The second gives only a ranking within this request, which is all a selector needs, and it lets the judge compare the candidates against each other rather than each against a rubric. The call counts in the table below price the second arrangement, so a build that reaches for the first will find its arm costs more calls than the comparison promised.

Both arms were measured on the same fresh 400 tickets, with the parallel arm sampling at temperature 0.9 and selecting with the same cross-family rubric judge the gate uses.

Sampling wins on the mean by 1.3 points and loses on the 10th percentile by 2, at a third of the latency and slightly more cost, because the selector is a fourth call and three full drafts cost more output tokens than a critique and a revision do. Those two columns disagreeing is the useful part of the table: the loop's gate is a floor, so it protects the bad tail, and sampling has no floor because the selector can pick a bad draft when all three are bad.

The regression column says the same thing in the units that matter and is worth reading as gaps. Sampling regresses against the first draft at twice the loop's rate, and the hybrid in the last row sits between the two — which is what a floor applied after a floorless step produces, since running one critique pass on the selector's winner puts a gate back in front of the answer without undoing the choice among three draws. Read as levels these are three small percentages that all look acceptable. Read as gaps they say that the arm with the highest mean is also the arm that damages the most requests, and that adding the gate back recovers most of that at the price of the latency the sampling arm was bought for.

The cost column carries a smaller lesson that is easy to skip. Four calls cost less than four times the single-pass bill, which tells you the four calls are not four of the same thing — the selector reads three drafts and writes a choice, so its output is short where a draft's output is long, and output tokens are the expensive half. That is the general caution about pricing an arm by its call count: a call is not a unit of cost, and two arms with the same call count can differ by a factor if one of them is mostly reading.

The property of the errors decides which arm wins#

Sampling helps when good answers are reachable by the model and it does not always reach them, which is to say when the failure is variance. If the model would produce a correct, well-structured answer on four attempts out of ten, then three samples find one with probability 0.78, and the selector's job is only to recognise it. Revision helps when the failure is systematic — a misunderstood instruction, a missing constraint, a structural problem the model can detect once it is looking at its own output — because a second look at a specific draft addresses something a fresh attempt would reproduce.

That 0.78 is worth deriving rather than accepting, because the derivation is what shows when it stops being true. Four attempts in ten succeed, so six in ten fail. Three draws all failing means the first fails and the second fails and the third fails, and when the draws are independent the probability of all three is the probability of one multiplied by itself three times — six tenths cubed, which is a little over two in a hundred. Everything else is the complement: if all three fail about two times in a hundred, at least one succeeds the other ninety-eight, and 0.78 falls out once the arithmetic is done at four in ten rather than the rounder figures. What the derivation licenses is a rule of thumb: the failure rate falls as a power of the number of draws, so the first extra draw buys the most and each further one buys less.

Two conditions sit under that and both fail in practice more often than the arithmetic suggests. The multiplication needs the draws to be independent, and three samples from one model at one temperature are not independent in the sense that matters — they are three draws from one distribution, so they share whatever that distribution is centred on. And the 0.78 is the probability that a good draft exists among the three, which converts into a good answer only if the selector recognises it. A selector that picks correctly nine times in ten turns a 0.78 into something lower, and the table's regression column is where that shows up.

The failure both arms share is the second row. Every sample of a ticket whose retrieved policy says three business days will say five, and every pass of a critique loop will leave it there, because the error is in the weights rather than in the draw. Nothing about the arrangement of calls addresses that, and the fix is the same one chapter 3 named: information from outside, in this case retrieval verification catching the unsupported number, which is why the verification signal is worth more than either arm's structure.

That row is also where the comparison in this chapter is most easily misread, and the misreading is worth naming because it is what the write-up's framing invites. Both arms were matched on budget, so the budget is the quantity being held constant across the comparison. The property that decides which arm wins is which row of that table a task's errors fall in, and nothing about a call count carries any information about that. So a team can match budgets scrupulously, run the comparison correctly, and still be reading a result whose cause is not in the experiment — the answer will be about this traffic's mix of variance and misunderstanding, and it will change when the mix does.

Where the comparison stops being valid#

The hybrid in the fourth row is the natural response and it is a real configuration rather than a compromise, since sampling addresses variance and revision addresses local defects, and those are different failures. It scores best on the table and it costs 4.2 calls, so it fails the stated budget of three. Under a hard budget of three calls, the honest options are two samples with a selector, or a two-pass loop, and the table above does not contain either — which is the general lesson about these comparisons. The arm you measure has to be the arm you can afford, at the call count you actually have.

Two numbers are not a decision document. The per-pass curve belongs in it, because a reader needs to see that the gain is exhausted by pass three, otherwise the obvious follow-up question is whether five passes would give eight points. The alternative arm belongs in it, because 3.1x the calls has other uses and a proposal that does not compare against them is asking for a budget without justifying it. Latency at p50 and p95 belongs in it, since a 13.6-second p50 changes the product's interaction design in a way a cost table does not show. The 10th percentile and the regression rate belong in it, because the arms differ there in the opposite direction to the mean and a support system usually cares more about the floor. And the split between the tuned set and the fresh set belongs in it, with both numbers, so that nobody discovers the gap later and stops trusting the rest.

Where this is going, in the literature and in production systems, is the merger of the two arms under an external verifier. Sampling with a verifier that scores candidates, and search over partial drafts guided by that verifier, which means extending several half-finished answers a step at a time and keeping the ones the verifier rates highest, are the same idea at different granularities, and both improve with the quality of the verifier rather than with the number of samples past a certain point. Reflexion is the reference implementation of the serial arm with an environment signal in the loop, and it is worth reading alongside LangChain's Reflection Agents, which builds basic reflection and Reflexion side by side and makes the difference between them concrete in code. What none of that literature offers is a way to make the unaided loop keep improving, which remains the honest position: self-critique is an expensive heuristic that buys a couple of passes of real gain and then needs something from outside to continue.

Running one of these on your own task takes an afternoon, and the order that wastes the least of it is to build the ledger first, measure the spread between three independent samples, then run the per-pass curve with a gate, and only then decide which arm to ship — with all four columns in the write-up rather than the one that supports the answer you already like.

That order is not arbitrary; each step makes the next one cheaper or unnecessary. The ledger is what lets any of the rest be measured at all. The spread costs an hour and can end the question outright, since a task whose independent draws barely differ has nothing for a selector to choose between. The per-pass curve is the expensive measurement and it is worth its cost only once you know sampling is not the obvious answer. And the choice of arm comes last because it is the only step that depends on all three.