Nine points of improvement can be nine points of extra compute

The paper reports 80.4% against a baseline of 71.2% on a 500-question reasoning benchmark, and the method section describes a four-stage pipeline: decompose the question into sub-questions, answer each, generate a critique of the combined answer, then revise. The baseline in Table 2 is the same model, one call, "Let's think step by step." Nine point two points is a large number in that literature, the method has a memorable name, and by the time it reaches your team's channel it has been compressed to "this prompting technique adds 9 points."

The comparison as run is between one model call and five. Whatever else the pipeline is doing, it is spending roughly five times the inference to answer the same question, and inference spent is itself a known way to raise accuracy on reasoning benchmarks. Sampling the same model five times at temperature 0.7 and taking the majority answer — self-consistency, which has been standard since 2022 — costs about the same and requires no new prompt engineering at all. Until the paper reports that number, the result establishes that five calls beat one call, which was not in doubt.

Matching the budget, and deciding which budget#

"Compute-matched" needs a unit, and the four candidates disagree with each other often enough that the choice is part of the argument. The pipeline above issues 5 calls, but its calls are short, and a single long chain-of-thought response can burn more output tokens than all five together. Which axis you match on depends on what constrains you in production.

The pipeline costs 6.1× the baseline in dollars and 3.1× in wall-clock, and those two ratios point at different fair comparisons. On dollars, the honest baseline is self-consistency with six samples or a model tier that costs six times as much per token. On latency, the honest baseline is a single call given three times the thinking budget, since a long response is serial in a way that six parallel samples are not.

Running those on the same 500 questions is a few hours of work and produces the table the paper should have contained.

The pipeline still wins, by 1.5 points over self-consistency at 5 samples and by 0.3 points over 8 samples at three quarters of the cost. That is a real if modest result, and it is a completely different result from the one in the abstract. A 1.5-point advantage over a baseline anyone can implement in twenty lines invites a different question about whether the four-stage pipeline, its prompt templates and its failure modes are worth maintaining, and it sits close enough to the noise floor that the next chapter's concerns become load-bearing rather than pedantic.

The arithmetic that produces those cost figures is worth having as code, because it is the part teams skip and then argue about from memory.

PRICES = {  # USD per million tokens
    "mid":   {"in": 3.00,  "out": 15.00},
    "large": {"in": 15.00, "out": 75.00},
}

def cost_per_question(tier, calls):
    """calls: list of (input_tokens, output_tokens) for one question."""
    p = PRICES[tier]
    return sum(i * p["in"] + o * p["out"] for i, o in calls) / 1_000_000

pipeline = [(620, 210), (740, 380), (900, 320), (880, 330)] + [(620, 180)]
baseline = [(620, 180)]

print(cost_per_question("mid", pipeline))   # 0.02797
print(cost_per_question("mid", baseline))   # 0.00456

# What budget does the baseline get in a fair fight?
budget = cost_per_question("mid", pipeline)
samples = int(budget // cost_per_question("mid", baseline))
print(samples)                              # 6  -> self-consistency @ 6

# Or the same dollars spent on a larger model, one call:
print(cost_per_question("large", baseline)) # 0.02280  -> affordable, test it

Most papers do not report token counts, so the inputs to that function have to be estimated, and an estimate good to within 30% is enough to change a decision. The method section tells you how many calls each question triggers; the prompt templates in the appendix tell you the fixed input size, usually a few hundred tokens; and the output size can be read off any example trace the authors print, or approximated at four characters per token from the sample outputs in the figures. A pipeline that re-sends the original question and all prior stage outputs to each subsequent stage has input tokens growing roughly quadratically in stage count, which is why the 3,140 figure above is five times the baseline's input rather than five times a smaller number. Where the paper gives neither traces nor templates, the absence is itself informative, because a method whose cost cannot be estimated from its own description is a method the authors did not consider as an operational proposal.

That last line matters as much as the self-consistency line. Spending the pipeline's budget on one call to a model tier five times more expensive is a legitimate use of the same money, and on tasks where the bottleneck is knowledge rather than search, it frequently beats any amount of orchestration around a weaker model. A paper proposing a scaffold has, implicitly, made a claim that structure buys more than capability at equal cost, and the table above is where that claim gets tested.

Why the missing baseline is missing#

The norm being applied here is old and comes from outside prompting research. Comparing architectures at matched FLOPs — floating-point operations, the unit in which the arithmetic cost of a training run is counted — rather than at matched parameter counts became standard in the scaling-law literature because parameter count and training compute were confounded, and any architecture given more of either would win. Test-time compute is the same confound moved to inference. Once you can trade dollars for accuracy by sampling more, every method that samples more inherits a gain it did not earn, and a substantial share of the prompting techniques published between 2022 and 2024 shrank to within a point or two of self-consistency once anyone ran the matched comparison. "Quantifying Independently Reproducible Machine Learning" is worth reading on which paper-level signals predict this kind of survival.

The specific baseline that dissolves the most results is self-consistency, and it is worth understanding why it is so hard to beat. Sampling the same model several times and taking the modal answer exploits the fact that a model's errors on a reasoning problem are more scattered than its successes: wrong answers disagree with each other, correct answers coincide. Any method that improves accuracy by making the model try more than once is, at some level of description, doing the same thing with more machinery, and the machinery has to earn its complexity against a twenty-line majority vote. Methods that genuinely beat it tend to do so by adding information the model did not have — an executed test, a database schema check, a retrieved document — rather than by rearranging the model's own output.

Authors do not usually omit the baseline to deceive. Running self-consistency at six samples on the full benchmark costs them real money and produces a table that makes their contribution look small, and the incentive structure of a conference deadline does the rest. The more common and more forgivable version is baseline neglect, where the proposed method's prompts went through forty iterations and the baseline's prompt is one line copied from a 2022 paper. Underspecifying the baseline is the cheapest way to manufacture a gap, and it is invisible from the abstract.

The first three checks, in order#

Given a claim of nine points, the ordering that finds problems fastest is: what did the baseline get, what did the baseline get at the same budget, and how large is the run-to-run variation on the same configuration.

The first is a reading task and takes ten minutes. Find the exact baseline configuration — model version, prompt text, temperature, shot count — and check that it is the configuration a competent engineer would have used. If the baseline prompt is in the appendix and is a single sentence while the method's prompt is a page, the gap includes prompt engineering effort that has nothing to do with the mechanism.

The second is the compute-matched table above, and it is the one that most often collapses the result, because it is both the most consequential omission and the easiest for authors to leave out. If the paper reports no cost, count calls from the method description and estimate tokens; an order-of-magnitude estimate is enough to tell whether you are looking at 1.2× or 6×.

The third is variance, which the next chapter takes further. A 500-question benchmark scored at 80% has a standard error of about 1.8 points from sampling alone, the standard error here being the spread you would expect purely from having drawn those particular 500 questions rather than 500 others of the same kind, so two configurations differing by 1.5 points are not distinguishable on that set without repeated runs. A single-seed comparison — one run of each configuration, where the seed is the number that fixes the random draws so that a rerun reproduces them exactly — of a stochastic pipeline against a greedy baseline is not a comparison of methods at all.

There is a fourth check that belongs just after these three and is often decisive for adoption rather than for validity, which is whether the gain concentrates in a slice of the benchmark you do not care about. A nine-point average over a set that is 40% grade-school arithmetic and 60% multi-step word problems can be twenty points on the arithmetic and nothing on the rest, and if your traffic contains no arithmetic the method is worth zero to you while being entirely real. Papers that break results down by subset make this visible; papers that report one aggregate number are asking you to assume the effect is uniform, which it almost never is.

The reflex of checking first whether the code is released, whether it was peer reviewed — read and judged by anonymous researchers in the same field before a journal or conference accepted it, which most preprints never are — and whether others have replicated it is not wrong so much as slow and weakly correlated with the thing you care about. Peer review at machine-learning venues does not reliably enforce compute-matched baselines; released code tells you the method runs, not that the comparison was fair; and waiting for third-party replication means waiting six to eighteen months, by which point the technique has either been absorbed into everyone's stack or forgotten. Reading Table 2 with a calculator is available today and rules out more candidates than all three.

Where compute matching is the wrong objection#

Some methods are cheaper than the baseline, and for those the whole framing inverts. A paper proposing a router that sends 80% of questions to a small model and 20% to a large one is claiming to hold accuracy while cutting cost, so the fair comparison holds accuracy fixed and compares dollars, and demanding a compute-matched baseline is a category error.

Compute matching also does not settle the question when the shape of the compute differs. Six parallel samples and one four-stage pipeline may cost the same dollars while behaving differently under load: parallel samples multiply your requests-per-minute against a rate limit and finish in one round trip, whereas a four-stage pipeline is four sequential round trips with four opportunities to time out and a partial-failure story you have to write. If your service has a 3-second budget, the 4.9-second pipeline is disqualified whatever its accuracy, and if your rate limit is the binding constraint, self-consistency at eight samples may be the thing you cannot afford. Matching on the axis that binds you is the point, and the axis that binds you is a property of your system rather than of the paper.

Cache behaviour can also legitimately break the symmetry. A four-stage pipeline that re-sends the same 620-token system prompt and document context to every stage is a good candidate for prompt caching, which on most providers discounts repeated input tokens heavily, so the $0.0280 above may be $0.0180 in a steady-state service while self-consistency's parallel samples share the same prefix and get the same discount. Recomputing the table with your provider's cache pricing occasionally reverses the ordering, and it is a legitimate thing to do as long as both sides of the comparison get the same treatment.

Finally, a technique that survives compute matching by a point and a half has not been shown to be worth adopting; it has been shown to be worth investigating. The 1.5 points in the table above came from one run of one benchmark, and a benchmark is a fixed set of questions that a model may have encountered before, scored by a metric someone chose while watching the number.