Reproducing at small scale surfaces what the paper left out

Two engineers spent three days building the decomposition stage — the one the ablation table suggested was carrying most of the effect — and ran it against 200 questions drawn from their own logged traffic rather than the benchmark. The baseline scored 71.9%, the decomposition stage scored 74.1%, and the paper had implied something closer to seven points for that stage alone. The instinct in the room was to argue about whether 2.2 points meant the method did not transfer. The more useful output of those three days was a list of eleven things the paper had not said, six of which changed the number.

Reproduction at small scale is not primarily a verification exercise. The published result is usually approximately right about its own setting, and confirming it tells you little you did not already suspect. What the exercise produces is an inventory of the decisions the authors made and did not report, each of which you will have to make too, and any of which can be worth more than the effect under study.

Scoping it so it takes three days and not three weeks#

The reproduction that gets done is the one scoped to the smallest faithful version of the mechanism, which chapter 3's ablation reading usually identifies for you. Rebuilding all four stages of that pipeline would have taken a fortnight and produced a single number confounded across four components; rebuilding the one stage the ablation table credited with most of the effect took three days and produced an interpretable one. Where no ablation exists, the rule of thumb is to implement the component that the paper's own narrative treats as novel and stub everything else with your current system's behaviour.

Sample size follows from the effect you are trying to detect rather than from what feels thorough. Detecting a 3-point difference around 72% accuracy with reasonable confidence needs a few hundred items and several seeds, which at 200 questions times 5 seeds times two configurations is 2,000 model calls per stage; detecting a 10-point difference needs far fewer. Running 5,000 questions once is a worse design than running 200 questions five times, because the seed variance is usually the larger term and only repetition can see it.

The data should be yours. Reproducing on the paper's benchmark answers whether you implemented the paper correctly, which is a debugging question, whereas reproducing on 200 questions from your own logs answers whether the mechanism helps you, which is the decision question. Doing both is ideal and doing only the first is the common mistake, because it ends with a number that agrees with the paper and still says nothing about production.

The eleven things, and the six that mattered#

The prompt was the first. Appendix C of the paper contained the decomposition template, and the template included three worked exemplars, each a question-and-decomposition pair taken from the benchmark's training split. The baseline in Table 2 used a zero-shot prompt. Three exemplars against zero is a difference that has nothing to do with decomposition, and matching the baseline's shot count closed 1.1 points of the reported gap on its own.

The second was assembly. The method section said sub-answers are "combined and passed to the final stage," which turned out, in the released code, to mean the original question is restated above the sub-answers with an instruction to prefer them where they conflict. Dropping that restatement — the obvious reading of "combined" — cost 3.4 points in the team's own ablation. A single sentence of prompt scaffolding was worth more than the entire critique stage.

The third was what happens when a stage fails. In 4.1% of questions the decomposition came back as prose rather than a numbered list and could not be parsed, and the authors' code caught the exception and fell back to a single call. That fallback is never mentioned in the paper, and it means the reported 80.4% is a blend of the method on 95.9% of questions and the baseline on the rest. It also means any implementation without the fallback will look worse than the paper for reasons unrelated to the idea.

[repro] q_id=1183 stage=decompose attempt=1
  response: "To solve this, we first need to know how many crates were
             delivered on Tuesday, then work out the shortfall..."
  parse: FAILED (expected numbered list, found prose)
  action: fallback -> single_call
  final: correct
[repro] parse failure rate: 41/1000 samples (4.1%), of which 27 correct after fallback

The fourth was answer extraction. The team's harness pulled the first number from the model's response; the released evaluation script pulled the last. On chain-of-thought outputs that restate intermediate quantities, this is not a small difference, and it accounted for 1.8 points of the measured gap, in the paper's favour once corrected.

The fifth was leakage, of a mild but real kind. Two of the three exemplars in Appendix C appear, with different numbers, among the benchmark's test items in the split the paper evaluates on. That is not the wholesale contamination of the previous chapter; it is the ordinary porousness of a dataset where train and test were drawn from the same generator, and it inflates the method's number by an amount nobody can cleanly estimate.

The sixth was variance, which the team measured properly because the previous day's work had made them suspicious. Five seeds of the decomposition configuration on the same 200 questions ranged from 72.8% to 76.0%.

The remaining five findings changed no numbers and were still worth having: the model version the authors used was deprecated four months ago, temperature was unreported and inferred from the code as 0.7, the benchmark's own answer key contains at least three errors, one prompt contains a typo that a reimplementation would silently correct, and the released code pins a tokenizer version that no longer resolves. None of that is negligence on anyone's part. It is what the literature on reproducibility barriers describes as the ordinary gap between a paper as a scientific argument and a paper as a set of instructions, and it is the reason the "Reproducibility in Machine Learning-based Research: Barriers and Drivers" work treats reporting completeness as a measurable property rather than a virtue.

What the reproduction is actually for#

The output of three days is not a verdict but a reconciliation, and it is worth writing in a form where each line is either an explanation or an admitted unknown.

Roughly half the reported improvement, arrived at this way, is an informative result rather than a disappointing one. It says the mechanism is doing something on your data, that its magnitude is a few points, and that a specific fraction of the shortfall is explained by choices you have now made deliberately. What remains unexplained is bounded and named.

Making that reconciliation possible requires recording the run in a way that a colleague can rerun in six weeks, which means a manifest rather than a notebook.

run: decompose-repro-2026-07-14
paper: arXiv:2503.01188v2, Table 4 row "decompose+answer"
model: claude-mid-2026-05-01          # paper used a build deprecated 2026-03
temperature: 0.7                      # inferred from released code, unreported in paper
seeds: [11, 23, 37, 41, 59]
dataset:
  source: internal_analytics_questions_v3
  n: 200
  sampling: stratified by question_type, drawn from 2026-05 traffic
prompts:
  decompose: sha256:8f31...c2  # appendix C verbatim, 3 exemplars retained
  baseline:  sha256:1ab9...7e  # matched to 3 exemplars, same style
harness:
  answer_extraction: last_number     # matches authors' eval script
  on_parse_failure: fallback_single_call
results:
  baseline: [71.4, 72.3, 71.9, 70.8, 72.9]
  method:   [74.6, 76.0, 74.1, 72.8, 75.4]
  delta_mean: 3.1
  delta_sd: 1.4

Every field in that manifest exists because a discrepancy turned on it. The two prompt hashes are there because the shot-count mismatch was the largest single correction; on_parse_failure is there because omitting it makes the method look 4.1% worse for reasons that are not about the method; the model line is there because the paper's model no longer exists and any comparison is therefore approximate in a direction nobody can quantify.

What to say when you have half the number#

The conclusion that a paper is overstated and should not be pursued is the wrong one from this evidence, and it is the one most likely to be offered. It conflates three distinct claims: that the authors' number is wrong, that the mechanism does not work, and that adopting it is not worth the effort. The reproduction speaks to none of the first two and only partially to the third.

What can be said with confidence is that on 200 questions from your own traffic, with five seeds, the mechanism produced 3.1 points ± 1.4 at 2.4× the inference cost of the baseline, that four specific differences from the paper's setup have been identified and controlled, and that two remaining differences — the exemplar overlap in their split and the distribution gap between their benchmark and your traffic — are unresolved and would take real work to close. That last point cuts both ways: your traffic may be harder in ways that suppress the effect, or your baseline may be stronger than theirs because your prompts have been tuned for eighteen months.

What cannot be said is that the paper failed to replicate. A reproduction at n=200 with one implementation by two engineers over three days is a weaker measurement than the original in sample size and in tuning effort, and treating it as an authoritative refutation inverts the epistemics. It is also worth saying explicitly that no reproduction of this kind can establish absence: if you find nothing, the most likely explanations are that you built it slightly differently, that your data does not exercise the mechanism, or that the effect is real and smaller than your noise floor.

The framing that holds up in a room is that the reproduction converted a claim about someone else's benchmark into a measurement on your data with an error bar, at a cost of three engineer-days, and that the measurement is now the input to a decision rather than the decision itself. Three points at 2.4× cost is a number people can argue about productively, in a way that "the paper says nine" never was.

Two further things belong in that conversation because they change what people do next. The first is which of the unexplained differences is worth closing and at what price: re-running on the paper's own benchmark rather than your traffic would take a day and would separate "we implemented it differently" from "our data is different," which is the single most valuable remaining experiment. The second is the finding about the restatement sentence, which is independent of whether the pipeline is ever adopted — a 3.4-point effect from one line of prompt scaffolding applies to several other places in your system, and it is the kind of transferable detail that only ever appears from building something. Reproductions frequently pay for themselves in these side findings even when the headline mechanism is dropped.

There are also papers not worth reproducing at all, and recognising them saves the three days. If the mechanism requires training on data you do not have, if it depends on model internals your provider does not expose, or if the pre-test from chapter 1 already showed your traffic has no headroom for it, a reproduction can only tell you what you know. The same applies when the reported effect is smaller than the variance you would be able to measure at your feasible sample size: attempting to detect a 1.5-point difference with 150 questions and two seeds produces a number that will be over-interpreted in whichever direction the team was already leaning.

A small reproduction is also not a production experiment. It runs offline, on a curated sample, with no latency budget, no concurrency, no cache behaviour and no users changing what they ask because the system got better. What it gives you is enough evidence to decide whether to spend the far larger sum that a production trial costs — and the harder version of that decision is the one where the evidence points at no while most of the team wants yes.