A defensible no cites your workload, not the paper's flaws

Three people brought up graph-structured retrieval in the same week. One had seen a thread with four thousand reposts, one had watched a conference talk, and one had already sketched an extraction pipeline in a scratch branch. The proposal was to replace the current retrieval layer for the internal documentation assistant with one that first builds a knowledge graph over the corpus, then answers questions by traversing entities rather than by embedding similarity, on the strength of a paper reporting large gains on multi-hop question answering.

The tech lead's first response was a methodological one, and it was accurate. The evaluation was 90 questions, single seed, with no compute-matched baseline, and the comparison retrieval system was a bare vector search with no reranking, which is not what the team was running. All true, and by the end of the week the effect was that two engineers had stopped bringing ideas to him and one had escalated to the director with a framing about the team not keeping up. The objection was correct and it lost, because an argument about someone else's methodology is an argument about whether the tech lead is more discerning than the people who liked the thread, and that is a status contest rather than a technical one.

Four grounds for a no, in ascending order of durability#

Evidence quality is the weakest ground to stand on, because it is unstable and impersonal. Its importance is not in question. A better paper appears in three months, someone posts a replication, and the objection evaporates without anything about your situation having changed. It also cannot be settled inside your team, since neither side can produce the missing experiment.

The other three grounds are all facts about your system, which means you can measure them, and the person who disagrees has to argue with a number rather than with you.

The fit row is the one that decided this case, and it came from a query the team could have run at any point in the previous month. Sampling 500 logged questions and labelling them by how many documents a correct answer requires gave 78% single-document, 16% requiring two documents from the same product area, and 6% genuinely multi-hop across areas. Graph traversal earns its keep on the last group. Even a spectacular result there — going from 41% to 80% correct on multi-hop questions — moves end-to-end accuracy by 2.3 points, because 6% of traffic multiplied by a 39-point improvement is 2.3 points. Against that, a failure analysis of the 500 showed that 22% of wrong answers traced to a chunking bug that split tables across boundaries, which one engineer could fix in three days.

That labelling exercise is worth describing because it is the piece of work that makes the whole argument possible, and it is not glamorous. Two people spent an afternoon reading 500 sampled questions with their logged answers and tagging each with the number of source documents a correct answer required and, where the answer was wrong, the reason. Sampling was stratified by product area so that a single noisy area could not dominate, and the two labellers overlapped on 80 items to check they agreed, which they did on 71 of them. The result is a table nobody can wave away, and it is reusable: the same 500 labelled questions have since answered three other proposals, because most retrieval arguments turn on the same distribution.

The opportunity cost row needs the same discipline or it becomes rhetorical. "We could be doing something better" is not an argument; "a four-day spike showed a cross-encoder reranker lifting recall@5 from 0.71 to 0.77 on the same 500 questions" is, because it puts a measured alternative on the table at a known price. Naming what the three weeks buys instead is also what stops the conversation being about whether the advocate's idea is good, since two good ideas competing for one engineer is a scheduling problem rather than a judgement on either.

The operational cost row is where enthusiasm usually meets arithmetic. Extraction over 1.2 million documents at roughly $0.011 per document is about $13,200 of inference before anyone measures anything, the graph goes stale as documentation changes at a few thousand edits a week so there is a reindex path to build and pay for, and the traversal adds two sequential lookups to a request path with a 2-second budget. None of that appears in the paper because the paper was not operating a service.

Say it in the order that keeps you in the room#

The same four grounds land differently depending on sequence. Leading with the paper's weaknesses reads as gatekeeping; leading with your own traffic reads as engineering, and it gives the advocate somewhere to go that is not a fight.

A version that has worked: "Multi-hop is 6% of our questions, so the ceiling on this is about two points end to end, and I want to see the traffic number challenged if anyone thinks it is wrong. Standing it up is about fourteen thousand dollars of extraction plus a graph store we would then own, and the same three weeks would clear the chunking bug that is causing a fifth of our failures. The paper's evidence is thin, but that is not the reason I am hesitant — if the traffic mix were different I would want to try this despite the evidence."

The last clause is doing more work than it appears to. It states what would change your mind, which is the difference between a judgement and a reflex, and it concedes that the technique might be good while your situation is wrong for it. People accept "not for us" far more readily than "this is bad," and it happens to be the more accurate claim.

When half the team still wants it#

Sometimes the fit argument does not settle it, either because the traffic measurement is contested or because the advocates believe the technique will change what users ask. Both are reasonable positions and neither can be resolved by discussion. The move that preserves both the decision quality and your standing is a timeboxed trial with the bar agreed in advance, which converts the disagreement into an experiment whose outcome neither side controls.

The pre-registration — a practice borrowed from clinical trials, where what will be measured and what will count as success are filed publicly before any data exists — is the whole mechanism, and it has to be written down before anyone writes code, because a bar agreed afterwards is a bar someone negotiated against a result they had already seen.

trial: graph-retrieval-2026-08
advocate: sofia
sceptic: marcus
question: Does graph traversal beat our current retrieval on OUR questions?
sample:
  n: 300
  source: logged questions, 2026-06 to 2026-07, stratified by product area
  frozen_on: 2026-08-04        # nobody looks at these while building
bar:
  primary: end-to-end answer accuracy +8 points vs current retriever
  guardrail_latency: p95 increase <= 400 ms
  guardrail_cost: cost per query increase <= 30%
  seeds: 3
budget:
  engineer_weeks: 2
  inference_usd: 900
  extraction_scope: one product area (~40k docs), not the full corpus
if_missed:
  action: park, do not revisit for two quarters
  reopen_triggers:
    - multi-hop share of traffic exceeds 15%
    - a replication on a corpus > 500k docs reports the gain with matched baselines
    - extraction cost per document falls below $0.002

Two features of that document matter more than the rest. The bar is set at a level you would genuinely accept, which means if the trial returns +9 points within the guardrails you adopt the technique without further argument, and everyone should be able to tell in advance that you would. A bar nobody could clear is a veto with extra steps, and the team will read it correctly the first time. The other is the reopen triggers, which turn a no into a conditional no with named conditions, so the next person who raises it in six months has something to check rather than a decision to relitigate.

The budget line keeping extraction to one product area is what makes the two-week timebox real. A trial that requires the full production build to produce a number is not a trial, and scoping it to 40,000 documents costs about $440 of extraction rather than $13,200 while still being enough to measure retrieval quality on 300 questions.

What this is not, and where it goes wrong#

A pre-registered trial is not a governance process and does not belong on every proposal. Most ideas cost an afternoon to test, and demanding a written bar for a two-hour experiment is bureaucracy that will make people route around you. The instrument is for proposals with an asymmetry between the cost of trying and the cost of arguing: weeks of work, new infrastructure, or a change that is hard to reverse.

It is also not a way to run out the clock, and attempting to use it that way is visible within one cycle. If the sceptic writes the bar, sets the sample, controls the timeline and then finds reasons the trial was not conclusive, the mechanism has been converted into the thing it was supposed to replace. The protection is that the advocate co-signs the document and runs the trial, while the sceptic's role is to agree the bar and then stay out of it.

The honest limitation is that a timeboxed trial answers a narrow question. Two weeks on 300 frozen questions measures retrieval quality on today's traffic; it cannot tell you whether the capability would change what people ask, which is the advocates' actual claim in cases like this one, and it cannot see effects that only appear at full corpus scale where graph density is different. Where that is the crux, say so in the decision record rather than pretending the trial settled it — an unresolved question written down is worth more than a false resolution.

Sometimes the trial clears the bar and you were wrong, and how that goes matters more for the next disagreement than for this one. The correct response is to say the measurement settled it and then help build the thing, without relitigating the evidence quality that you were right about and that turned out not to be decisive. A sceptic whose objections are dropped cleanly when the evidence arrives is a sceptic whose next objection gets a hearing; the cost of defending a lost position through a second cycle is that people stop pre-registering with you and start routing around the process entirely.

The related case is being overruled without a trial, which happens and is survivable. The move is to write the note anyway — what you expected, what you would have measured, and what would tell you early that it was going wrong — and then to build the thing properly rather than half-heartedly. Six months later that note is either a mild vindication or a useful record of how you were wrong, and both are better than a memory of having objected.

A rejection also has a shelf life that is shorter than most teams assume. Extraction costs fall, models get cheaper, traffic mixes shift as products change, and the 6% multi-hop share that made this a clear no could be 20% a year from now for reasons entirely unrelated to the technique. The reopen triggers exist because the decision was correct given conditions rather than correct in general.

Whichever way the trial goes, the outcome has to be explained to people who were not in the room, and the explanation will be repeated by people who did not read the paper. What survives that retelling is reliably the number, and reliably not the conditions the number depended on.