What gets lost in retelling is the strength of the evidence

A pull request description in a legal-tech company read, in full: "Switching the clause-extraction path to structured decomposition — it's about 9 points better." Tracing that sentence backwards took twenty minutes and passed through four hands. The paper it descended from reported exact-match accuracy rising from 61.8% to 70.9% on 90 questions over a corpus of commercial leases, with a section noting that the advantage disappeared, and on two configurations reversed, once documents exceeded roughly 30,000 tokens. A conference slide compressed that to "+9.1 on legal QA." A widely-shared post rendered the slide as "structured decomposition gives you 9 points on legal documents." A colleague pasted the post into the team channel with the comment "we should probably be doing this." The company's median contract was 47,000 tokens.

Nothing in that chain was dishonest and every step lost the same category of information. The number 9.1 is a token that survives copy-paste intact, whereas "on 90 questions, single run, and the effect inverts above 30k tokens" is three clauses that each need a sentence and none of which fit on a slide. Compression selects for what is short and quotable, and evidence strength is neither.

The four things that have to survive#

An explanation that a colleague can act on carries the mechanism, the strength of the evidence, the conditions the result depends on, and what it would mean for your system. Drop any one and the remainder becomes misleading in a predictable direction.

The mechanism has to be concrete enough that someone can picture the runtime change, in the vocabulary established in chapter 1 — what extra calls happen, in what order, and what is done with the results. Without it, listeners substitute their own guess, and the guesses diverge wildly for anything with a name as evocative as "self-verification."

Evidence strength means the size and provenance of the measurement, said in ordinary words: how many items, how many runs, whose data, whether anyone independent has seen the same thing. A paper's own evaluation on 90 questions and a replication across three labs on 4,000 items are both "evidence" and they are not the same object.

The conditions are the boundaries the authors themselves stated plus the ones your reading found. Document length above 30,000 tokens is the kind of condition that decides everything and appears in one paragraph of section 5.

The practical consequence translates all of the above into a sentence about your system, including the case where the consequence is nothing. "For us this would apply to about 6% of requests" is a complete and useful consequence.

The column that empties first is evidence strength, because it is the only one with no natural home in a short message. Conditions at least sound like caveats and occasionally get repeated as such; sample size and run count sound like pedantry.

Writing it so the loss does not happen#

The fix is a document short enough that nobody needs to compress it further, which in practice means about 200 words with a date and a name on it. When the canonical version is one screen long, the retelling is a link.

## Structured decomposition for clause extraction — 2026-08-06, marcus

**What it does.** Instead of one call per document, the model first lists the
sub-questions a clause-extraction answer needs, answers each against the
retrieved passages, then assembles a final answer with the original question
restated above the sub-answers.

**How strong the evidence is.** One paper, its own evaluation: 90 questions on
commercial leases, single run, no compute-matched baseline. Our own run on 200
questions from our traffic, 5 seeds: +3.1 points ± 1.4 at 2.4x inference cost.
Nobody outside the authors has published a replication.

**Where it stops working.** The paper reports the gain vanishing above ~30k
tokens per document; our median contract is 47k. Our run used the 200 shortest
documents in the sample, so it has not tested the regime we actually operate in.

**What it would mean here.** Applies to roughly a third of our corpus as
implemented. Estimated +1 point end-to-end at +$1,400/month. Not proposing
adoption; proposing we re-run on long documents before deciding.

**Revisit if:** someone replicates on long documents, or our chunking work
brings effective document length under 30k.

There is a spoken version of the same content that fits in thirty seconds, and it needs to be prepared rather than improvised, because the standup version is what most people will actually receive. The order that works is consequence first, then evidence strength, then the condition: "This would apply to about a third of our documents, and on our own run it was worth three points at more than double the cost. The evidence is one paper on ninety questions plus our own two-hundred-question run, and the effect is reported to disappear on documents longer than thirty thousand tokens, which most of ours are." That is four sentences containing two sample sizes, a cost multiple and a boundary condition, and anyone who hears it can decide correctly without reading anything.

The predictable follow-up is some form of "but should we do it," and the useful answer names the next measurement rather than a position. Saying that the question turns on long documents, that testing it is two days, and that you would rather spend the two days than argue, converts an opinion contest into a scheduling decision. The unhelpful answers are the two obvious ones: an unqualified yes borrows certainty the evidence does not have, and an unqualified no invites everyone to re-run the argument in three weeks when the next post circulates.

That note is not shorter than the channel message it replaces, and it is the shortest thing that cannot be misquoted into a merged pull request. The section headings are deliberately not technical vocabulary, because the failure this document exists to prevent is a reader skipping the part they cannot parse.

Uncertainty in words that mean something#

The most common way evidence strength gets hidden is not omission but euphemism, where a technical-sounding phrase stands in for a number the speaker does not have. "Statistically significant" said aloud in a standup usually means "the number went up." "Consistent improvement across settings" often means two settings. "State of the art" describes a leaderboard position that may be worth a tenth of a point and may be contaminated in the way chapter 3 described. Each of these sounds more rigorous than the plain version while carrying less information.

Plain terms in this sense does not mean simplified. The numbers stay, the units stay, and the sample sizes stay; what goes is vocabulary whose function is to signal expertise rather than to transmit a quantity. A colleague who hears "90 questions, one run, effect reverses on long documents" knows exactly how much to trust it, and a colleague who hears "significant gains on legal QA" does not.

The test is whether they can correctly ignore it#

An explanation succeeds when the listener can reach the right decision, and for the large majority of developments crossing your feed the right decision is to do nothing. That makes "they decided to ignore it, and ignoring it was correct" the ordinary success case, which sits awkwardly with the instinct to make an explanation persuasive.

Consider explaining a genuinely important development that your team should nonetheless ignore. Longer effective context windows are a real capability change with real evidence behind them, and for a team whose median request is 1,800 tokens the consequence is nothing at all this quarter. An explanation that conveys the mechanism, notes that the evidence is strong and largely independent, states the condition that matters — the gains show up on tasks that need more than about 30,000 tokens in a single request — and then says that 96% of your requests are under 4,000 tokens, has done its job when everyone goes back to work. An explanation that leaves out the traffic number invites a quarter of speculative work.

Where the note lives determines whether any of this holds up over a year. A channel message is unfindable within a week, a wiki page nobody owns rots quietly, and the version that survives is the one committed next to the code or the eval set it concerns, with the date in the filename and the revisit trigger written as a condition somebody could actually notice. Teams that keep these in a single directory accumulate something more useful than the individual notes: a record of what was considered and declined, which is what makes the fourth person to bring up graph retrieval a two-minute conversation rather than a fresh debate. Reviewing that directory once a quarter against the reopen triggers takes half an hour and occasionally surfaces a decision whose conditions have quietly expired.

The same standard applies to things you are advocating. If you have run the trial from chapter 5 and it cleared the bar, the explanation still has to include the evidence strength and the conditions, because the people building on your recommendation need to know when to stop trusting it. A recommendation without its conditions attached becomes a rule, and a rule outlives the situation that justified it.

What a plain explanation is not#

It is not a paper summary. A summary is organised around the authors' argument and spends its length on what they did; the note above is organised around your team's decision and spends its length on evidence strength and fit. If your explanation follows the paper's section order, it is a summary wearing a different hat.

It is not advocacy, and the reliable tell is whether the limitations section is proportionate to the claims section. A note where the mechanism gets 90 words and the conditions get eight has decided in advance what the reader should conclude.

It is also not the decision. The note above recommends re-running on long documents and explicitly declines to propose adoption, which is a legitimate output and stops the document from having to carry more certainty than the evidence supports.

The honest limitation is that none of this stops the drift, it only slows it and leaves a place to check. Someone will still quote your +3.1 without the ± 1.4, and the value of the note is that when the number reappears in a pull request six weeks later, there is a dated document with the conditions in it that anyone can open. Treat the drift as a property of how information moves through organisations rather than as a failure of the people repeating it, and the design goal becomes making the canonical version cheap to find rather than making everyone rigorous.

Across this topic the loop is now complete: extract one falsifiable claim from a paper, demand the compute-matched comparison, check whether the effect is a property of the test set, reproduce a small version to find what was left out, decide with reference to your own workload rather than the paper's flaws, and write the result down in a form that survives being repeated. Running that loop once end to end on a paper currently circulating in your team is a week of work at most, and the artifact it should leave behind is the note — dated, with its conditions and its revisit triggers — because the loop's output is not an opinion about a paper but a decision your team can audit later.