Wrong and unjustified are different review verdicts

A design document lands on a Wednesday with a review request and a launch date. Reviewing it means issuing a verdict on each decision in it, and the useful verdicts are not "good" and "bad". They are the two distinct faults a design decision can carry, which have different evidence behind them, different costs if you get them wrong, and different remedies.

A decision is wrong when it does not suit the material the system will run over — a chunk size that cuts answers in half on this particular set of documents, a retrieval method that cannot match the queries this particular audience types. A decision is unjustified when nobody has established whether it suits the material or not. Those are separate properties, and the whole of this chapter follows from the fact that they are separate. A wrong decision can be well justified, if somebody measured the wrong thing carefully. An unjustified decision can turn out to be exactly right, and the team will have no way of knowing when it stops being right.

The document under review#

Here is the specification, as submitted:

service: knowledge-assistant (internal, engineering department)
ingest:
  source: confluence space ENG (everything)
  chunking: 512 tokens, no overlap
retrieval:
  index: dense only (text-embedding-3-small)
  k: 20
  rerank: none
generation:
  context: all 20 chunks
evaluation: user feedback (thumbs up/down) collected after launch
timeline: 3 weeks to GA

Every line of that specification appears somewhere in the preceding eight chapters, and the reviewer who blocks all six is going to be wrong about at least two of them. Blocking everything also costs the review its authority, because the author knows some of these are fine, and a reviewer who cannot tell which is a reviewer who gets routed around by the second sprint.

The timeline is part of the specification too: three weeks to GA, general availability, the date the service stops being a thing the team demos and opens to the whole engineering department, after which every setting in the document is running against real questions. That date is what makes the distinction between the two faults operational rather than academic. Before GA a wrong setting costs a config change; after GA it costs answers given to people who act on them.

The two questions behind one verdict#

It is worth being precise about what a reviewer is actually able to observe, because the answer is narrower than it feels while reading a document.

Ask two questions about each line. First: does this decision suit the corpus the service will run over? Second: has anybody established that it does? The first question is about the relationship between a setting and a body of documents. The second is about the relationship between a setting and a measurement. Those relationships are independent of each other, so the two questions have four combinations between them rather than two.

Now work out which of the four a reviewer can actually distinguish. The corpus is not in front of them; a description of the corpus is. Answering the first question would require knowing the distribution of answer spans in a Confluence space, the share of queries carrying identifiers, and how many superseded copies of each runbook are sitting in it — none of which is in the document, because if any of it were in the document the second question would already be answered. So the reviewer's access to the first question runs entirely through the second. Where a measurement exists, the reviewer can read it and judge suitability. Where no measurement exists, the reviewer has no channel to suitability at all, and every impression of wrongness they have is a prior about corpora in general rather than a fact about this one.

That is the whole derivation, and its consequence is uncomfortable. A verdict issued on an unmeasured line can only ever be a verdict about the absence of the measurement. Dressing it up as a verdict about the setting borrows authority the reviewer does not have, and the author — who has seen the corpus, even if they have not measured it — can usually tell the difference. The two faults collapse into one word, "block", and the word then means whichever of the two the reader assumes.

Reading it the other way round gives the useful test. A reviewer who wants to block a line should be able to name the measurement that would change their mind. If they can, the verdict is about evidence and the block is honest. If they cannot, they are asserting the corpus is a certain way, which is exactly what they are blocking the author for doing.

Six decisions, three verdicts#

The verdict column closes on the claim made above it. Two of the six lines carry a note, three carry a block, and one carries no verdict of its own because it was folded into another line — two plus three plus one accounts for all six with nothing left over. The two notes are precisely the lines a reviewer who blocked everything would have been wrong about, which is why the earlier sentence says at least two rather than exactly two. The folded line is the reason for the hedge: blocking the prompt-assembly decision separately would be a third error of a different kind, counting one defect twice, and whether that counts as being wrong about a line is a matter of taste rather than arithmetic. Every column in this chapter that has a residual has it for a nameable reason, and this one is it.

The third column is the load-bearing one, and reading across a row shows why. Each entry names a measurement, and each measurement is one a person could run this week. That is what makes the verdict in the middle column a verdict about evidence rather than about the setting.

The chunk size is a note because 512 is the setting most likely to be adequate on a wiki, and the absence of overlap costs a sentence cut in half at a boundary now and again. If the span measurement comes back saying half the answers span 900 tokens, the design needs to change, and the measurement is an afternoon. Nothing about the failure mode is severe enough to hold a launch on the assumption that it is wrong.

Dense-only is a note with a deadline because the measurement takes an hour and the answer decides it. An engineering-department wiki is a corpus where identifiers are likely, so the prior is against the design, but the log sweep settles it and no argument in a review comment can. Requiring a lexical index before seeing that number would be requiring two weeks of infrastructure on a guess.

The k=20 line blocks for a reason specific to this design rather than to the number. Twenty candidates go into a single generation prompt with nothing between retrieval and the model, which is the configuration where context precision governs answer quality, and the document contains no precision number and no plan to produce one. The evidence that clears it is a measurement, not a reranker; if precision at k=20 on their corpus comes back at 0.6, the design is fine as written and the reranker would be waste.

That last clause is where a reviewer's instinct is most likely to be a defence that does not defend. Adding a reranker to the design feels like insurance: worst case it is unnecessary, best case it rescues a precision problem. What it actually does is change the number without revealing it. A design with a reranker in it and no precision measurement is in exactly the same evidential state as the design under review — the setting has moved and the unmeasured question has not — and it now carries a second model's latency and bill for reasons nobody can state. The insurance covers the reviewer's discomfort rather than the service's failure mode.

Ingesting the whole space blocks because the failure it produces is the one users cannot detect. A stale answer with a citation reads as authoritative, and an engineering wiki reliably contains three generations of the same runbook. The census is half a day of work and either finds duplicate canonical groups, in which case status metadata is required before GA, or finds none, in which case the design is fine and the reviewer has learned something about the corpus.

Notice that this block is the one where the second question and the first come apart hardest. The absence of a census is the fault a reviewer can see. The damage, though, is done by a property of the corpus the census would reveal, and that property has no effect on any number the service reports. Duplicate runbooks do not lower a similarity score, do not lower a groundedness score, and do not produce a refusal. The measurement that would settle it has to be run against the documents, from outside the running system, because nothing inside the system varies with the thing that matters.

Evaluation after launch blocks unconditionally, for the asymmetry chapter 6 sets out — that every item on an anti-pattern list is the right design for some corpus, except this one, which is a defect whatever the corpus looks like. Every other verdict in the table is a claim about the corpus that a measurement resolves, and thumbs-up data collected after GA resolves none of them: it is unlabelled, sparse, biased toward users who are annoyed, and it arrives after the decisions have been made. The other five lines can be cleared by evidence; this line is the one that removes the ability to produce evidence for the other five, which is why it does not have a corpus on which it is fine.

Three weeks, and the list is a month of work#

The author replies that the team ships in three weeks and this review is a month of work, and that reply is correct. What actually gets required is the subset that keeps every remaining question answerable later.

That is four and a half days inside a three-week schedule, and it is the whole of what the review requires. The cost column is worth reading carefully, because it does not add up the way it first appears to and it is right anyway. Adding the three whole-day and part-day entries gives two days plus half a day plus one day, which is three and a half days, and the two remaining entries add an hour and two hours on top of that. Three hours is not a further day of effort. The figure of four and a half days is the same work charged in half-days, because you do not get the rest of a person's afternoon back after booking them for an hour of regex. The effort total and the schedule total measure different quantities — hands-on hours in one case, capacity removed from the sprint in the other — and the gap between them, about a day, is the real residual and is what a plan should carry as slack.

Explicitly not required: a reranker, a lexical index, re-chunking, or a full evaluation harness. Those become decisions the team makes with numbers in hand, and if context precision comes back at 0.6 and the census finds no duplicates, the team ships the design exactly as written and the review was still worth running, because six assertions became six measurements.

Two of those items are worth defending if the negotiation continues. The labelled query set is the one that cannot be deferred, because every question in the review reduces to it and building it after launch means the first six months of production data cannot be interpreted. Trace logging is the other, since without it the thumbs-down feedback the team is relying on arrives with no way to tell whether a bad answer came from an empty context, a stale document or a model error, and the audit from chapter 8 — reading the settings actually in force out of a trace rather than out of a design document — becomes impossible to run on their own service.

The negotiation itself is the skill. A review that hands back a list of thirteen findings and no ordering has delegated the decision to the person under deadline pressure, who will implement the cheapest three. Ordering the list, naming what is negotiable, and stating what each item buys converts a gate into a conversation about evidence.

Tightening the thresholds makes the same gate harder to pass#

The reflex fix is to tighten the thresholds — chunk size between 256 and 1,024, a mandatory reranker, a minimum evaluation set size — and it makes the gate harder to pass without making it better at improving quality. The failure is that the gate accepts assertions instead of measurements, and a numeric mandate is still an assertion, just with a number in it. Fixed thresholds also guarantee wrong answers on the corpora that legitimately fall outside them: the FAQ team from chapter 6, whose short self-contained entries made a small chunk correct, would have been forced away from the chunk size that was right for their corpus, and the log-search assistant would have been forced to add a reranker that destroys the repetition its reader depends on.

What replaces it is a gate whose passing condition is an artefact rather than a paragraph. A link to a query set in version control, with a commit hash. A link to a run that produced context precision and context recall on that set, with the arms and the date. A trace of one randomly sampled production query. None of those can be produced by writing the right words, and all of them are things the team needed anyway. The gate stops asking "did you avoid the anti-patterns" and starts asking "what did you measure, and where is it".

The reason artefacts work where thresholds do not is that they restore the variation the gate lost. A commit hash either resolves to a query set or it does not, and it costs a team that measured nothing to produce. A precision run either exists with a date on it or it does not. Each of those is cheap for a team that did the work and impossible for a team that did not, which is the property a check has to have before its output means anything.

Two other changes help more than tightening. Move the check from design time to a standing report per service, refreshed on every deploy, so the numbers are visible continuously rather than argued once before launch when the least is known. And replace uniform shallow review with sampled deep review: three services a quarter audited properly against a real trace, rather than twenty design documents skimmed. A hundred per cent pass rate across twenty shallow reviews and three genuine findings from three deep ones are not close in value.

None of this eliminates judgement, and none of it makes review sufficient. A reviewer with a query set and a trace can still approve a system whose corpus has no owner, and the per-diem answer from chapter 4 — where an index still holding a superseded copy of an expenses policy answers with last year's rate, and no reranker can reach a document the index does not contain — will keep being wrong until somebody at the company is accountable for which document is current. Review catches designs that cannot justify themselves; it does not catch the organisational absences underneath.

The one-page report at the end of the week#

The fixture from chapter 1 — a small hand-enumerable corpus, a fixed query set, and one switch per suspected defect — is a day's work against your own stack, and it is the thing that makes every chapter here operational rather than theoretical. Build it, label forty queries from your real logs, and run the baseline. Then ask your own service for a trace of one randomly chosen query from yesterday and read it stage by stage against the audit table from chapter 8, which lists each stage's settings beside the value actually observed, counting how often each default binds across a day of traffic. Most teams find two or three settings nobody chose, and one of them is usually doing more damage than anything in the design document.

What you will have at the end of that week is a one-page report: the corpus census, the identifier share, context precision and context recall at your current k, the list of defaults in force with their frequencies, and the two changes worth measuring next. That report is what the anti-pattern list was always trying to produce, and having it turns each item on the list from a prohibition into a question with an answer specific to your corpus.

The report also changes what a verdict can be. With a census and a precision number in front of them, a reviewer can answer both questions instead of one: whether the setting suits the corpus, and whether anybody established it. The two faults stop being indistinguishable, "block" stops being a single word covering both, and the reviewer's own claim becomes checkable by the person they are reviewing. That symmetry is the point. A review that can be checked is a review that can be argued with on the evidence, and a review that cannot is one the author will eventually stop attending.

Every design document asserts things about a corpus nobody has looked at. That is not misconduct, it is the ordinary condition of building a system before you have measured the material it runs over, and it describes the reviewer as much as the author. What separates a useful review from theatre is whether it says so — whether each verdict names the measurement that would settle it, and each block admits that what is missing is the number rather than the judgement.