Hidden defaults live in the trace, not in the design document

The design document for an internal compliance assistant — a question-answering system over a company's own policy documents — is four pages long, and the retrieval section is one paragraph: "User questions are embedded and matched against the policy index using cosine similarity. The ten most similar chunks are passed to the model along with the question." The team has been running it for five months. Asked for a trace of one real production query — a stage-by-stage record of what each step of a pipeline received, what parameters were in force, and what it passed on — they take two days to produce it, which is itself informative, and this is what comes back.

{"query": "what is the retention period for audit logs?",
 "stages": [
  {"stage": "rewrite",   "out": ["audit log retention period",
                                 "how long are audit logs kept",
                                 "log retention policy duration"],
                          "note": "multi_query=True (default)"},
  {"stage": "retrieve",  "per_subquery_k": 4, "returned": 12,
                          "note": "k defaults to 4 in the retriever wrapper"},
  {"stage": "filter",    "score_threshold": 0.75, "dropped": 7, "kept": 5},
  {"stage": "dedupe",    "dropped": 2, "kept": 3},
  {"stage": "assemble",  "chunks": 3, "truncated_at_chars": 1000,
                          "context_tokens": 690},
  {"stage": "generate",  "model": "…", "citations": 1}]}

Three chunks reached the model, not ten. Nobody chose three; it fell out of four defaults interacting. The rewriter, enabled by a flag someone set while following a tutorial, produced three sub-queries. Each was retrieved at the wrapper's default k of 4 rather than the ten in the document. A similarity threshold of 0.75, which nobody in the room could account for, discarded seven of the twelve candidates. Deduplication removed two more because the sub-queries overlapped, which is what sub-queries do. The assembly step then truncated each surviving chunk at 1,000 characters, so the tail of the longest one never reached the prompt at all.

Those four defaults are not a list of suspects. They reconcile, and the reconciliation is what turns the trace from a story into evidence. The rewriter emitted three sub-queries and the retriever was asked for four candidates each; three fours are twelve, which is exactly the returned count the retrieve stage logged, so nothing went missing between rewriting and retrieval. The filter reports seven dropped and five kept, and seven and five are twelve. Dedupe reports two dropped and three kept, and two and three are five. Assembly reports three chunks, which is what dedupe handed it. Every candidate the system ever held is accounted for by a stage that says what it did with it, and the column closes with no residual.

That closure licenses one thing precisely. The gap between the ten chunks in the design document and the three that reached the model is fully explained by the four stages above, so there is no need to go hunting for a fifth cause, an unlogged filter or a bug in the assembler. It licenses nothing about whether the three that survived were the right three — a count reconciles counts, and it is silent on content. It is also silent about everything that never entered the index, because a chunk that was never ingested is not a candidate some stage dropped. It is a candidate that never existed to be logged, and no column in the trace has a place to put it.

The assembly line needs one qualification before it is leaned on. The trace records truncated_at_chars as 1,000, which is the cap that was in force rather than a report that any chunk reached it. Whether a specific chunk lost its tail is recorded elsewhere, on the per-chunk chunks_meta that the counting script further down reads a truncated flag from, and the excerpt above does not print that field. What survives without it is that a cap was applied to every surviving chunk. The stronger claim, that the longest chunk was cut, needs the flag, and asking for it is the kind of follow-up a reconciled column earns.

Ask for the trace before anything else#

The first request in a review is a full trace of one real production query through every stage, with the parameter values in force at each. What it will contain that the design document does not is the settings nobody selected, and the reason to expect them is structural: a design document records decisions, and defaults are the absence of decisions.

That sentence is worth walking rather than asserting, because each link is checkable on its own. A design document records decisions, so a parameter nobody decided about is absent from it. The code path still reads that parameter at run time, and reading it has to return something. A library therefore supplies a value. The set of values in force is consequently the union of two sets — the ones somebody configured and the ones a library defaulted — and only the first of those two is written anywhere a reviewer can read it. A trace is the one artefact produced by both sets at once, because it records what executed rather than what was intended.

That makes a design document and a trace two instruments that share no step. One is written by hand before the system runs; the other is emitted by the system while it runs. Where they agree on a value, somebody configured it and the configuration took effect. Where they disagree, a default is in force and has overridden the intent, which is what ten chunks against three is. Where the document is silent and the trace carries a value, the value is a default nobody has examined. And where both are silent, the parameter is either not read on this path or read and not logged. Every parameter falls into one of those four cells, so the cells sum, and the last one is the only cell no instrument can see — a parameter unaccounted for is a real residual, and the way to find it is to compare the trace's stage list against the code rather than against the document.

The belief this displaces is that reviewing the design document is a defence. It is tempting for good reasons. The document is the artefact the team wrote deliberately, it is short enough to read in full, and every sentence in it is true. What it cannot do is change when the thing you are worried about changes: four pages that say ten chunks read exactly the same whether ten chunks arrive or three do. Its completeness stays constant while the effective configuration varies underneath it, and a quantity that holds still while the dangerous property moves is useless as a detector of that property. That is the whole reason the first request is the trace and not the document.

The stages that hide the most are the ones with no obvious owner. Rewriting covers everything a pipeline does to the question before it reaches the index: splitting it into several sub-queries, as here, or the trick called HyDE, where the model first drafts a hypothetical answer and that invented passage is embedded instead of the question. It is often on by default in a framework's higher-level chain, and it changes the query the index actually sees, which means every measurement you took on raw queries was measuring something else. Score thresholds are worse, because they are the one setting that turns a retrieval miss into an empty result with no error, and a system that returns three chunks when it planned for ten still produces a fluent answer. Assembly is where character limits, token budgets and per-document caps live, and they are usually expressed in characters while everything else is in tokens.

The threshold is worth following through one link at a time, because it changes the shape of a failure rather than its frequency. A retriever with no threshold always returns as many candidates as it was asked for, so a bad retrieval arrives as a full set of irrelevant chunks and the answer built on it is visibly unsupported. Add a threshold and the candidates below it are discarded, which means the stage returns fewer than it was asked for. Returning fewer than requested is not an error in any retrieval library; it is documented behaviour, so nothing raises and nothing logs a warning. The generator is then handed a short context and writes fluently from it, because fluency depends on the model rather than on the evidence. The same underlying failure that used to be loud is now silent, and the only place the change leaves a mark is the dropped count in a trace.

The embedding row is the one that surprises teams most, because the setting is not in their code at all. A splitter configured at 1,000 tokens feeding an encoder whose maximum sequence length is 512 means the second half of every chunk is truncated before embedding, silently, with no warning anywhere. The stored text is the full 1,000 tokens, so the model sees it once retrieved, but nothing in the tail can cause retrieval. It is testable in ten minutes: take fifty sentences from the last 200 tokens of long chunks, query the index with each verbatim, and see how often the parent chunk comes back. Healthy is above 90%. On a truncated index it is close to zero, and the shape of the failure is that specific facts near the end of long documents are unfindable while the same facts near the beginning are trivially found.

The test works because it makes the two possibilities predict opposite results, which is what separates a diagnostic from a look. If the tail was embedded, a query that is literally a sentence from the tail is about as close to that chunk's vector as any query can be, and the parent chunk should come back at or near the top nearly every time. If the tail was cut before embedding, that sentence contributed nothing to the vector, and the parent comes back only in the cases where the head happens to be about the same thing. There is no third mechanism that produces a near-zero hit rate on text that is provably sitting in the stored chunk.

Which is also the second defence that does not work, and this one is worse than reading the design document because it involves actually checking. Search the stored chunks for the missing sentence and it is there, every time, since storage was never truncated — only the embedding was. The quantity a reviewer checks holds still while the property that decides retrieval varies, so the check passes on a broken index and passes on a healthy one identically. The distinction the check has to respect is that being present in a chunk and being able to cause that chunk's retrieval are two different properties, and only one of them is what a search system runs on.

Be specific about which query you want traced, because the one a team volunteers is a query they have looked at before. Ask for three: a randomly sampled query from yesterday's log, chosen by a script in front of you, one that a user complained about, and one the team considers a showcase. The random one is where undecided defaults are visible in their normal operating state; the complaint shows the failure mode under investigation; the showcase establishes what the pipeline looks like when everything works, which is the baseline you compare the other two against. Differences between the three are often more informative than any of them alone, since a threshold that binds on the complaint query and not on the showcase is a lead that took no instrumentation to find.

Insisting that a script picks the random one in front of you is not suspicion of the team, it is the only way to get a sample that was not selected on its outcome. A query somebody volunteers has already been looked at, which means it has already survived whatever scrutiny it was going to get. A complaint was selected for having failed. A showcase was selected for having worked. Each of the three is evidence about the population it was drawn from and none of the last two is evidence about the normal case, and the random draw is the only one of the three that costs nothing to make unbiased.

Turning a trace into an audit#

One trace is a sample of one, and the audit is what turns it into a list of claims worth checking. For each stage, the question is what value is in force, who chose it, and what evidence supported the choice. Most rows come back as "the library's default" and "none", which is the expected answer and is not by itself a problem, since a default can be correct. What it does mean is that the value has never been measured against the corpus, so it belongs on the same evidence footing as the settings in the design document.

The second half of the audit is frequency, which one trace cannot give you. A threshold that binds on 4% of queries is a footnote; one that binds on 60% is the system's dominant behaviour. Counting is cheap once the trace fields exist, and the counts are usually more persuasive than any argument. The reason they are more persuasive is that they support a different kind of claim. One trace supports "this can happen", and a count supports "this happens on this share of traffic" — and only the second form supports a decision, because the cost of a fix is fixed while its benefit scales with the share.

counts = Counter()
for t in traces_from(last_24h):
    r = stage(t, "retrieve"); f = stage(t, "filter"); a = stage(t, "assemble")
    counts["threshold_bound"]   += f["dropped"] > 0
    counts["ctx_under_3_chunks"] += a["chunks"] < 3
    counts["ctx_empty"]          += a["chunks"] == 0
    counts["chunk_truncated"]    += any(c["truncated"] for c in a["chunks_meta"])
print({k: v / len(traces) for k, v in counts.items()})

Run it over a full day rather than an hour, because traffic composition shifts across the working day and a morning sample over-represents whoever starts early. A day of a modestly used internal assistant is a few thousand traces, which is enough for these proportions to be stable to within a couple of points, and small enough to hold in memory while you look at the outliers by hand.

On the compliance assistant that produced: threshold binding on 71% of queries, fewer than three chunks in 34%, an empty context in 6%, and truncation in 22%. Six per cent of queries reaching the model with no retrieved context at all is the finding that ends the meeting, because those answers were generated from parametric knowledge — whatever the model absorbed during training, held in its weights, with nothing from the corpus to check it against — and presented with the same confidence as everything else.

Those four proportions say more read as gaps than as levels. Queries arriving with fewer than three chunks are 34% of traffic and queries arriving with none are 6%, and the second set sits inside the first, so the gap between them — a little over a quarter of all traffic — is the band where the model got something and got less than the design document promised. That band is the one nobody will ever notice, since a short context still produces an answer with citations in it. The empty 6% is the smaller number and the easier finding, and the quarter-and-a-bit standing behind it is the larger problem.

The traced query at the top of this chapter is worth locating in those counts. It arrived with exactly three chunks, so it falls outside the 34% entirely. It was not chosen for being bad; it is on the healthy side of the count the audit flags, and it was still answered from a context that had lost seven of its twelve candidates to a threshold nobody had chosen.

Narrowing one failing query to one change#

Aggregate counts justify attention; confirming a cause needs a single query and the smallest change that would flip its outcome. The audit query above is a good one to work: the retention period for audit logs is in compliance/logging-standard.md, section 4, and the answer said thirty days when the standard says thirteen months.

The trace gives one candidate cause per stage, and the discipline is to test one at a time, starting with the stage whose counts were worst. The hypothesis is that the 0.75 threshold discarded the gold chunk. The smallest change that tests it is to re-run the single query with the threshold set to zero and nothing else altered, which takes a minute and touches no configuration that anyone else depends on. The gold chunk arrives at rank 7 with a score of 0.731, confirming that it was retrieved and then thrown away.

The confirmation is stronger than one reading, because two instruments that share no step land on the same account. The trace was emitted before anything was touched and records a threshold of 0.75 with seven candidates dropped and five kept. The re-run was taken afterwards with the threshold at zero and nothing else altered, and it records the gold chunk at rank 7 of twelve scoring 0.731. Neither measurement uses the other's machinery, and they reconcile: ranks one to five cleared 0.75 and were kept, ranks six to twelve did not and were dropped, five and seven are the twelve the retriever returned, and 0.731 sits below 0.75 by under two hundredths. The gold chunk is the second item on the discarded side of a cut it missed by that much.

Two sevens appear in that paragraph and they are different quantities. Seven is how many candidates the filter discarded; rank 7 is where the gold chunk sat in a ranked list of twelve. They are equal by arithmetic accident — twelve candidates with five kept leaves seven discarded, and the gold chunk happened to land one place inside that tail. Reading them as the same seven would suggest the gold chunk was the last one dropped, which it was not, and nothing in the argument depends on where inside the tail it fell.

The rank and the score together are the confirmation, and they are not yet the fix. Removing a threshold that binds on 71% of queries changes every answer the system gives, so the next step is the fixture and the arms table from chapter 7: with the threshold at zero, six queries recover their gold chunk, none lose theirs, context grows by about 900 tokens per query, and context precision falls from 0.31 to 0.22. That combination argues for removing the threshold and adding a reranker to absorb the extra candidates, which is a different proposal from the one the team would have written after reading the trace alone.

Keep the smallest change small in the other sense too. Re-running with the threshold at zero is reversible, affects one query, and produces an unambiguous answer; re-chunking the corpus to test the same hypothesis would take two days and answer a different question. When the candidate cause is a parameter, the minimal test is nearly always to rerun one query with that parameter moved to a value that makes the hypothesised mechanism impossible.

Two comparisons to refuse while reading that paragraph. The precision figures in it are not the ones in the previous chapter: chapter 7's arms table measured a different system on a different query set, and 0.31 falling to 0.22 is this assistant's own fixture, so the method transfers between the two and the numbers do not. And the two figures inside the sentence do not measure the same thing as each other — context precision counts chunks while the nine hundred extra tokens counts tokens — so the fall in one cannot be derived from the rise in the other without knowing how long the recovered chunks were.

The limits of a single trace#

A trace shows what happened, not how often or how much it mattered, and treating one as evidence about the system is the error that mirrors the pull request in the previous chapter. It is also silent about what was never in the index. If the audit log standard had failed to ingest because the connector skips pages over 2 MB, the trace would look identical up to the point where nothing relevant came back, and the finding would sit in an ingest log nobody reads.

The other limit is that this only works where traces exist. Plenty of systems can produce a final answer and its citations and nothing in between, and the team's answer to the request is that they would have to add instrumentation. That answer is the audit's first finding, and it outranks anything about chunk size, because a system whose retrieval stages cannot be observed cannot be debugged and every claim in its design document is unfalsifiable.

A trace is the only artefact a retrieval system produces that reports what executed rather than what was intended, and that is why it is the first thing to ask for and not the last. Its counts are what make it evidence instead of an anecdote. Three sub-queries at four candidates each is twelve; seven dropped and five kept is twelve; two dropped and three kept is five; three chunks assembled. A column that closes like that says the shortfall from ten chunks to three needs no explanation beyond the four stages that produced it, and it says nothing whatever about the material that never reached a stage at all. What the trace cannot supply is how much any of it matters, which is what the counting script is for: a threshold binding on 71% of queries is a system's dominant behaviour, and a threshold binding on a few per cent of them is a footnote. Between one query traced in full and one day of queries counted, a reviewer can state what a system does and how often it does it, and neither claim was available from the design document.

An audit like this typically produces eight to fifteen findings, which is more than any team will fix before shipping. Deciding which of them block a launch is a different judgement from finding them, and it is where the anti-pattern list does the most damage when it is applied as a list.