Retrieval cannot return text the parser never emitted

A missing answer looks exactly like a ranking failure, and for a good reason: ranking is the part of the system that decides what comes back. A retriever embeds a question, scores it against every stored vector, and returns the best k of them. When the clause somebody needed is not among those k, the natural reading is that it scored too low, and the natural next moves are a stronger embedding model, a larger k, or a reranker. Every one of those moves changes the order of what the index holds. None of them changes what it holds.

That distinction is the whole of this chapter, and the test that settles it takes about thirty seconds. Call it a presence test: query the index by metadata rather than by similarity, pull every chunk carrying one document's identifier, and search the result for the phrase that should have been returned. A negative answer is not a hint about ranking, it is a proof about ranking — nothing can be ranked into the top k if it is not in the set being ranked at all. The reason this is worth writing down is that the two suspects are not equally visible. A retriever and a model are instrumented, watched and interesting. A parser ran weeks ago, logged a success, and has no dashboard, so nobody looks at it first unless there is a reason to.

The presence test, which takes about thirty seconds#

Query the index by metadata rather than by similarity, pull every chunk carrying that document's identifier, and search the concatenation for the phrase.

import re, unicodedata

def normalise(s: str) -> str:
    s = unicodedata.normalize("NFKC", s)          # fi -> fi, nbsp -> space
    s = s.replace("\u00ad", "")               # soft hyphen
    s = re.sub(r"-\s*\n\s*", "", s)                # line-break hyphenation
    s = re.sub(r"[‘’“”]", "'", s)
    return re.sub(r"\s+", " ", s).strip().lower()

chunks = index.fetch_by_metadata({"doc_id": "msa-vantis-2023"})
haystack = normalise(" ".join(c.text for c in chunks))

for needle in ["termination for convenience",
               "terminate this agreement for convenience",
               "thirty (30) days' prior written notice"]:
    print(f"{normalise(needle) in haystack!s:>5}  {needle}")

print(f"chunks: {len(chunks)}  chars: {len(haystack)}  "
      f"pages: {sorted({c.meta['page'] for c in chunks})}")

False  termination for convenience
False  terminate this agreement for convenience
False  thirty (30) days' prior written notice
chunks: 47  chars: 61,208  pages: [1, 2, 3, ..., 10, 11, 13, 14, ..., 34]

Fetching by metadata rather than by similarity is a decision, and it is worth separating from the requirement it serves. The requirement is to determine whether a phrase is anywhere in the index for that document. A similarity search cannot do that: it returns the best k matches, and a phrase whose best-matching chunk falls below that cutoff is indistinguishable from a phrase that is not stored at all, unless k is raised to the size of the whole index. Fetching by document identifier replaces a ranked sample with the complete set for one document, which turns a question about degree into a question about membership and bounds the work at one document's chunks rather than the index's.

Page 12 is absent from the page list. Forty-seven chunks exist for this document and none of them came from that page, which settles the question completely: this was never a retrieval problem, because no ranking of the index could have returned text the index does not contain. The next artefact to open is what the parser produced for page 12, and what is found there is invariably something that happened at ingestion, succeeded loudly enough to be recorded as a success, and left the page out.

The other two facts an engineer reaches for first do not survive contact with this result. k is 5. The embedding model answers every other query in the eval set correctly. Both facts are true and neither is relevant, because the first thing to establish is whether the text was ever in the index at all. The reason they are not relevant is worth stating rather than assuming: the embedder's behaviour on other questions is fixed no matter what happened to page 12 of this file, so it would read the same whether the page parsed perfectly or was never opened. A fact that cannot vary with the thing it is being offered as evidence about is not weak evidence, it is no evidence.

The normalisation in that function is doing real work and is worth keeping. A negative result from a naive substring check is worthless, because PDF extraction routinely produces termina-\ntion from a line break, fi as a single ligature codepoint, curly apostrophes in "days'", and non-breaking spaces inside "30) days". Three of the four times this test appears to prove absence, the phrase is present in a form the comparison did not anticipate, and the fix is in the comparison. Running the test with and without normalisation and comparing the two answers is itself informative: a phrase that is present only after normalisation tells you your chunks contain hyphenation artefacts that are also degrading their embeddings.

That failure has the same shape as the k and embedder facts above. An unnormalised substring check returns False when the page was never parsed and False when the page was parsed perfectly and hyphenated across a line break. It gives one answer to two situations that demand opposite responses, which is exactly the property a diagnostic must not have. Each transformation in normalise removes one way in which two strings can differ while meaning the same thing — a composed ligature against its two letters, a soft hyphen that exists only to permit a break, a hyphen plus newline inserted by the typesetter, a typographic quote against a straight one, runs of whitespace against one space. What is left is a comparison that can only fail for reasons that matter.

The value of this diagram is the branch on the left, because that is the branch the whole team skips. When an answer is wrong, attention goes to the model and to the retriever, both of which are visible, interesting and instrumented. The parser ran weeks ago, logged a success, and has no dashboard.

The presence test has its own way of lying#

Run across sixty questions rather than one, the test's failure modes start to matter, because a false content_absent sends an engineer to read parser output for a page that parsed perfectly well. The first full run on this corpus reported twenty absent spans. Fourteen of them were real, and the other six came from three defects in the harness rather than in the pipeline.

Three were documents where fetch_by_metadata returned zero chunks. The ingestion pipeline had keyed doc_id on the object path, contracts/2023/MSA-Vantis-2023.pdf, while the eval's gold references — the hand-written record, for each question, of which document and page holds its answer — used the slug the contract management system assigns, msa-vantis-2023. An empty chunk list and a chunk list that does not contain the phrase produce the same verdict from a naive substring check, and only one of them is about the parser.

Two were phrases that straddle a chunk boundary. The vector store returned chunks in insertion-adjacent but not document order, so joining them put page 19 next to page 3, and a span such as "terminate this agreement for convenience upon thirty (30) days' notice" — split by a chunker whose overlap did not cover it — was never contiguous in the haystack. Sorting by page and character offset before joining fixes it and costs a line.

The last one was a document that ingested partially. Its chunks were present and correctly keyed, and the file stopped at page 34 of 61 because a malformed cross-reference table made the parser return early without raising. The character total from chapter 2's per-page check is what catches this: 61,208 characters over the pages actually present is a healthy 1,800 per page, and the same figure against the page count printed on the document's own cover sheet is 1,003, which is the shape of a document half-read.

Those two rates are the same numerator over two different denominators, and the choice of denominator is the entire check. Divide by the pages the chunks happen to cover and the result cannot detect a dropped page, and the reason is mechanical: when a parser drops a page it removes that page's characters from the numerator and that page from the denominator in the same movement, so a rate computed from the surviving data is undisturbed by any amount of loss. The check only acquires power when the denominator comes from outside the extraction — a cover sheet, a page count read from the file's own catalogue, a figure in the vendor's manifest. The exact figure of 1,003 against the sixty-one-page count is the interesting one; 1,800 is a rounded rate against the span the chunks cover, and its roundness is a fair reflection of how little it can tell you.

The three defects add to six, and six plus the fourteen real absences is the twenty spans the run reported, with nothing unaccounted for. That closure is what makes the exercise worth finishing rather than abandoning at the first false positive: every reported span was either a real gap in the index or one of three named harness bugs, so the harness could be fixed rather than distrusted.

def presence(index, doc_id, span, expected_pages=None):
    chunks = index.fetch_by_metadata({"doc_id": doc_id})
    if not chunks:
        return "no_chunks_for_doc_id"           # not a parser verdict
    chunks.sort(key=lambda c: (c.meta["page"], c.meta["offset"]))
    hay = normalise(" ".join(c.text for c in chunks))
    pages = {c.meta["page"] for c in chunks}
    if expected_pages and len(pages) < expected_pages:
        return f"partial_ingest_{len(pages)}_of_{expected_pages}"
    return "present" if normalise(span) in hay else "absent"

Each of the three harness defects has become one line of that function, and the order of the lines is not arbitrary. The empty-chunk check comes first because every later test would read an empty haystack and report absent. The sort comes before the join because a haystack assembled out of order can break a span that is genuinely contiguous in the document. The page-count comparison comes before the substring test because a document that stopped halfway will produce true absences for everything in its second half, and reporting sixty of those as parser defects on individual pages buries the one fact that explains all of them. Each guard removes a way for the final comparison to be asked a question it cannot answer honestly.

Three of the four return values in that function are diagnoses rather than answers, which is the property worth keeping. A triage tool that emits one bucket per genuine cause is usable by whoever is on call; one that collapses four causes into content_absent sends them to the wrong artefact three times out of five and burns the credibility of the whole exercise the first time somebody checks page 12 and finds it fine.

Why "swap the embedder and see" cannot conclude anything#

Three weeks into a rollout, answer quality on a 60-question eval sits at 61%. The plan for the next sprint is a stronger embedding model plus a reranker, with the reasoning that if quality does not move, the loss must be upstream.

That is an elimination argument with no valid conclusion in either direction. If quality rises to 68%, it does not exonerate parsing, because a cross-encoder reranker — a second-stage model that rescores the candidates retrieval already returned by reading the query and the chunk together, rather than comparing two vectors computed separately — routinely adds five to eight points on top of a corpus with holes in it — reordering the documents that are present says nothing about the ones that are not. If quality does not move, the result is equally consistent with a weak eval set, a chunking defect, an unhelpful generation prompt, or a retriever that was already near its ceiling. The experiment costs a sprint and its two possible outcomes support the same set of hypotheses.

This is the same defect as the unnormalised substring check and the appeal to k, at the scale of a quarter's planning. An experiment earns its cost by having outcomes that point in different directions. When "the number moved" and "the number did not move" leave the same set of explanations standing, the experiment is a constant dressed as a measurement, and running it for longer or on more questions changes nothing about that.

What settles it is a partition rather than a comparison. For each of the 60 questions there is, or can be within an afternoon, a gold reference: the document and page containing the answer. Run the presence test for each, then check the retrieval trace, and every failure lands in exactly one of three buckets.

def triage(question, gold, index, retriever):
    chunks = index.fetch_by_metadata({"doc_id": gold.doc_id})
    hay = normalise(" ".join(c.text for c in chunks))
    if normalise(gold.answer_span) not in hay:
        return "content_absent"
    ranked = retriever.search(question.text, k=5)
    if gold.chunk_id not in {c.id for c in ranked}:
        return "not_retrieved"
    return "retrieved_but_wrong"

The three buckets are mutually exclusive and jointly exhaustive by the shape of the function rather than by anybody's judgement. The early returns make each failure land in the first bucket whose condition it meets, so no failure is counted twice; and the final return has no condition attached, so no failure escapes counting. That structural property is what allows the counts below to be added and subtracted at all.

On the corpus this chapter is drawn from, the 23 failures partitioned as follows:

Two arithmetic checks are worth doing on that table before trusting anything built on it. The counts add to twenty-three, which is the failure total, so nothing was dropped between the buckets. And twenty-three failures out of sixty questions leaves thirty-seven passing, which is a shade over sixty-one percent — the eval score quoted above, arrived at by a route that shares no step with the triage. The small gap between "a shade over" and the quoted 61% is the residual, and it is rounding or partial credit on the scoring rubric rather than a missing failure; it is far too small to hide a bucket.

Fourteen of twenty-three failures are cases where no retriever could have succeeded. The maximum the sprint's plan could deliver is the three not_retrieved cases, which is five percentage points, assuming every one of them is fixed. Fixing the parser addresses fourteen, which is twenty-three points. That is the number that changes the plan, and producing it took one engineer about four hours: no GPU, no model calls, and a total inference cost of the 60 retrieval queries.

The two twenty-threes in that paragraph are unrelated quantities that happen to share a numeral. One is a count of failing questions; the other is a share of the whole eval, being fourteen questions out of sixty. Reading the second as a restatement of the first would make the parser's share look like the entire loss, when in fact the remaining nine failures are real and belong to other subsystems.

The measurement has a limitation worth stating. It depends on a gold answer span per question, which someone has to write, and the content_absent bucket can be inflated by a gold span quoted from the printed page in a form that differs from any faithful extraction — small-caps rendered as capitals, an em dash where the PDF has a hyphen. Reviewing the content_absent cases by eye before reporting the number is part of the four hours rather than an optional extra.

What to promise about a corpus nobody has read#

The PM asks how long it will take to fix parsing. There is a version of this answer that is honest and a version that ends badly, and the difference is whether the commitment is to a fixed-cost measurement or to an unbounded quality outcome.

What can be committed to is a characterisation with a known price. Stratify the corpus by source, which means dividing it into groups that differ in the way that matters and sampling within each group rather than across the whole, then sample twenty documents per source, run the parser, and have a human read the output against the original — not the whole document, but a fixed protocol: first page, one page containing a table, one page chosen at random. That is roughly fifteen minutes per document, so 120 documents across six sources is about four engineer-days, and it produces a defect rate per source with the defects named and classified. From there, each source gets a scoped fix with an acceptance test attached to it, and each fix can be estimated because its scope is known.

Both figures in that sentence are worth reconstructing, because a commitment whose arithmetic you cannot redo in front of a PM is a commitment you will discount under pressure. Twenty documents per source across six sources is a hundred and twenty documents. A hundred and twenty documents at a quarter of an hour each is thirty hours of reading, which is four working days for one engineer. The protocol's three pages per document are what hold the quarter of an hour steady: the cost is per document read to a fixed depth rather than per document, so a four-hundred-page contract and a six-page datasheet cost the same. Adding a seventh source adds a further twenty documents and a proportionate slice of a day, which is a sentence that can be said out loud before anybody has looked at that source.

Stratifying rather than sampling uniformly is a decision, and it follows from what is being measured. Defect rate is a property of a source — of one vendor's export settings, one scanner, one template — so a uniform sample across the whole estimates a mixture of six unrelated rates, and its precision improves the mixture rather than any of its parts. Sampling within each source estimates six things, each of which somebody can act on.

The refusals are not caution for its own sake. A single quality number for 40,000 documents from six sources is a weighted average of six unrelated distributions and will be wrong in a way that hides the useful structure — one source at 97% and one at 34% average to something that describes neither. A date for "parsing fixed" implies a defect list that does not exist yet, since the whole point of the characterisation is that nobody has read the output. And an eval-score projection couples a parsing commitment to the behaviour of a model and a prompt you do not control.

The last row of the left-hand column is hours where the first row is days, and the difference is not a discount. A fix is scoped to one named source, so the re-run reads that source's sample rather than all six, and one source's sample is a small fraction of the hundred and twenty documents. The commitment table holds together only because every row in it is scoped to a source; a promise to re-check "the corpus" after each fix would cost four days every time and would not be made twice.

Some documents should not be in the index#

The third answer people find uncomfortable, after the characterisation with a known price and the list of things to refuse, is that a class of documents can be excluded on purpose, with the gap stated to users. When a source parses unreliably and fails silently — handwritten claim forms, decks that are 80% raster charts, forty-column spreadsheets exported to landscape PDF — the realistic options are to invest heavily in a specialised path, to ingest it and accept a stream of confident wrong answers, or to leave it out and say so.

Leaving it out has a property the middle option lacks: the failure becomes visible to the person who can act on it. An assistant that says "I don't have the 2018–2021 scanned claim files; those are in the archive system" sends the user to the right place. An assistant that answers from a 43%-accurate OCR of those files sends the user to a wrong number with a citation attached. In domains where a wrong figure costs more than a missing one — legal, clinical, financial — that asymmetry decides it, and the exclusion should be recorded as a documented scope boundary rather than a backlog item, so that it is reviewed deliberately rather than forgotten.

The asymmetry is the step to be able to reconstruct, because it is what makes the argument specific to these domains rather than general advice. A missing answer costs the reader one lookup elsewhere and leaves them knowing they do not know. A wrong answer with a citation costs whatever the wrong figure costs, and removes the reader's reason to check. Where those two costs are comparable — an internal search over engineering notes, say — ingesting a poorly parsed source is defensible. Where the second is much larger than the first, no accuracy figure short of very high makes the trade work, and the exclusion is the cheaper of the two honest options.

None of this argues that retrieval never fails or that rerankers are not worth having. The three not_retrieved cases in the table are real and a reranker would likely have fixed them. The argument is about order of operations: reading what the parser emitted is an afternoon and usually answers the question, so it comes before a sprint spent on components that cannot reach the majority of the loss. The same ordering applies to the eval set itself: a 60-question set that produces 23 failures is worth partitioning before it is worth expanding, because the partition tells you which of the three subsystems to write more questions about, and a larger set with the same blind spots measures the same thing more precisely.

Everything in this chapter is forensic and reactive, triggered by somebody noticing a bad answer. It works, and it works one incident at a time. It would not have told you the week a supplier changed how it produces the files it sends you, because every one of those files still ingests without error.