A page embedding retrieves what a page looks like, not what it says

An index can only rank things it has a representation of. That sentence sounds like a truism and it hides the mistake this chapter is about, because a representation is not the same object as the input it was computed from. A page rendered as an image plainly contains every character printed on it. The fixed-length vector computed from that image is a summary of the page, and a summary made by pooling is under no obligation to carry any particular character through. What retrieval ranks on is the summary.

A dense vision encoder is a model that takes a picture and returns one fixed-length list of numbers, built so that pictures a person would call similar come back as similar lists. Point it at a rendered page and it gives you a page embedding: a few hundred to a few thousand numbers standing in for everything on that sheet of paper. The question worth asking before building anything on top of one is which properties of a page it preserves and which it flattens, because a query can only ever reach the preserved ones.

What a pooled page vector keeps and what it flattens#

Consider a five-thousand-page hardware manual put through two indexes. Index A is an existing text arrangement: extracted text, chunked at roughly 500 tokens, embedded with a text encoder. Index B is new — each page rendered to a 1,600-pixel PNG and embedded whole with a vision encoder, one vector per page, no text involved at any stage. The pitch for B is that a page image contains strictly more information than its extracted text, so B should be able to answer everything A can and more.

The claim in that pitch is a conflation of two different statements. "The page image contains the part number" is true — the glyphs are right there in the raster, at roughly 9 points, perfectly legible to a human. "The page embedding encodes the part number" is false, and it is the second statement that governs retrieval.

A dense vision encoder maps a page to a fixed-length vector by pooling over patch representations. That vector has on the order of a thousand dimensions and has to summarise a page carrying several thousand glyphs, a layout, ruled lines, headers, and often a figure. The pooling is lossy in a specific direction: it preserves what recurs and discriminates between pages, and it discards what is idiosyncratic and local. Page structure recurs. Typography recurs. The visual signature of "a table of fasteners" recurs. One eleven-character string in row eleven of forty is exactly the kind of local detail that pooling flattens. Two pages differing only in one cell's contents produce nearly identical embeddings, because visually they are nearly identical.

That last sentence is the whole failure stated as a property, and it is worth turning over slowly. The embedding is the quantity being compared, and it holds still when one cell changes. The contents of that cell are the quantity the question is about, and they range over every fastener in the manual. So the thing being measured is nearly constant while the thing that decides whether the answer is right varies across its entire range. Any amount of care in the ranking function operates downstream of that, on numbers that no longer distinguish the cases.

The tempting reply is that the encoder simply needs to be better, and it is tempting because encoders do keep improving on every benchmark anyone runs. What makes it wrong is that the flattening is the encoder doing its job. A single vector of a thousand numbers cannot separately record several thousand glyphs; something must be discarded, and a model trained to make similar-looking pages come back as similar vectors will discard exactly the details that make two similar-looking pages different. Better training moves which details survive. It does not remove the requirement that most of them do not.

Why a rare string is a lexical index's strongest signal#

Lexical search has the opposite bias by construction. An inverted index treats 47-A2210 as a token with a document frequency of two, which makes it the highest-signal term in the corpus. Rare strings are where BM25 is strongest and where dense semantic retrieval is weakest, and it is not an accident that the two indexes fail on opposite populations. BM25 is the lexical ranking function that scores a chunk by which of the query's words it literally contains, weighting a term by how rare it is in the collection.

The document frequency of two is not a separate measurement — it is the same fact as "the string appears exactly twice in the manual", counted by the index instead of by a person. Two instruments that share no step agreeing to the unit is what licenses treating the rarity as real rather than as an artefact of how the text was tokenised. What it also does is bound what lexical matching can promise. Of those two occurrences, one is the fastener row and one is the entry in the back-of-book index, and a scoring function keyed on rarity alone has no reason to prefer either. The reason the fastener row comes first is that the query carries a second word, torque, which the row's column header supplies and the back-index entry does not. Rarity gets the right two candidates into contention. The ordinary word decides between them, which is a residual worth knowing about before assuming a rare string is self-sufficient.

Late interaction, and the string index it does not become#

Multi-vector late interaction, covered in topic 8.15, softens this considerably and does not remove it. A ColPali-style index — named after the paper that introduced retrieving PDF pages as images with late-interaction embeddings — keeps one vector per image patch and scores a query against the maximum-similarity patch, so a query token can align with the specific region of the page where that token is rendered — which is precisely the local detail a single pooled vector destroys. In practice this recovers a good deal of exact-match performance on strings that are rendered large and clean. It degrades on strings that are small, on strings the encoder's tokenisation splits oddly, and on strings that appear in dozens of visually identical positions across the corpus. The mechanism is still visual matching; it has just been given much finer spatial resolution to work with. It is not a string index, and treating it as one produces a system that mostly works on part numbers and then silently does not.

The causal step there is worth making explicit, because "finer resolution" sounds like a difference of degree that could eventually close the gap. Keeping one vector per patch removes the pooling step that was destroying local detail, so the detail survives into the index. What survives is still a visual description of a region, scored by similarity to a visual description of the query. Similarity is the operation, and two renderings of two near-identical strings are similar. Removing the pooling fixed the part of the problem that came from summarising. It left untouched the part that comes from matching on appearance, and that part is what the next section's numbers are measuring.

What each index answers that the other cannot#

The two rows in the middle are the whole argument. There is a population of questions each index answers and the other does not, and neither population is small. On this manual, a 200-question exact-match set — part numbers, error codes, revision identifiers, proper names of subassemblies — scored recall@5 of 0.96 on A and 0.31 on B. A 150-question figure set — pinouts, exploded views, wiring, tolerance diagrams — scored 0.22 on A and 0.79 on B. Neither index is a superset of the other, and the intersection where both work is the ordinary prose that made up most of the original evaluation set and therefore hid the problem during development.

Two of the table's rows are arithmetic rather than observation, and checking them confirms the two indexes are being compared over the same document. Five thousand pages at about 1,030 patches each is a little over five million, which is the multi-vector figure exactly, so nothing is being compressed or sampled in that row. And ~34k chunks over five thousand pages is close to seven chunks to a page, which at roughly 500 tokens each is a few thousand tokens of text on every page — a plausible density for a dense technical manual, and a sign that A really is indexing the same material B is rendering.

Set the size row against the latency row and they refuse to move together, which is the useful thing the table hides. The multi-vector index holds about a thousand times the vectors the single-vector one does, and answers in a little under five times the time. Storage and query cost have come apart because the patch-level scoring is a reranking step: it runs over a shortlist rather than over every patch in the index, so the vector count sets the size of the store and not the size of the scan. The cost of those five million vectors is real and it lands elsewhere — in memory to hold them and in the hours of GPU time in the build row, both of which do scale with the count.

The two evaluation pairs are more informative as gaps than as levels. On the exact-match set A leads by sixty-five points; on the figure set B leads by fifty-seven points. Two gaps of comparable size pointing opposite ways is what "neither is a superset" looks like when measured, and it also means a combined score is decided by the mix rather than by the indexes. Write x for the share of exact-match questions in a set. A scores 0.96x + 0.22(1 − x) and B scores 0.31x + 0.79(1 − x), and the difference between them collects into the exact-match gap times x minus the figure gap times (1 − x) — collecting is allowed because each expression is linear in x and the four slice scores are properties of the indexes, not of the mix. The two are level when sixty-five points of x equal fifty-seven points of (1 − x), which puts x at fifty-seven parts in one hundred and twenty-two: a little under half. The sets actually used weight exact match at four sevenths, above that break-even, so A wins on their union — a verdict about the test set's composition and not about either index.

Absence and substitution#

Both indexes fail. They do not fail equally visibly, and the difference matters more than the raw numbers when deciding what to run in production.

When index A fails on a figure question, chapter 1 described what you get: a topically correct page and a confident wrong answer. That is bad, and it is at least a wrong answer about the right page. If someone checks the cited page, the missing evidence is visible immediately — the chart is right there, the answer does not match it, and the diagnosis takes thirty seconds.

When index B fails on an exact-match query, it returns five pages that all look exactly like the page you wanted. The generator is now reading a fastener table — the right kind of table, in the right document, with the right columns — that simply does not contain row 47-A2210. It has forty other torque values on it. The failure surface is a page that survives every plausibility check a reviewer applies, and the answer is a torque value for the wrong bracket, stated with the same confidence as a correct one.

The reviewer's check is the second quantity here that stays constant while the dangerous property varies. "Does the cited page look like the page this question should be answered from" returns yes for the right page and yes for all five near-identical ones, so it separates nothing. The belief that a human in the loop catches this is reasonable everywhere the wrong page looks wrong, and a corpus of near-identical parts tables is precisely where that condition does not hold. It is worth being blunt about the consequence: a technician who applies 22 Nm where the spec says 9 Nm has no signal that anything went wrong, and the torque is out by a factor of well over two while the retrieval was out by one row.

An analogy helps here, offered as an analogy: matching a person by their silhouette rather than by their name. Silhouettes group people usefully and they cannot tell you which of two people of the same build you have got. The comparison breaks in the direction that makes the real case worse rather than better. Silhouettes are drawn from a fairly wide range, while page layouts in a technical manual are produced from a handful of templates, so the population of confusable items is far larger than the analogy suggests.

There is a way to see the difference before customers do, and it costs one column in the evaluation harness. Tag every query in the test set with whether it contains a rare literal — a token appearing fewer than, say, five times in the corpus, which is a corpus statistic you already have from the inverted index. Rare-literal queries are the population where a visual-only system substitutes rather than fails, and reported separately they show the problem immediately: on this manual that slice was 12% of the question set and 61% of the wrong answers. Aggregated into a single accuracy number, the same slice moved the total by four points and looked like ordinary variance.

Those two readings are the same measurement, and the distance between them is the point. A population holding 61% of the errors while making up 12% of the questions is carrying about five times its share, which is the kind of ratio that stops a release. The same population moving a headline by four points is inside the noise anyone would tolerate on a few hundred questions. Nothing changed except whether the slice was reported on its own, and a metric that can be made alarming or invisible by a reporting choice is not measuring the system so much as the harness.

The asymmetry is about how far the retrieved evidence is from the truth in the reviewer's eyes. A missing chart is an absence, and absences are noticeable once you look. A near-miss page is a substitution, and substitutions are the hardest defect class to catch by inspection because everything about them is right except the one thing being asked for. In production this shows up in the support queue as a trickle of reports that the system "gets some numbers wrong sometimes", with no pattern anyone can pin down, because the pattern is a query property (does the question hinge on a rare literal string?) rather than a document property.

Retiring the text index#

The proposal that follows from B working is to retire A and run a single visual pipeline, and the reasoning against it does not depend on how good the visual encoder gets. The two indexes are not two implementations of the same capability at different quality levels. They match on different things — one on strings, one on appearance — and a corpus of technical documentation contains questions of both kinds in roughly stable proportions. Retiring A does not trade a little precision for a lot of recall; it removes the only component in the system that can find a literal.

There is a version of the proposal worth taking seriously, which is retiring A's role as the sole index rather than retiring A. Once B exists, a large share of A's chunks are earning nothing: prose paragraphs that B also handles adequately, boilerplate headers, tables of contents. Pruning A down to what it is uniquely good at — a lexical index over extracted strings, with the dense text half dropped — is a real cost saving and does not remove the capability. What you cannot do is drop the string-matching path and expect a visual retriever to cover it.

The counter-argument you will hear is that the failure is fixable with a better encoder, or with higher resolution, or with more patches. Resolution genuinely helps: an 8-point part number that occupies four pixels of height at 1,024 px is unreadable to the encoder and legible at 2,048 px, and the ColPali paper's whole premise is that keeping the page as an image and matching at patch granularity beats extracting text first on the document-retrieval benchmarks it targets. The relation behind that is simple enough to reason with directly: glyph height in pixels scales with the render resolution, so doubling the render doubles the height of every character on the page, and a glyph that fell below whatever the encoder can resolve may rise above it.

But the ceiling here is structural rather than incidental. A visual matcher scores similarity, and similarity between two renderings of near-identical alphanumeric strings is high by definition — 47-A2210 and 47-A2110 look almost the same and mean entirely different fasteners. An inverted index treats them as unrelated tokens. That distinction is free in one representation and expensive in the other, and no amount of encoder scaling inverts it. Raising the resolution makes the glyphs legible; it does nothing about the fact that one legible glyph differing in the middle of a string produces two images that a similarity function is built to call alike.

Assume you now have both indexes and a fused candidate list that puts the right page image in front of the generation step. The failure from chapter 1 is only half fixed at that point: if the generator still receives the OCR text of that page, the answer is wrong for exactly the reason already diagnosed, and the fact that a visual retriever found the page changes nothing. Retrieval decides which page is discussed. It does not decide what the model is shown.

The pixels have to reach the model, which turns out to have its own constraints — a page image costs an order of magnitude more tokens than a text chunk, only a handful fit in one prompt, and the downscaling you do to make them fit is what erases the axis labels and small table cells you retrieved the page for.