Page, figure or table: the indexed unit decides what a citation can point at

Every retrieval system has to decide what one entry in its index stands for. That entry is its indexed unit: the thing that gets a vector, the thing a similarity score is computed for, and the thing whose identity comes back when the query matches. In a text system the unit is usually a chunk of a few hundred words, and the choice feels administrative. In a visual system the obvious unit is a rendered page, and it is easy to take that as given, because pages are what a document is already divided into and rendering one is a single call.

Taking it as given costs two things that are difficult to recover afterwards. A unit is what competes in the ranking, so a small figure on a page of prose is judged as part of that page rather than on its own merits, and it can lose to pages that are wholly about the query's topic while containing none of its answer. A unit is also what a result is identified by, so a citation can name a page and nothing narrower. Both consequences are set before a single query runs, by a decision that is often nobody's in particular.

Why a page vector under-weights a small figure#

Take a concrete page. Page 212 of an annual report is mostly prose: two columns of management commentary, about 900 words, and in the lower right quadrant a small bar chart of segment revenue occupying roughly 11% of the page area. A query asks which segment grew fastest last year, and the answer is in that chart.

The reason a page-level index handles this badly follows from how a page vector is built, and the steps are worth naming one at a time. A visual encoder does not read a page as a whole. It divides the image into a grid of small patches and produces one vector per patch. Those patch vectors are then pooled, which is to say averaged into a single vector standing for the whole page. Because the grid is uniform, the number of patches covering a region is proportional to that region's area. And because pooling averages, every patch carries the same weight in the result. So a region's contribution to the page vector is proportional to its share of the page, and nothing else about the region enters into it.

Put the page's own figures through that chain and three numbers turn out to be one number said three ways. That is the pooling behaviour from chapter 2 seen from a different angle: the page vector is a summary of the page, the page is 89% commentary, and the chart contributes about a ninth of the signal. Eleven parts in a hundred and one part in nine are the same statement, and the commentary's 89% is the remainder of the first, so the accounting closes with nothing unattributed. The retriever is not failing to see the chart. It is weighting it correctly for a page-level representation, and a page-level representation is the wrong granularity for a question about one figure on that page.

What that closure licenses is the important part: no encoder change addresses this, because the ratio was fixed by the page's layout before any model was involved, and a fix therefore has to change the denominator rather than the model. The one place the proportionality is approximate is the assumption that every patch counts the same. Where an encoder weights some patches above others the ninth becomes an estimate, and the area alone does not say in which direction it errs.

Measured across 300 figure-backed questions on a mixed corpus, recall@5 from a whole-page index sorts almost perfectly by how much of the page the figure occupies: figures covering more than 40% of the page came back at 0.86, figures covering 15–40% at 0.61, and figures under 15% at 0.34. A slide deck, where a figure usually is the page, looks like a triumph for page-level indexing. A dense report does not, and the difference is a property of the documents rather than of the encoder.

Those three numbers are more useful read as the gaps between them than as three levels. From the top band to the middle is a fall of about a quarter, and from the middle band to the bottom is a fall of about the same size again. Recall does not hold up and then collapse at one threshold; it comes down at a roughly even rate as the figure shrinks. So there is no safe figure size to sit just above, and the question a team can actually answer is not "are our figures big enough" but "which band is our median document in".

The single page and the three-hundred-question sweep are separate measurements that agree. Page 212's chart, at about a ninth of the page, falls in the bottom band, where roughly one question in three finds its figure inside the top five results. A page returned at rank 24 is one of the other two in three. The anecdote is not a worst case dug out to make a point; it is the ordinary outcome for a figure of that size, which is why the fix has to be structural.

The unit is a choice, and it propagates#

Anyone who has picked an index in a relational database knows the shape of this decision: the index determines which queries are cheap, and no amount of query tuning recovers a lookup the index cannot serve. The indexed unit in a visual retrieval system is the same kind of commitment, with the added wrinkle that it also determines what your citations can say. A retrieval result carries the identity of the unit that matched, so a page-level index can only ever hand downstream a page. The bounding box from chapter 3 was the model reconstructing, at answer time, a locality the index had thrown away at build time.

That comparison is an analogy and it has a limit worth naming, because the limit is where the decision gets its weight. A relational index can be added later over rows that never went anywhere, and a query that has no index to use falls back to a scan: slower, still correct, still complete. A figure unit that was never built has no scan to fall back to. The figure is not slow to retrieve, it is absent, and the only way to add it is to re-run detection and encoding over the whole archive. The analogy also understates the reach of the choice in a second way. A database index never changes what a row is; here the unit decides what counts as a retrievable thing at all, which is why it propagates into citations and prompts rather than staying inside the retrieval layer.

Four units are in common use, and they trade against each other on dimensions that matter differently per corpus.

The last row is the one that decides whether the unit choice is additive or exclusive. A page index has no detection step, so it has no detection failure mode; whatever is on the page is in the index, at bad granularity. A figure index depends on a layout model finding the figure, and a figure the detector missed does not exist as far as retrieval is concerned. That is a hard failure with no fallback, and on the corpora where it matters most — dense reports with inline plots and no ruled frames — detection recall is the thing to measure before anything else. A layout detector at 0.93 figure recall means 7% of your figures are unreachable, permanently, no matter how good the encoder is.

The step from that recall to that miss rate is a subtraction, and it is only legitimate because of the row above it. Recall of 0.93 says ninety-three figures in a hundred were found; the complement is the seven that were not. Turning "not found by the detector" into "unreachable" needs the last row to be true — needs there to be no other unit in the index that contains the figure. In an index that also holds pages, those same seven figures are still reachable at page granularity, and the subtraction gives a granularity loss rather than a loss of the evidence. The word doing the work is permanently, and it is earned only by a figure-only index.

Read the first two rows across rather than down and the table settles a question that otherwise gets argued from taste. The tight crop is Strong on retrievability and "often not" interpretable. The figure-with-caption unit is Strong on retrievability and interpretable, and it carries exactly the same detection dependency, the same order of unit count, and a citation granularity that is strictly finer. There is no column in which the tight crop wins. It is dominated, and the next section is about why it nevertheless gets built first.

Two more readings fall out of the same table. The unit-count row is counter-intuitive in a way that catches people budgeting storage: a figure index over a 5,000-page manual is smaller than the page index, at roughly three figure units for every five pages, because most pages carry no figure at all. And the row index is the largest of the four by unit count while being the only one whose entries are not images, which is why it prices differently from everything else here. The value row is the other singleton — "Filterable on values" is Yes in exactly one column, and that column is also the only one needing cell extraction on top of detection. One unique capability, bought with one extra dependency.

Cropping tightly is the tempting mistake#

Once you decide to index figures, the obvious implementation is to take the detector's bounding box and crop to it. The crop is small, so it encodes cleanly and costs few tokens at generation time, and it contains exactly the plot and nothing else. It also frequently cannot be interpreted.

A detector's box around a chart usually excludes the caption, which sits below it and is a separate layout element. Depending on the detector it may also clip the axis tick labels and the legend, which are visually separated from the plot area by whitespace. What comes back is a rectangle containing three coloured bars of different heights. The model can read the ratios between them and cannot tell you what they measure, what units they are in, which segments they represent, or what period they cover. In a spot check of 120 tight crops, 34 could not be interpreted without their caption — the model either declined or produced an answer that was internally consistent and about the wrong quantity.

A little over one crop in four, then, and that is the "often not" from the table's second row with a number on it. The two failure modes inside that share are not equally survivable. A decline is loud: it appears in the output, it can be counted, and a system that declines too often gets looked at. An answer that is internally consistent and about the wrong quantity is the dangerous half, because self-consistency is the thing a reviewer checks. The bars really are in that ratio, the sentence really does describe them, and the only thing wrong is the quantity the bars measure — which is precisely the fact the crop removed.

The fix is padding plus explicit attachment, and it is worth being deliberate about which of the two is doing the work. Padding the box by a fixed fraction — 6% of page width on each side, 10% vertically — recovers axis labels and legends that sit just outside the detected plot area. It does not reliably capture the caption, which may be a full line-height away and may be above the figure in some templates. Captions are better attached explicitly: find the nearest text block whose reading order neighbours the figure and whose first token matches a caption pattern, and index it as text alongside the crop rather than trusting the crop to swallow it.

The tempting belief here is that padding is a defence, and it is worth saying why it is tempting and why it fails. It is tempting because it demonstrably works on the thing you tested it against: axis labels and legends sit a predictable short distance outside the plot area, so a fixed fraction catches them, and the crops you inspect afterwards look complete. What makes it unsafe is that the padding is a constant while the thing it has to catch is not. A caption's distance and its side of the figure are properties of the document template, and a fraction tuned on one template is a fraction untuned for the next. Nothing in the output announces the difference: a crop with its caption and a crop without one are both well-formed images of a chart, and the second only reveals itself when a question needs the units.

def figure_unit(page, box, page_w, page_h):
    pad_x, pad_y = 0.06 * page_w, 0.10 * page_h
    crop = (
        max(0, box.x0 - pad_x), max(0, box.y0 - pad_y),
        min(page_w, box.x1 + pad_x), min(page_h, box.y1 + pad_y),
    )
    caption = nearest_caption_block(page, box)   # may be None
    return {
        "unit_id": f"{page.doc_id}#p{page.number}#fig{box.index}",
        "unit_type": "figure",
        "doc_id": page.doc_id,
        "page": page.number,
        "bbox_norm": [c / d for c, d in zip(crop, (page_w, page_h, page_w, page_h))],
        "detector": {"model": "layout-v4", "score": round(box.score, 3)},
        "render": {"long_edge_px": 2048, "path": f"renders/{page.doc_id}/p{page.number}/fig{box.index}.png"},
        "caption": caption.text if caption else None,
        "parent_unit_id": f"{page.doc_id}#p{page.number}",
    }

Two fields in that record exist purely for later. parent_unit_id lets fusion recognise that a figure crop and its page are the same evidence, which chapter 6 needs when deduplicating a merged candidate list. detector.score is what you sort by when someone reports a figure the system cannot find and you want to know whether the detector saw it at all — a figure absent from the index and a figure indexed at detector score 0.31 are different bugs with different fixes.

That last distinction is worth spelling out, because it is the difference between two investigations that would otherwise look identical from the outside. A figure with no record at all means detection never fired on it, and the fix is upstream: a different detector, a lower threshold, a retrained model. A figure present at a low detector score means detection fired and ranking buried it, and the fix is downstream: the crop exists, so it can be re-encoded, re-scored or promoted. Without the score in the record both cases present as "the system cannot find my chart", and the team guesses.

A table indexed as an image answers a different question than a table indexed as rows#

Tables sit awkwardly across the whole framing so far, because they are simultaneously the most visual and the most structured thing in a document. Their meaning is entirely positional — a number means what it means because of the row and column it sits in — and yet unlike a chart, every bit of their content is text.

Indexed as an image, a table retrieves and reads well. A model looking at a rendered table gets cell alignment for free and answers "what was Q3 gross margin in the EMEA segment" by reading across a row, which is exactly the operation OCR's flattening destroys. What it cannot do is answer "which quarters had gross margin below 20% in any segment", because that question requires scanning values across many tables, and an image index can only return tables that look relevant, after which a model has to read each one. Twelve retrieved table images at 3,000 tokens each is 36,000 tokens to answer a question that a SELECT over extracted rows answers exactly and instantly.

That multiplication is the whole argument in one line, and it is worth seeing what makes it a fair comparison rather than a rhetorical one. The per-image figure is the cost of a page-sized render at the resolution the previous chapter priced, so a table image is priced like a page image and not like a text chunk. The lookup question needs one such image; the aggregation question needs as many as there are tables that might qualify, and the model must read every one before it can rule any out. So the cost of a lookup is flat in the size of the corpus and the cost of an aggregation is linear in it — which is the property that decides the unit, rather than the size of either number on its own.

Indexed as structured rows, the reverse holds: filtering and aggregation become possible, and you have committed to cell extraction being correct. A merged header cell misread as spanning two columns shifts every value in the table by one position, and nothing downstream will notice, because shifted values are still numbers of a plausible magnitude in a plausible schema. This is the substitution failure from chapter 2 wearing different clothes, and it argues for indexing tables both ways on any corpus where the numbers matter: rows for filtering, image for reading and for citation, linked by a shared parent_unit_id so a filtered row can show a reviewer the actual rendered table it came from.

The defence people trust here is the schema, and it is the clearest case in the chapter of a quantity holding still while the dangerous property moves. Type checks pass, ranges pass, required columns are present, row counts are what the table said they would be — all of that is invariant to the shift, because a shift permutes values between columns without creating or destroying any. The property that actually varies is which column each value belongs to, and no check that looks at one column at a time can see it. That is why the second index earns its cost: an image of the table is the only artefact in the system that still carries the original alignment.

What multi-granularity indexing costs#

Indexing pages and figures and tables means the same evidence appears in the index three times, and the consequences are not only storage.

Retrieval returns near-duplicates. A query about the segment chart retrieves the figure crop at rank 1 and its parent page at rank 3, and both occupy slots in a top-k that was meant to hold distinct evidence. Without deduplication, a five-slot prompt can end up carrying two views of one figure and nothing else.

Scores are not comparable across unit types. A figure crop is a small, visually homogeneous image and tends to score higher against a matching query than a busy full page does, because its embedding is less diluted. The higher score is not a sign that it is better evidence. Mixing units in one ranked list applies a systematic bias toward small units, on top of the cross-modal calibration problems that are chapter 6's subject.

That bias is the same mechanism as the opening section, run backwards. A page vector averages patches over the whole page, so a query's matching region is diluted in proportion to how little of the page it covers; a crop vector averages patches over the region alone, so there is nothing to dilute it with. A crop of a chart and the page containing that chart therefore score differently for a reason that has nothing to do with which is better evidence, and the difference grows as the figure gets smaller — which is exactly the case where the crop was added.

And the cost multiplies at build time in a way that is easy to under-plan: rendering 5,000 pages at 2,048 px, running layout detection over each, then rendering roughly 3,100 figure crops and 1,400 table crops, then encoding all of it, is on the order of 9,500 encoder passes rather than 5,000. On a single GPU at about 6 pages/second for the page encoder and 14/second for crops, that is roughly 25 minutes of pure encode time for this manual and about 34 hours for a 400,000-page filing archive — which is fine as a one-off and is a real constraint when someone wants to re-index after a detector upgrade.

The pass count closes exactly, and it is the one figure in that paragraph you can rebuild from the table. Five thousand pages, plus roughly three thousand one hundred figure crops, plus one thousand four hundred table crops, is nine thousand five hundred with nothing left over. What that licenses is a build cost that is additive per unit type: adding table crops to an index that already holds pages and figures costs the table crops and nothing else, so the plan can be priced one unit type at a time.

The table's row count and this pass count sit near nine and a half thousand apiece, and they are counting different things, which is worth stopping on because it looks like a discrepancy. The table's ~9,400 counts rows pulled out of tables — nearly seven to a table, on those 1,400 tables — and rows are text records that never go near the image encoder. The 9,500 counts encoder passes over images. The near-coincidence is exactly that, and reading either as a check on the other would be a mistake.

The two time figures close on each other as well, and noticing that they do tells you what kind of estimate the second one is. Four hundred thousand pages is eighty times five thousand, and about 34 hours is roughly eighty times roughly 25 minutes. So the archive figure is not a second measurement; it is the manual's figure scaled by corpus size, which is legitimate because every term in the build is per-unit and the unit mix is assumed to hold. The one thing that does not close is the arithmetic from the throughputs: five thousand page passes at six a second is a bit under fourteen minutes, four and a half thousand crop passes at fourteen a second is a bit over five, and the two together come to a little over nineteen rather than twenty-five. The quoted figure carries about a quarter more than the rates as stated will account for, so treat it as an allowance rather than as a sum, and re-measure it on your own hardware before planning a re-index around it.

Choosing the unit for a corpus#

The decision is corpus-shaped rather than universal, and three properties of the documents settle most of it.

The first is figure area as a fraction of page area, which you can measure directly from the layout detector's output over a sample of a few hundred pages. If the median figure covers more than about 40% of its page, page-level indexing is adequate and the detection dependency is not worth taking on; slide decks, pitch material and one-figure-per-page technical notes fall here. If the median is under 15%, figure-level units are close to mandatory, because the page index will simply not surface them.

Those two thresholds are the boundaries of the top and bottom bands, and the middle band is where the rule stops giving an answer. A corpus whose median figure sits between them retrieves at about six in ten from a page index, which is neither good enough to rely on nor bad enough to force a rebuild. There is no figure to appeal to there, and the decision moves to the two properties below — which is the honest position, and better than a threshold invented to cover the gap.

The second is whether the questions asked of the corpus aggregate across documents. "Which quarters had margin below 20%" is an aggregation and needs structured rows; "what was margin in Q2" is a lookup and does not. A corpus that is only ever asked lookups can index tables as images and skip cell extraction entirely, which removes a whole class of silent shift errors.

The third is how much the citations are relied upon. A research assistant whose users skim answers tolerates page-level citation. A system whose output feeds a compliance filing does not, and region-level citation forces figure units regardless of what the recall numbers say, because the recall gain is not what you are buying.

Detection recall cuts across all three and can override them. A corpus of dense reports with inline plots and no ruled frames may want figure units by every argument above and still not be able to have them, because a detector that finds nine figures in ten hands you an index missing a tenth of the evidence with no signal that anything is absent. In that position the page index is not a compromise, it is the only unit that is complete, and the work belongs on detection before it belongs on granularity.

Every decision in this chapter — which unit, how much padding, whether tables go in twice — is stated as a trade-off, and so far the only evidence offered for any of them is recall numbers on a question set. That evidence is weaker than it looks. A retrieval score of 0.79 on a set of screenshots and questions can be produced by a system that reliably finds pages resembling the question and never checks whether the page can answer it, and a test set assembled the obvious way rewards exactly that behaviour. Before any of these choices can be defended, the measurement has to be able to tell resemblance from evidence.

What survives from this chapter regardless of how that measurement turns out is the ordering. The unit is chosen once, before any query exists, and everything downstream inherits it: what can win a ranking, what a prompt slot contains, what a reviewer can be shown. A page index is complete and coarse; a figure index is precise and conditional on a detector; a row index is queryable and conditional on cell extraction. Those are three different bets on which failure you would rather have, and the only wrong move is to make the bet by default and discover its terms from a support ticket.