VLM-based parsing

Read these first

• Document parsing
• Image inputs

What you will know

• Parse a page as an image with a vision model and compare against a parser

  Use identical pages and evaluate normalized text, reading order, coordinates, table structure, and extracted chart facts rather than relying on visual inspection alone.

• Build a benchmark containing layout-heavy pages, tables, and charts, then measure where each parser wins

  Include ground truth for cell relationships, captions, legends, and reading order; aggregate quality by page type because average accuracy hides costly outliers.

• Measure per-page cost and end-to-end latency for both paths at representative corpus scale

  Include image encoding, model retries, concurrency limits, and downstream validation, since token price alone understates ingestion cost and tail latency.

• Implement routing by document and page type using measured quality, cost, and latency thresholds

  A mixed corpus usually needs a cheap default with targeted escalation, while low-confidence or high-value pages need a deliberate fallback rather than silent substitution.

• Reproduce extraction failures caused by reading order, merged cells, chart legends, and small text, then patch or reject the output

  Vision models can produce plausible but fabricated structure, so preserve page-level evidence and fail closed when the result cannot support reliable retrieval.

Chapters

1. 1

   A vision model transcribes a page; a parser copies it

   Both paths end in text, but one lifts the exact characters the producer embedded while the other looks at pixels and writes what it believes is there. That difference decides even the unlimited-budget case: on a born-digital single-column page the parser is correct by construction, whereas re-transcription can drift on a long account number. You render pages to images, run both, and compare normalised text, reading order and coordinates. The two paths break reading order differently, and you reproduce both: the parser interleaves columns because it sorts glyph placements, while the vision model narrates the page as a person would and can drop a sidebar or fold a pull-quote into the body with nothing missing from the output to show it.

   Read chapter
2. 2

   A gap-free table out of a 150 DPI scan is the alarm

   A smudged invoice returns garbled fragments from the parser and a complete, well-formed table from the vision model, and the complete one is the failure. Cells the pixels could not support were produced from a prior about how invoices look, so the empty cell was the honest signal and filling it from the other path deletes it. Validation that flags gaps catches the wrong output. Re-render at higher resolution and run again.

   Read chapter
3. 3

   What you ask the model to emit decides what can go wrong

   Before judging vision output you have to specify it: one page per call, at a resolution you chose, with the model told what kind of document it is looking at, and an emitted structure that is a constrained schema rather than free-form markdown — cells bound to their resolved header labels, coordinates on every block, legend entries named as their own field. The format settles in advance which defects are even representable. Rendered markdown cannot express a header merged across two fiscal years, so that defect becomes invisible until a customer finds it, while a schema carrying each column's resolved label shows it at extraction time. Coordinates you never asked for cannot be compared later.

   Read chapter
4. 4

   Side-by-side markdown outputs are not a measurement

   Two blobs pasted into a shared doc establish which output reads more pleasantly, not which one is right. Page-level ground truth replaces it: normalised text, reading order, cell-to-header relationships, caption and legend attachment, and the specific facts each chart states, over a sample stratified into prose, layout-heavy pages, tables and charts. Each dimension is scored on its own, because one blended figure buries the pages that cost money.

   Read chapter
5. 5

   A chart's quantities appear nowhere on the page as text

   A table's fabricated cell can sometimes be caught against a text layer or a column total; a fabricated data point cannot be caught against anything, because the chart states no string for the values it depicts. The model reads them off pixel positions against axis ticks and binds each series to a legend entry by proximity and colour, so a mislabelled legend silently swaps two series and every number stays plausible. What remains checkable is the surround: axis labels, units and the caption, the count of series against the count of legend entries, and the magnitude and direction of the extracted points against what the prose around the figure says.

   Read chapter
6. 6

   A 14-point corpus average argues for escalation, not migration

   When the vision path comes back fourteen points ahead across the whole corpus, that is the wrong shape of number for the decision in front of you. On a corpus that is mostly prose, such an average means a layout-heavy minority moved enormously while the bulk tied, so it supports escalating some pages rather than moving all of them at forty times the cost. Reporting by page type turns the same benchmark into something a routing rule can consume.

   Read chapter
7. 7

   Per-page cost is not the provider's per-image price

   A backfill budgeted from the image price landed three times over and took nine days instead of two, and those are two separate misses with two separate causes. Rendering and encoding, the output tokens of a full-page transcription, retries and a validation pass account for the money; a provider concurrency ceiling that turns throughput into a queue accounts for the wall clock. Fixing only the token arithmetic misses the next budget the same way.

   Read chapter
8. 8

   Route on a measured signal and record which path ran

   With the parser as default and the vision model reserved for pages that earn it, the rule needs something measurable underneath: detected table and figure regions, text yield against page area, parser confidence, each with a threshold taken from the benchmark rather than intuition. Every page stores which path produced it so the rule stays auditable. A classifier with no fallback quietly substitutes on exactly the pages it misjudges.

   Read chapter
9. 9

   A merged header span relabels every column beneath it

   Header cells merged across two fiscal years flatten into a span that gives each column under it the same label, so segment figures attach to the wrong year while the table stays well-formed and its column totals still add up. Reproducing that six weeks later needs the page image and the model version that ran, and you kept neither. This chapter settles what page-level evidence to retain and what failing closed costs in coverage. It also answers the objection that you cannot know in advance which extractions are wrong: the signals are post-extraction and apply to a page class rather than to individual correctness — two models or two runs disagreeing on the same page, an emitted cell grid that does not line up with the detected layout regions, and resolution and confidence floors below which nothing is indexed.

   Read chapter