A vision model transcribes a page; a parser copies it

Turning a page of a document into text is done by one of two operations, and which one ran decides what can be wrong with the result. The first walks the file. A layout parser reads what the producer stored — glyph placements with coordinates and a font encoding, or, for a scan, whatever optical character recognition recovered from the pixels — and reconstructs a reading order over that material. The second looks at a picture of the page. A vision language model, abbreviated VLM, is a language model whose input can include images alongside text, so a rendered page arrives as pixels and the reply comes back as ordinary text — prose, a table, or whatever structure the instruction asked for. Pointing one at document pages, which means rendering each page to an image, handing it over, and keeping what the model writes about it, is VLM-based parsing, and it is the alternative to the route the previous topic covered, where text is recovered from what the PDF file itself already contains.

The reason that alternative exists is what a parser has no access to. A parser works on what the producer put into the file, and everything a page communicates by arrangement rather than by characters is absent from that material. That a shaded box is an aside rather than the next sentence of the body, that a header cell spanning two columns silently relabels both of them, that a bar rises to a little under 4,000 against an axis whose value is printed nowhere near the bar: none of it is in the content stream to be looked up, so a text parser discards it. A VLM has the arrangement because arrangement is what an image is, which is why the same call can read a scan with no text layer at all, say what a chart depicts, and narrate a two-column spread in the order a person would read it.

Where this shows up in practice is as a second path inside an ingestion pipeline, running beside the parser rather than replacing it: scanned supplier invoices, financial statements printed landscape across eleven columns, slide decks exported to PDF, engineering drawings whose only text lives inside the picture. Hosted "document extraction" products are largely this arrangement behind a document-shaped API, and there is a further variant, taken up in chapter 8, that skips extraction altogether and hands the page image to a model at question time.

Copying and predicting are different operations#

The previous topic established that a PDF stores glyph placements rather than paragraphs, and that reading order is something a parser reconstructs rather than reads. What that leaves out is where the characters come from, and that is the axis this topic turns on.

A parser walks the content stream and resolves each glyph to a code point through the font's encoding table. The character identity is a lookup. If the producer embedded 9268, the parser emits 9268, and there is no mechanism by which it could emit 9286 instead — no scoring, no ranking, no alternative candidate.

It is worth being precise about what "no mechanism" means, because it is the load-bearing claim on this side of the comparison. A font encoding is a table from a byte in the content stream to a Unicode code point, and reading a table has exactly one outcome per key. There is no distribution over outcomes, so there is nothing to sample from, so there is no second-best answer that an unlucky draw could return instead. A parser can fail to find the table, in which case it emits nothing or a replacement character, and you can see from the output that it did. What it cannot do is find the table, read it correctly, and hand back a different character that happens to look reasonable in context.

The operation therefore has no error term on character identity. It has plenty of error terms elsewhere: which glyph belongs to which column, where a paragraph ends, whether a number belongs to the header above it or the one two rows up. Those are all reconstruction, and reconstruction can be wrong.

A vision model receives a raster. The page is cut into patches, the patches become embeddings, and text is produced by decoding — each output token sampled from a distribution conditioned jointly on the pixels and on the language prior, meaning everything the model has learned about which characters, words and layouts tend to follow which in documents of this kind.

Follow that one position at a time, because the whole asymmetry is visible in a single decoding step. At each position the model holds a distribution over the next token, and two things shape it: what the pixels in the relevant region support, and what usually comes next in a document of this kind. Where the pixels are informative, the image contribution is sharp, the distribution collapses onto one token, and the prior barely matters. Where the pixels are uninformative — a smudged glyph, a digit reduced below the size that survived the rendering, a region the model attended to weakly — the image contributes almost nothing, and what is left standing is the prior. The decode still has to emit something, because emitting a token is what each step does. So the token that comes out of a low-information region is drawn from what usually follows rather than from what is on the page, and it arrives in the output looking exactly like a token that was drawn from the page.

Whether that is a free repair or a fabrication depends entirely on the shape of the prior at that position, and the two extremes are worth holding side by side. Halfway through an ordinary English word the prior is sharply peaked: given the letters so far, only a few continuations are plausible at all, so the pixels have only to narrow the field and the prior finishes the job correctly. That is a powerful error-correcting code, and it is the reason the vision path can read a degraded page at all. Halfway through a twenty-two character banking identifier the prior is close to flat, because every digit is about as likely as every other and nothing in the preceding characters constrains the next one. The error-correcting code is gone, the pixels have to carry the entire load, and where they cannot the model still emits a digit — a plausible one, indistinguishable in the output from a digit that was read.

That conditioning is the whole reason the vision path can read a smudged scan, understand that a shaded box is a sidebar, and tell you what a bar chart depicts. It is also the reason 9268 can become 9286. The capability and the exposure are not two features that could be tuned apart. They are one operation described from two sides.

Consequences fall out of that asymmetry directly:

Read the middle of that table as a set of gaps rather than a set of levels. The parser column reads No down five consecutive rows, and the same word is doing two opposite jobs inside that run. On the first three rows it is a guarantee: the parser cannot misspell, cannot invent a cell, cannot drop a region without a trace. On the next two it is a limitation: the parser cannot read a page with no text layer, and cannot state a quantity a chart depicts but never prints. The vision column reads Yes down all five, and the same word is doing the same two opposite jobs. No setting separates the halves, because the mechanism that produces the capability rows is the mechanism that produces the exposure rows.

The last row of the failure signature column is the one that governs everything else in this topic. A parser's failures are loud in the output — you can see the interleaving, you can see the naked numbers with no header attached, you can grep for a clause you know is on the page and find the two halves in the wrong places. A vision model's failures are quiet, because the same machinery that produces good output when the pixels support it produces equally fluent output when they do not.

Running both paths over the same pixels#

Comparing the two honestly means both of them see the same page and you compare on named channels rather than on impression. The parser reads the PDF; the vision model reads a raster you produced, so the raster is now part of the experiment and has to be recorded alongside the results.

One requirement governs the comparison, and everything else in it is a decision somebody makes. The requirement is that both paths describe the same page: if the raster is not a faithful rendering of what the parser read, the two outputs are not two answers to one question. Every other setting is a design decision, and each one moves the numbers. The render resolution is a decision. Whether the raster is kept alongside the result is a decision. The wording of the instruction is a decision. The normalisation applied before the strings are compared is a decision. Writing them down is what makes a later run a repeat rather than a fresh experiment with the same name.

import base64, io, json
import pypdfium2 as pdfium
from anthropic import Anthropic

client = Anthropic()
DPI = 200                      # 1700 x 2200 px for US Letter

def render(pdf_path: str, page_index: int, dpi: int = DPI) -> bytes:
    page = pdfium.PdfDocument(pdf_path)[page_index]
    bitmap = page.render(scale=dpi / 72)
    buf = io.BytesIO()
    bitmap.to_pil().save(buf, format="PNG")
    return buf.getvalue()

def vlm_transcribe(png: bytes, doc_kind: str) -> str:
    msg = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=4000,
        messages=[{"role": "user", "content": [
            {"type": "image", "source": {
                "type": "base64", "media_type": "image/png",
                "data": base64.b64encode(png).decode()}},
            {"type": "text", "text":
                f"This is one page of a {doc_kind}. Transcribe every text "
                f"region on the page in natural reading order. Do not "
                f"summarise, do not omit sidebars, captions or footnotes."},
        ]}],
    )
    return msg.content[0].text

The comment on the DPI constant checks out against the constant itself, which is a small thing worth doing on every such line you inherit. US Letter is eight and a half inches by eleven, scale=dpi / 72 converts a resolution in dots per inch into a multiplier on the 72-point typographic inch, and eight and a half inches at two hundred dots to the inch is seventeen hundred dots across while eleven inches is twenty-two hundred down. The comment reproduces from the code with nothing left over, so the renderer is doing what the line above it says.

The instruction is doing real work here and it is worth reading closely. "Do not omit sidebars, captions or footnotes" exists because the default behaviour of a model asked to transcribe a page is to produce something a human would want to read, and a human reading a risk-factors spread aloud would skip the shaded box. Naming the regions does not eliminate the omission; it moves the rate.

Comparison then happens per channel, on normalised strings, not by reading two blobs. Normalisation here is Unicode NFKC — the compatibility fold that maps ligatures, full-width characters and other visually equivalent encodings onto one form — together with soft-hyphen removal, quote folding, whitespace collapse and case folding, so that neither path is charged for a difference in how it encoded a character both of them read correctly.

import re, unicodedata
from difflib import SequenceMatcher

def normalise(s: str) -> str:
    s = unicodedata.normalize("NFKC", s)
    s = s.replace("­", "").replace("‐", "-").replace("’", "'")
    return re.sub(r"\s+", " ", s).strip().casefold()

def char_error_rate(ref: str, hyp: str) -> float:
    ref, hyp = normalise(ref), normalise(hyp)
    matched = sum(b.size for b in SequenceMatcher(None, ref, hyp).get_matching_blocks())
    return 1.0 - matched / max(len(ref), 1)

def coverage(ref: str, hyp: str, window: int = 60) -> float:
    """Share of reference character-windows that appear anywhere in hyp."""
    ref, hyp = normalise(ref), normalise(hyp)
    windows = [ref[i:i + window] for i in range(0, len(ref) - window, window)]
    return sum(w in hyp for w in windows) / max(len(windows), 1)

Two numbers, deliberately separated, because they answer different questions and one blended score answers neither. Character error rate says how faithful the transcription is where it exists: it aligns the two strings, counts the reference characters that found a partner, and charges you for the rest. Coverage says whether whole regions went missing: it cuts the reference into fixed-width windows and asks how many of them turn up anywhere in the other output, in any order at all.

The relation between the two is worth working out once, because it is what lets you read a pair of numbers and name the failure that produced them. A contiguous block dropped from the output removes some share of the reference windows, which is a coverage shortfall of about that share. The same missing block removes about the same share of reference characters from the alignment, which is a character error rate of about that share. So an omission drives both numbers by roughly the same amount, and the small excess of the error rate over the coverage shortfall is the residual — ordinary wording difference on the material that was kept. A large error rate with coverage near its maximum is the other diagnosis entirely: nothing went missing and the transcription is simply poor.

Omission is worth measuring on its own rather than inferring it from a text score, because it is the failure the vision path has that the parser structurally cannot. Over 400 layout-heavy pages with hand-marked regions, the rate at which each region role went missing from the vision output was not uniform:

The second column is a set of different denominators rather than one sample size, and reading it as a sample size gets every rate beside it wrong. Every page has body text and every page has a running header, so those two rows are rates over the whole 400. Sidebars appear on 143 pages, so 14.7% is a rate over 143 — a page with no sidebar cannot omit one, and folding those pages into the denominator would report sidebars as far safer than they are.

The gradient runs with how peripheral a region is to a human reader, which is exactly what you would expect from a model trained to produce what a person would want. Running headers being dropped 61% of the time is mostly harmless and occasionally not, since the running header is where a statutory report puts the entity name that a footnote on page 63 depends on. Footnotes at 9.6% are more serious, because a footnote is where the qualification lives.

The parser column is all zeros by construction rather than by quality. Every glyph in the content stream comes out somewhere, so the parser's version of this failure is misplacement rather than loss, which is why the two paths need different measurements even when they are being compared. A column that is constant down every row is not a result about the parser; it is a statement that omission is not in the parser's outcome space. What it licenses is reading the vision column on its own terms — those percentages measure one path against hand-marked ground truth, with the other path contributing only the evidence of where the regions were.

Reading order needs its own measurement rather than being folded into text similarity. Score it as a rank correlation: take the sequence of text blocks each path emitted, align them to hand-ordered ground truth by matching content, and compute Kendall's tau over the resulting permutation — a rank correlation that reaches 1.0 when the two orderings agree on every pair of blocks and sits near 0 when the emitted order tells you nothing about the printed one.

Tau counts pairs rather than positions, and that is why it is the right instrument for this failure. A parser that interleaves two columns line by line leaves every block close to where it belongs when you measure how far each one moved, while getting the relative order of most pairs wrong. A score built on displacement would call that a small error. A score built on pairs calls it the large one it is. Two paths, two failures, opposite shapes, and a single blended quality number would report them as roughly equally wrong.

The parser is still right when money is no object#

Suppose the vision path costs nothing and returns instantly. On a single-column born-digital page whose embedded text layer is intact, the parser is still the correct choice, and this is worth stating as a property rather than a preference.

The argument is short and worth setting out in full, because it is the one place in this topic where a path wins on grounds no measurement can overturn. The producer chose the characters and stored them. Copying them out is a table lookup, and a lookup has no error term on character identity. So the parser's output on that page is right by construction — right in advance of any test, for the same reason that reading a value back out of a variable you have just written is right. The vision model re-derives those same characters from a picture of them, and re-derivation is a decode. A decode carries a small but nonzero error rate. Raising the resolution sharpens the image contribution and lowers that rate; it does not remove the sampling step, so it does not reach zero, at any resolution or any model size. Two operations over material where they must agree, one with an error term and one without: the one without wins, at any budget.

Where the errors land is what makes the small rate matter. The previous topic showed that OCR errors cluster in rare strings — policy numbers, dates, amounts — because those are exactly the places where no surrounding language constrains the guess. Vision models inherit that clustering for the same reason, and they inherit it in a more dangerous form, because the language prior is stronger. An OCR engine that cannot resolve a digit tends to return something visibly broken. A vision model asked for a settlement account will return twenty-two plausible characters, always, because twenty-two plausible characters is what a settlement account looks like.

The defence a team reaches for at this point is an aggregate accuracy threshold, and it is worth seeing precisely why it cannot work. Character error rate is a count of unmatched characters over the characters on the page, and an identifier is a few dozen characters on a page of a few thousand. Getting two of its digits wrong moves that ratio by a few ten-thousandths, which sits well inside the variation between any two honest transcriptions of the same page. So the threshold returns the same verdict whether the settlement account is right or wrong. A check whose output does not vary with the thing it is meant to discriminate is not a weak check — it carries no information at all, and the confidence it produces has been manufactured out of nothing. What does vary with the thing that matters is a field-level comparison on the fields that matter, identifier against identifier and amount against amount, and on a born-digital page the parser's copy is exactly the reference that comparison needs.

This is the answer to the question people ask when the budget stops being the constraint. "Which path do we use if we can afford the model everywhere" has a real answer for born-digital single-column prose, and the answer is the parser, on correctness grounds alone.

Scanned pages, and OCR on the predicting side#

None of this is a ranking. On the scanned pages in the same corpus, the parser produces nothing usable and the vision model produces a good transcription, so any claim of the form "the parser is more accurate" is a claim about one page class stated as though it were about documents. Chapters 4 and 6 are entirely about not making that mistake with a benchmark.

The copy-by-construction argument also carries a condition that is easy to state unconditionally and should not be. It holds for characters the producer embedded together with a correct mapping back to Unicode, which is the case the previous topic covered. Where a producer embedded a broken or missing mapping, the parser's lookup returns the wrong code point reliably — the same wrong one every time — and the output is confidently misspelled in a way that looks nothing like a decode error. That is still not the parser inventing a character; it is the parser copying faithfully out of a table that is wrong. The remedy differs too, since there is no draw to repeat: re-running gets you the identical wrong answer, and the fix has to happen at the encoding.

It is also not the same distinction as OCR versus a text layer, which the previous topic covered. OCR is also predicting characters from pixels, so it sits on the same side of the copy/predict line as the vision model. The difference between them is the strength and breadth of the prior: an OCR engine models glyph shapes and a narrow character-level language model, while a vision model models the whole document — that this is an invoice, that invoices have totals, that totals appear at the bottom right. That broader model is what lets it read a degraded page, and it is also what lets it write a total the page does not have.

Which operation produced a piece of text is therefore the first thing to establish about it, ahead of how good it looks. A lookup and a decode fail in ways that share no shape: one keeps every character and loses the arrangement, the other keeps the arrangement and can quietly alter or discard the content. Every measurement in this topic exists because those two failures cannot be scored on one axis, and every routing decision later exists because neither operation is the right one everywhere.

Nothing so far has said what to do when the two paths disagree and the vision output is the one that looks better. On page 17 the choice was easy, because the parser's output was visibly broken and the vision model's was visibly fluent, and you could open the page and see the sidebar it dropped. The next chapter takes a page where the vision output is not merely more readable but more complete than the parser's, with no gaps at all, and shows why that completeness is the thing that should stop the pipeline.