Parser choice is a per-source decision, not one standard

Two candidates for parsing a body of documents are rarely better and worse in the same direction. One is stronger on scans and weaker on throughput; one is free at the margin and poor on tables. Which of those trade-offs matters is decided entirely by what your documents look like, and a corpus — the whole set of documents an ingestion system is expected to answer questions about — is almost never uniform enough for one answer. It arrives from several sources, and one source is one production process: a supplier who exports from a word processor produces pages with text layers, a supplier who scans produces pages without, and a deck exporter produces text inside raster images.

Comparing two parsers therefore means choosing which documents to test on, what to score, and what to do with the result — and each of those has a cheap wrong answer that gets taken. Testing on whatever the vendor supplies, scoring with one number, and crowning a single winner all fail for the same underlying reason: each one averages away the variation the decision depends on. That reason is visible in the measurements themselves, which is why the numbers in this chapter are worth reading as differences between columns rather than as levels.

Evidence that cannot be used, and why#

A vendor deck says 98% table accuracy, $0.012 per page, and shows a scanned 1970s bank statement reconstructed cell-perfect. The incumbent is a pipeline built over the last seven chapters, running on a machine already in the rack. There are 40,000 documents from six sources: two send born-digital reports, three send scans, one sends slide exports. Somebody has to decide by the end of the month.

Nothing in the deck bears on the decision. The 98% was measured on the vendor's sample, under the vendor's definition of accuracy, against documents chosen by the vendor, and there is no way to map it onto a corpus it has never seen. That is the piece of evidence to refuse outright, and refusing it is not scepticism about the vendor. Published table accuracy in this field is usually cell-content accuracy on a scientific-paper benchmark like PubTabNet, whose tables are clean, ruled, single-header and typeset by LaTeX. Your rate tables have merged two-level headers, footnote markers, no ruling lines and a total row, and there is no coefficient that converts one number into the other.

The temptation is to treat that 98% as a conservative floor — if a parser is that good on hard scanned material, surely it is at least that good on yours. What makes the reasoning fail is that the benchmark's tables hold constant every property that separates one parser from another on your documents. Ruling lines are present, so a parser that finds cell boundaries only by ruling lines scores the same as one that infers them from alignment. Headers are single-level, so a parser that cannot resolve a merged span scores the same as one that can. The metric returns a high number for both, and it would return a high number for both whatever they did with a two-level header, because no two-level header is in the test set. A check whose verdict stays put while the property it is trusted to discriminate ranges from best to worst is not a weak check; it is not a check at all on that property.

The related refusal is an evaluation run by the vendor on documents you supply. Handing over 200 files and receiving back a scored report gives the vendor's engineers a week to tune preprocessing to those files, and it gives you a number you cannot reproduce. A result you cannot reproduce is a result about those 200 files rather than about the parser, and the next document is not one of them.

Stratify on the properties you have already measured#

The metrics table from chapter 7 is what makes the sample design cheap, because it already describes every document in the corpus by page count, OCR rate, table density and producer. Drawing four strata from it is a query rather than a survey.

A stratum is a group chosen so that the documents inside it fail in the same way. The selection rules are the point: each one is a threshold on a field the instrumentation already stores, so membership is decided by a query rather than by somebody's judgement about which files look representative. That is what makes the design reproducible — run the same query in six months and you get the strata as they are then, not as somebody remembered them.

The four strata are a design decision and not a requirement. The requirement is that the sample contain the failure modes the corpus contains; four is the number of distinct production processes this particular corpus has, and it comes straight from the source census above — two born-digital senders, three scanners, one deck exporter. A corpus whose sources all export from one word processor needs one stratum, and a corpus with a handwritten-forms supplier needs another.

Both numeric columns close exactly on the totals stated elsewhere. Twenty-five documents in each of four strata is the hundred documents put through both parsers; thirty pages in each of four strata is the hundred and twenty pages of ground truth. Neither column has a residual, and that absence is the check worth making: a stratified design where the parts do not sum to the whole has either double-counted a document that satisfies two selection rules or dropped one that satisfies none. Note that the selection rules could overlap — a scanned table-heavy document passes two of them — so the clean sum is a fact about how the sample was drawn rather than something the rules guarantee.

A hundred documents go through both parsers end to end, which is automated and costs the price of 2,000 vendor pages, about $24. That figure is the vendor's own per-page price multiplied by the pages in the sample, and it closes against the price row in the results table below to the cent, which is what licenses treating the trial as having no hidden cost on the vendor side. It also pins down a number the table does not print: two thousand pages across a hundred documents is an average of twenty pages a document, so the sample is drawn from a corpus of medium-length documents rather than of one-page forms or of book-length filings.

Ground truth is the expensive part and is deliberately confined to 120 pages: for each sampled document, the first page, one page containing a table where the document has one, and one page chosen at random. Keying reference output — the text in correct reading order, and each table as a CSV with resolved headers — took about eight minutes for a prose page and about twenty for a table page, so 120 pages is roughly thirty hours, four days for one person or a single day if four people split it. That number is the whole budget for the decision, and it is worth stating to whoever is asking, because the alternative on offer is a nine-times cost increase in perpetuity.

Those three time figures are worth taking apart, because they tell you what drives the budget. Thirty hours over a hundred and twenty pages is an average around a quarter of an hour a page, which sits much nearer the table figure than the prose one — the estimate only lands where it does if well over half the keyed pages carry tables. So the cost of this exercise scales with how table-heavy the corpus is, and a team whose documents are mostly prose should expect a budget closer to the lower bound. The two ways of spending thirty hours are consistent with each other under one unstated assumption: a working day of seven and a half hours makes both "four days for one person" and "a single day if four people split it" true at once, and no other day length makes both true.

Ground truth has to be keyed from the rendered page rather than adjusted from either parser's output. Starting from parser A's text and correcting it produces a reference that shares A's segmentation decisions, and A then scores several points higher for reasons that have nothing to do with quality.

Three scores, never one#

Collapsing extraction into a single quality number destroys exactly the structure the comparison exists to reveal, since a parser can be excellent at characters and useless at tables. Three separate measurements, each reported per stratum:

Extraction completeness is token recall against the reference — the share of the reference's words that appear anywhere in the parser's output — measured after the normalisation from chapter 6, so that hyphenation and ligature differences do not count as misses. It answers whether the text arrived at all.

Reading order is a rank correlation between the reference sentence sequence and the order those sentences appear in the parser output. Kendall's tau-b over matched sentence positions gives a number that is 1.0 for perfect order, around 0.0 for a shuffle, and — usefully — around 0.3 to 0.5 for the two-column interleave from chapter 1, because an interleave preserves a lot of local order while destroying the global sequence.

from scipy.stats import kendalltau
from rapidfuzz import process, fuzz

def reading_order_tau(reference_sentences, parsed_sentences, cutoff=85):
    positions = []
    for ref_i, ref in enumerate(reference_sentences):
        hit = process.extractOne(
            ref, parsed_sentences, scorer=fuzz.token_sort_ratio, score_cutoff=cutoff
        )
        if hit:
            positions.append((ref_i, hit[2]))     # (reference index, parsed index)
    if len(positions) < 3:
        return 0.0, len(positions) / max(len(reference_sentences), 1)
    ref_idx, out_idx = zip(*positions)
    tau, _ = kendalltau(ref_idx, out_idx)
    return tau, len(positions) / len(reference_sentences)

That function returns two values, and the second is the one people drop. It is the share of reference sentences that were matched at all, and without it the tau is unreadable: a parser that emits three sentences in the right order and loses the rest scores a perfect correlation on the three it kept. The score_cutoff guards against pairing a reference sentence with an unrelated one, and the minimum of three matched positions guards against computing a correlation from a sample too small to have a direction. Both guards make the function return early rather than return a flattering number, which is the right failure for a measurement that feeds a purchasing decision.

Table fidelity is measured as the binding accuracy defined in chapter 4 rather than as a published table metric. For each reference cell, check whether the parser's output contains a record tying that value to the correct row label and the correct fully-qualified column label. TEDS, the tree-edit-distance-based similarity reported in table-recognition research, and similar tree-edit metrics are reasonable research measures and reward structural resemblance that does not affect retrieval; binding accuracy measures the property the pipeline actually depends on.

def binding_accuracy(reference_grid, emitted_records) -> float:
    index = {r["row_label"]: r["text"] for r in emitted_records}
    hits = total = 0
    for row_label, columns in reference_grid.items():
        for column_label, value in columns.items():
            total += 1
            text = index.get(row_label, "")
            hits += f"{column_label} = {value}" in text
    return hits / total if total else 0.0

The denominator there is every cell of the reference grid, counted whether or not the parser emitted anything for that row. That is the choice that makes the score mean what its name says: a row the parser dropped entirely contributes zeros to hits and its full complement to total, so losing a row costs exactly as much as mislabelling every cell in it. Scoring only over emitted records would have made a parser look better the more it dropped, which is the standard way a retrieval-side metric ends up rewarding silence.

Latency and cost per page go alongside, measured on the hardware you will actually run, at p50 and p95 — the times half the pages and ninety-five percent of the pages come in under. The p95 is where the pathological files from chapter 7 live, and a parser whose p95 is forty times its p50 has a throughput problem that an average will not show.

What the numbers said#

The vendor is clearly better on scans and on tables, four times slower, and roughly nine times the cost — the incumbent is not free, since amortised hardware and power come to about $0.0013 a page, and comparing a paid service against zero is how these decisions get made badly. That $0.0013 is stated as hardware and power, so it is a compute cost; where a source's pages go through a paid recognition service, that service's per-page fee sits on top of it.

Read the table as gaps rather than as levels and the case for three scores stops being an assertion. Take the difference between the two columns, row by row. Completeness separates the parsers by three thousandths on born-digital prose, eight thousandths on table-heavy, and its widest gap anywhere is on scans, at a little over six hundredths. Binding accuracy separates them by five hundredths, twenty-nine hundredths and forty-five hundredths on those same three strata. On the scanned stratum the binding gap is close to seven times the completeness gap measured on the identical documents.

That divergence is what a single collapsed score would destroy. Completeness sits near its ceiling for both parsers on three strata out of four, so it barely moves; average it together with the other two and it drags any composite toward "these are much the same". The composite would return roughly that verdict whether the binding gap were the five hundredths seen on prose or the forty-five hundredths seen on scans — which is the same defect as the benchmark number refused earlier, arriving by a different route. A summary statistic that stays put while its largest component ranges across nine-tenths of its own scale is not summarising; it is discarding.

The row that decides the shape of the answer is the first block. On born-digital prose the two parsers are within 0.003 on completeness and 0.01 on reading order, differences well inside the noise of a 30-page sample. Paying nine times the price on those two sources buys output that is not measurably different from what you already have. The slide stratum says something else again: both parsers are poor, with a fifth of the reference text missing under each, because the missing text is inside raster images and neither parser is reading them. That stratum needs the OCR-the-picture-shapes step from chapter 5 rather than a purchasing decision.

The missing cell in that last block is the finding rather than an omission. Binding accuracy is n/a for both parsers on slide export because slides carry no tables to bind, so the score is undefined rather than bad. What that removes is the one measurement that separates the parsers most sharply everywhere else, which leaves a stratum where the two candidates differ by less than a tenth on both remaining scores and both fall far short of the other strata. A stratum on which the leading candidates are close and both poor is a stratum the purchasing question cannot fix, and identifying those is part of what the exercise is for.

Routing, and what it costs to run two paths#

default:
  parser: incumbent
  page_overrides:
    - when: page.tables_detected >= 1
      parser: vendor
sources:
  meridian:      { parser: vendor }          # scanned since 2024-03
  ashford:       { parser: vendor }
  kelso-claims:  { parser: vendor }
  northwind-fin: { parser: incumbent }       # born-digital, no measured gap
  halloway:      { parser: incumbent }
  decks:
    parser: incumbent
    pipeline: [shapes, notes, ocr_picture_shapes]
review:
  next: 2025-03-01
  triggers: [parser_version_change, source_ocr_rate_step]

That file reproduces the source census from the opening exactly: six named sources, three routed to the vendor for scans, two kept on the incumbent as born-digital, one deck exporter with its own pipeline. No source is missing and none is invented, which is the reason the page shares closed to the whole. The default block is what makes the config exhaustive — a source added tomorrow and not named here still has a parser — and that default is a design decision with a visible consequence: an unmeasured new source is silently assigned to the cheaper parser rather than held out for measurement.

The comment on northwind-fin reads "no measured gap" while the table shows the vendor five hundredths ahead on binding accuracy for born-digital prose, and both statements are true because they are about different pages. The born-digital prose stratum is selected with fewer than a tenth of its pages carrying tables, so its binding score is computed over very few cells and is the noisiest number in the block. More to the point, the page_overrides rule sends any page with a table to the vendor whatever its source, so the pages that actually stay on the incumbent under this config are the table-free ones — and on those the binding score has nothing to measure. The source-level comment describes the traffic the rule leaves behind, not the stratum as sampled.

The saving against standardising on the vendor is about $4,600 a year, which is real and is not the main argument. The stronger one is throughput: at 2,100ms per page, a full reingest of 800,000 pages is 467 core-hours against 116 for the incumbent, and reingest happens whenever the chunking strategy changes, the embedding model changes, or a parser is upgraded, which on an active project is several times a year. Routing keeps the two born-digital sources — the ones that reingest most often, because they are the ones people query most — on the fast path. Those core-hour figures are the page volume times the two p50 latencies, and their ratio recovers the four-times figure that the latency row gives directly, so the slowdown is linear in volume with no fixed overhead hiding inside it.

Routing is not free, and the costs are worth naming rather than discovering. Two parsers mean two output formats to normalise, two failure modes, two version numbers in the metrics table, and a routing configuration that is a piece of production logic somebody has to own. It also means a document's parsed representation depends on when it was ingested and under which rule, so the parser_version field from chapter 7 stops being an audit convenience and becomes necessary for explaining why two documents from the same source look different. The threshold for accepting that complexity is a measured gap between the parsers on a stratum you actually have, which is what the table above provides and what a vendor demo never can.

The other honest limitation is that this comparison expires. Layout parsing is moving quickly: the open-source side has changed substantially with tools like Docling, whose approach to reconstructing reading order and table structure is described in Docling: An Efficient Open-Source Toolkit for AI-driven Document Conversion, and the hosted side is folding vision-language models into the pipeline, as in Google's Document AI layout parser. A measurement taken today is a statement about two specific versions on your corpus as it exists now, so the review date in the routing config and the triggers beside it are part of the deliverable. The Meridian step change from the last chapter is exactly the kind of event that should force a re-run: a source that becomes scan-heavy has moved into a stratum where the parsers were measured to differ by a large margin.

The triggers list is where the two halves of this join up. source_ocr_rate_step is the per-source alert from chapter 7, named here as a condition that invalidates the routing rule. That makes the instrumentation part of this decision's maintenance rather than a separate concern: the config is correct only while each source stays in the stratum it was measured in, and the alert is the thing that notices when one does not.

The through-line from the takeaway holds at every step of that chain. A parser's error modes follow from properties of the input; those properties arrive clustered by source, because a source is a production process; a score averaged over a mixed corpus is therefore a weighted blend of gaps that no single document exhibits; and a decision taken on that blend is right for a document that does not exist. Measuring per stratum and routing per source is what it takes to make the answer about documents you actually have — and to know, on a stated date, when it has stopped being true.

The output of all this — what the parser hands to everything downstream — is a stream of text with reading order reconstructed as well as you can afford, tables carried as row records bound to their labels, boilerplate stripped, superseded revisions excluded, and per-page provenance — a record travelling with each span of which document and page it was taken from — attached to every span. That is the material the chunker will now cut up, and its properties bound what chunking can achieve: a chunker cannot respect a section boundary the parser did not mark, cannot avoid splitting a table it was never told is a table, and cannot attach a heading to a passage if the heading arrived as an ordinary paragraph three pages away.

Which is why the decisions that look like chunking parameters — window size, overlap, whether to split on headings — are mostly decisions about what the parser emitted, and why chunking is worth approaching with the parsed artefact open beside it rather than as a set of numbers to tune.