A 14-point corpus average argues for escalation, not migration

A stratum, in a benchmark like this, is a class of pages scored on its own — single-column prose, layout-heavy pages, table pages, chart and figure pages — with its own ground truth and its own governing metric. Stratifying is what makes a document-extraction benchmark affordable and interpretable at the same time: fifty annotated pages of each kind cost the same as two hundred random ones and tell you four things instead of one blurred thing. The sample is built that way deliberately, and the deliberate part is that its composition does not resemble any real document population.

That design choice has a consequence at the moment somebody asks for the headline. Two decisions are on the table whenever a new extraction path scores well. Migration moves every page onto the new path and retires the old one. Escalation, also called routing, keeps the cheap path as the default and sends only some page classes to the expensive one. They are different questions about different populations, and there is exactly one number that answers each. Getting the wrong number in front of the wrong question is not a subtle statistical error; it is a forty-fold difference in the monthly bill.

Where fourteen comes from#

The benchmark uses four strata of fifty pages each, which is the right design for estimating per-class quality and is the reason the sample exists at all. Each stratum is scored on the dimension that governs pages of that kind, so no page's score blends measurements of things that page does not contain.

Two columns in that table close exactly, and each closure licenses something specific about how the rest of the chapter is allowed to use it.

The Δ column is the Vision column minus the Parser column, row by row, with nothing left over in any of the four rows. Subtract and check: 98.4 from 99.2 leaves eight tenths on the parser's side, 60.5 from 78.7 leaves eighteen and two tenths, 42.0 from 61.4 leaves nineteen and four tenths, 1.6 from 20.6 leaves nineteen exactly. No row carries an adjustment, a normalisation or a weighting hidden inside it. That is what licenses treating a Δ as a plain difference later — multiplying it by a share, or adding it to another Δ — which would be illegitimate if any of these numbers had already been scaled by something.

The share column sums to one hundred percent with no residual: ninety-four, plus two and four tenths, plus two and four tenths, plus one and two tenths. The four strata therefore partition the corpus. There is no fifth class of page sitting outside them and no overlap between them, so a weighted average over these four rows is an average over every page in the corpus rather than over most of it. Without that closure the weighted figure computed in the next section would be a lower bound with an unknown gap above it.

The mean of −0.8, 18.2, 19.4 and 19.0 is 13.95, and since each stratum contributes exactly fifty pages, the mean over the 200 sample pages equals the mean over the four strata. That equality is worth pausing on, because it is the only reason an unweighted mean appears anywhere in this chapter. An average over pages is the sum of every page's value divided by the number of pages. Group the pages by stratum and every page inside a stratum carries the same value, so the sum becomes each stratum's value times its page count. Divide through by the total and each stratum's count becomes its share of the sample. When the four counts are equal, the four shares are equal, and multiplying four numbers by the same share and adding is exactly what taking their plain average does. Equal sample sizes are the condition that collapses a weighted mean into an unweighted one, and that condition holds for the sample and fails for the corpus. Fourteen points is arithmetically unimpeachable and describes a document population in which a quarter of all pages are charts.

Two of these scores are worth a note before the argument continues. Ordered coverage on the layout-heavy stratum is the share of ground-truth blocks that are both present and in the right relative position, and it is the reason the vision model scores 78.7 there rather than in the nineties: its reading order on those pages is nearly perfect, and it omits about one block in seven, which is chapter 1's dropped sidebar showing up as a number. The chart stratum uses a strict figure-level criterion — a figure counts only if every assertion the ground truth records is recovered and no false assertion is emitted — which is why 20.6 sits so far below the 72% assertion recall reported earlier. Both scores were chosen to be conservative about the vision path, and it still comes out fourteen ahead on the sample.

Reweighting is four lines and changes the decision#

The corpus is 94% ordinary prose. Applying its actual composition to the same per-stratum results is the only step between the two numbers.

strata = {                    # (delta, corpus share)
    "prose":        (-0.8, 0.940),
    "layout_heavy": (18.2, 0.024),
    "tables":       (19.4, 0.024),
    "charts":       (19.0, 0.012),
}

sample_mean = sum(d for d, _ in strata.values()) / len(strata)
corpus_mean = sum(d * w for d, w in strata.values())

print(round(sample_mean, 2))   # 13.95
print(round(corpus_mean, 2))   # 0.38

The two expressions differ in one place. sample_mean divides by the number of strata, which is the same as multiplying each delta by one quarter, because the sample really does hold one quarter of its pages in each class. corpus_mean multiplies each delta by that class's share of the corpus instead. Nothing else changes — same deltas, same four rows, same arithmetic operation. The entire gap between fourteen points and four tenths of a point is the substitution of one set of weights for another.

Work the weighted sum by hand and the shape of the result is visible before the total is. The prose term is a small loss against a very large weight, and it comes to about three quarters of a point on the negative side. The three minority terms are large gains against very small weights: layout-heavy and tables each contribute a bit over four tenths of a point, charts a bit over two tenths, and together they come to a little over one and one tenth points on the positive side. Add the negative term to the positive ones and four tenths of a point survives.

Migrating the whole pipeline moves average page quality across the corpus by about four tenths of a point, because 94% of the pages get slightly worse and 6% get much better, and 94% is a heavy weight. That is what the fourteen was hiding — not an error in the benchmark, but a mismatch between the population the number describes and the population the decision applies to.

The three minority terms are also, by themselves, the escalation option. A routed pipeline sends only those three classes to the vision path and leaves prose where it is, so its corpus-weighted gain is that positive sum on its own: a little over one and one tenth points, which the cost table below records as +1.13. Migration's four tenths is that same +1.13 with the prose term added to it. The prose stratum's contribution is not one influence among several on the difference between the two options — it is the difference, exactly, with nothing else in it. Every other term appears identically in both options and cancels in the subtraction. That is why the argument for routing does not require any claim about how good the vision path is on tables or charts: those numbers are on both sides of the comparison and drop out.

The uncertainty makes it starker rather than softer. At fifty pages per stratum, the interval around each score is roughly ±11 points for scores in the middle of the range and much tighter near the ceiling, so the prose delta of −0.8 has an interval of about ±1.4 and the corpus-weighted total lands somewhere between −1.0 and +1.8. That band is recoverable a second way, and the second route shows where the uncertainty lives. Take the prose interval, multiply it by the prose weight, and almost the whole band is already there; fold in the three minority intervals, each shrunk by a weight of a few percent, and the band widens by almost nothing before it reaches the stated ±1.4 either side of four tenths. The corpus average inherits its uncertainty from the stratum that dominates its weight, which is the same stratum whose signal is smallest. Migration's effect on average corpus quality is not distinguishable from zero at this sample size, while the per-stratum effects on layout-heavy, table and chart pages are all far outside their intervals. The benchmark speaks clearly and precisely about three page classes and says nothing at all about the corpus average, which is the number the slide chose to lead with.

That last sentence is the stumble worth naming, because the corpus average feels like the conservative choice and it is the opposite of one. Blending looks like a defence against cherry-picking: one number, computed over everything, immune to a advocate picking the flattering stratum. The trouble is what the number does when the thing it is supposed to discriminate changes. Let the three minority deltas swing all the way from zero — the vision path being no better than the parser anywhere — up to their measured values, and the corpus average travels a little over one and one tenth points in total. Its own confidence interval is wider than that whole range. A defence that returns the same verdict whether the new path is worthless on tables or nineteen points ahead on them is not discriminating between those cases at all, and an improvement that moves it carries no information about where the improvement came from.

Which number answers which question#

Read that table by what is missing from it rather than by what it contains. The first three rows carry their numbers in the left-hand column; the last two carry none. That gap is not an omission in the writing. Rows four and five require a quantity the benchmark never measured — how many pages of each class arrive per month, and what a page costs on each path — and neither is a quality measurement, so no amount of further annotation produces them. The two rows that answer the operationally useful questions are the two the benchmark alone cannot fill in, which is why the sections below have to bring volume and cost in from elsewhere.

Only the third and fourth rows produce anything a pipeline can act on, because a pipeline processes individual pages and has to decide, per page, which path runs. A corpus-wide average is a number about a hypothetical wholesale replacement, and wholesale replacement is one of the options rather than the frame the evidence should be reported in.

Any benchmark report that leads with a single figure invites this, and the guard is procedural: agree before the benchmark runs that the deliverable is a per-page-type table, and that a single corpus-wide quality number will not be accepted as the answer regardless of which direction it points. Refusing the blended number when it favours your position is what makes refusing it later credible.

Costing the three options#

Measurement puts the vision path at roughly 40× the parser's per-page cost and about 12× its latency. Those figures come from the next chapter and are themselves optimistic; use them for now. Against a corpus arriving at 400,000 pages a month, they produce this:

Every cell in that table is generated by three inputs — the parser's monthly bill, the cost multiple and the latency multiple — and reproducing them by hand is what makes the comparison arguable rather than quoted. The escalated page count is the three minority strata's shares added together, six percent of four hundred thousand, which is twenty-four thousand. The all-vision bill is the parser's bill times forty. The escalation bill is the only cell with a modelling assumption in it: it charges the parser for the ninety-four percent of pages that stay on it and the vision path for the six percent that move, and those two products add to the eight hundred and eighty-five dollars shown. The vision latency is the parser's times twelve.

The assumption inside that eight hundred and eighty-five is a design decision rather than a requirement, and it is worth separating from the measurements around it. It assumes each page is billed on exactly one path. A router that decides where to send a page by looking at what the parser produced has already paid for the parser on every page, escalated or not, and its bill is the full parser run plus the vision run on six percent — about sixteen dollars a month more. That difference is small enough to ignore here and it is not zero, and which of the two models applies depends on what the routing signal is, which is the subject of a later chapter. The measured multiples do not change either way.

The cost-per-point row is the one most likely to be misread, because it is not the monthly bill divided by the quality delta. It is the increment over the all-parser baseline divided by the delta, which is the right construction because the baseline already delivers its quality for its own price and the question is what the extra quality costs. Take the all-vision bill, subtract the parser bill, divide by four tenths of a point, and the result is about twenty-seven thousand two hundred dollars a point. Do the same for escalation — its bill less the parser bill, divided by one and thirteen hundredths of a point — and the result is about five hundred and fifty. Both figures reproduce to the precision they are quoted at.

The median latency column hides a second construction worth stating, since a routed pipeline plainly does run some slow pages. The escalation column reads 0.35 s because the median page is a prose page: ninety-four percent of pages stay on the parser, so the page in the middle of the latency distribution is a parser page and the twenty-four thousand slow ones sit entirely in the upper tail. A mean would move; a median at this composition cannot. Quoting the median here is honest about what a typical page experiences and silent about what the escalated ones do, and a service level built on it needs a tail percentile beside it.

The quality column is the part that surprises people, because escalation scores higher than migration on the same benchmark. It has to: migration takes the vision path's three wins and also takes its loss on 94% of pages, whereas escalation takes the three wins and leaves the prose pages where the parser is better by construction, for the reason chapter 1 gave. Routing is not a compromise between quality and cost here. It is better on both, and the only thing it costs is the complexity of running two paths and a rule between them.

At $27,200 per corpus quality point against $550, the gap is not close enough for any plausible correction to the cost figures to change the ordering. Even if the 40× multiple turned out to be 8×, migration would sit near $4,900 per point.

The same table, weighted by what pages are asked about#

Page counts are one weighting and query volume is another, and they disagree here in a way that changes what to escalate first. Joining a month of question logs to the pages their answers cited gives a per-stratum query share, which is a measure of what users actually consult rather than what the corpus happens to contain.

The cited-share column sums to one hundred percent in its own right, which is what makes it a legitimate substitute weighting rather than a subset: every citation lands on a page of one of these four kinds. The final column is each Δ times its cited share and the four products add to the total shown with nothing left over, so this is the same weighted-mean machinery as before with one column swapped.

The two share columns are the pair worth reading against each other, because the finding is in the gap between them rather than in either one. Prose falls from ninety-four percent of pages to sixty-one percent of citations. Tables rise from two and four tenths percent of pages to twenty-four percent of citations, a tenfold move, and layout-heavy and chart pages roughly triple and quintuple respectively. The two columns are locked together under the assumption that users consult pages in proportion to how many there are, and they come apart hard. Weighted by citation, the vision path is worth about seven points rather than four tenths, because table pages are 2.4% of the corpus and 24% of what gets cited. That does not change the routing conclusion — the escalated 6% is still the same 6% — but it does change the priority, since the table stratum alone carries two thirds of the available benefit and is the stratum where a threshold miss costs most.

It also gives a straightforward answer to the question of which page class to work on next. Layout-heavy and chart pages contribute a similar amount, and chart pages are the ones where the vision path's own defect rate is highest, so effort spent on the chart gates from the previous chapter buys more than effort spent widening the layout threshold.

None of these three weightings is the true one. They answer different questions, and the mistake worth avoiding is reporting whichever weighting best supports a conclusion already reached.

Where the composition inverts the arithmetic#

The arithmetic above is driven entirely by the 94%, so it inverts when the composition does. A corpus of financial statements and pitch decks — say 55% table pages, 20% charts, 15% layout-heavy, 10% prose — produces a corpus-weighted delta of about 15 points, an escalation share of 90%, and a routed pipeline that is running the vision model on nearly everything while paying to maintain a parser, a router, a second set of failure modes and a second set of monitoring. Somewhere above roughly half the corpus escalating, the operational cost of two paths starts to outweigh the shrinking cost saving, and one path becomes the better engineering decision even though it is still slightly more expensive.

Corpus composition also moves. The previous topic's monitoring — the share of pages per source with no usable text layer — exists partly to catch a supplier switching to scans, and the same signal tells you when your escalation share is drifting upward for reasons nobody decided. A routing threshold set against a 6% corpus and left alone through a migration to a scan-heavy source is a threshold that will quietly route 40% of pages to the expensive path.

There is one more case where the average is the right number, and it is worth naming so the rule does not get over-applied: if you are choosing between two vision models rather than between a vision model and a parser, and both will run on the same page population, then a population-weighted average over that population is exactly the comparison you want. The objection in this chapter is to averaging over a population you will not process, in service of a decision that is per-page.

Finally, the corpus-weighted delta is a quality statement and carries no information about which pages matter. A corpus that is 94% prose might still derive most of its business value from the 2.4% of pages carrying financial tables, and a value-weighted view would put the table stratum far ahead of its 2.4%. That weighting is a product judgement rather than a measurement, and it belongs in the routing policy rather than in the benchmark.

None of this is a mistake in the benchmark. Every input to the fourteen was measured carefully, and a team that produced it did the hard part. The failure is in reporting, and specifically in reporting one number when the underlying measurements were deliberately taken per class. Nor is it evidence that the vision path is a poor investment. It is 19 points better on chart pages, which is the difference between answering questions about figures and not answering them at all, and 4,800 chart pages a month is a lot of unanswerable questions. The reweighting says where that investment goes, not whether to make it.

The general form is worth carrying to the next benchmark rather than only to this one. Any time a set of per-class measurements is collapsed into one figure, ask which weights were used and which population those weights describe, and check whether that population is the one the decision will be applied to. Where the two differ, the per-class numbers are the measurement and the blended figure is a claim about a hypothetical, and the per-class numbers are also the only form a per-page routing rule can consume.

Both cost figures used here — 40× on money, 12× on latency — arrived as measured multiples, and the way most teams obtain them is by multiplying a provider's per-image price by a page count. That method understates the real figure by roughly three times, and it understates the wall-clock by rather more, for reasons that have nothing to do with each other.