Route on a measured signal and record which path ran

Routing, here, means deciding per page which of two extraction paths runs: a parser at $0.00066 a page, or a vision model at $0.0265. The previous chapter established why the choice is worth making at all — sending a small minority of pages down the expensive path cuts both the bill and the wall clock by roughly the same large fraction, and it is the only available change that moves both. What it did not establish is how a page gets sorted, and that is the whole of the engineering, because the sorting has to happen in a few tens of milliseconds using only what can be computed cheaply from a page nobody has extracted yet.

The decision cannot use the thing you actually want to know, which is how much better the vision model would be on this page. That is only knowable by running both, and it is worth working out why, because the reason sets everything that follows. The quantity is a difference between two outputs, so obtaining it means producing both outputs, so it means running both paths. Running both paths on every page costs their sum, which is strictly more than running the expensive one alone — and running the expensive one alone is migrating wholesale, the option the previous chapter priced and rejected. A rule that needs the answer in order to decide therefore costs more than not deciding at all. What is available instead is a set of proxies that correlate with the answer, and the engineering is in choosing proxies that are cheap, and in setting their thresholds from the benchmark rather than from a meeting.

Four signals, all computable before the expensive call#

Two pages from one filing make the target concrete. Page 4 is three paragraphs of narrative under a heading, which any parser handles for a fraction of a cent. Page 5 is a landscape financial statement, rotated 90 degrees, eleven columns wide, with a two-row header — the page class where every earlier chapter in this topic showed the parser losing. A usable signal is one that separates those two before either has been extracted.

Detected table and figure area, as a fraction of page area. A layout detector — the same one already in the parser pipeline from the previous topic — emits region boxes with types. Summing the area of table and figure regions and dividing by page area takes about 35 ms of CPU on top of work already being done, which across 400,000 pages a month costs roughly $16. This is the signal that fires on page 5 and not on page 4.

Dividing by page area rather than using the raw region area is what makes the signal comparable across pages, and the step is easy to skip. Region areas are in pixels, and pixels depend on render resolution and on paper size, so a raw figure of a few hundred thousand means one thing on a portrait A4 page at 300 DPI and something else on a landscape spread. Dividing by the page's own area cancels the units, leaving a pure fraction between nothing and one that means the same thing on every page. A threshold set on a fraction survives a change of render settings; a threshold set on pixel counts silently reroutes the corpus the day somebody raises the DPI.

Text yield, characters emitted per thousand square pixels of inked page area. A born-digital prose page runs 1.8 to 4.0; a page that is one large embedded image runs near 0.0; a page whose text is mostly inside a diagram runs 0.1 to 0.4. It fires on scans, on slide exports whose content lives inside pictures, and on the engineering drawings that a text parser reports as almost empty. Inked area rather than total area matters, because a mostly-blank page with one paragraph should not look like a failure.

That last clause carries a step worth spelling out, since it is the difference between a working signal and one that fires constantly. Text yield is a ratio, and a ratio can be pushed toward zero from either end — by shrinking the numerator or by inflating the denominator. Marks on a page are what the numerator counts, so the denominator has to count marks too if the ratio is to mean "how much of what is drawn here came back as characters". Total page area does not count marks; it counts paper. A title page with one line on it has almost no characters and an entire sheet of denominator, so it scores like a scan while being perfectly parseable. Restricting the denominator to inked area makes both halves of the ratio measure the same thing, and the signal then reads as intended: marks present, characters absent.

Parser confidence. Where OCR ran, mean per-character confidence is available directly. Where it did not, a useful proxy is the parser's own reading-order confidence — the share of adjacent block pairs whose ordering the sorter resolved unambiguously — which drops on multi-column pages precisely because that is where the sort is hard.

Document class, from the ingestion route rather than from content. Contracts, financial statements and regulatory filings can be escalated on class alone because a wrong number in them costs more than the model call. This is the one signal that is a policy rather than a measurement, and it should be small and explicitly listed.

That distinction is worth holding onto through the rest of the chapter. Three of these signals are attempts to estimate something true about a page, and they can be right or wrong. The fourth is a decision about what an error is worth, and it cannot be right or wrong — it can only be agreed to or not. Sweeping a threshold against a benchmark is the correct treatment of the first kind and a category error on the second, which is why the class list is written out by hand and kept short rather than tuned.

from dataclasses import dataclass

@dataclass
class PageSignals:
    table_figure_area: float      # fraction of page area
    text_yield: float             # chars per 1,000 px^2 of inked area
    parser_confidence: float      # 0..1
    doc_class: str

THRESHOLDS = {"version": "r7", "area": 0.12, "yield": 0.40, "confidence": 0.86}
HIGH_VALUE = {"financial_statement", "contract", "regulatory_filing"}

def route(s: PageSignals) -> tuple[str, list[str]]:
    reasons = []
    if s.table_figure_area >= THRESHOLDS["area"]:
        reasons.append("table_figure_area")
    if s.text_yield < THRESHOLDS["yield"]:
        reasons.append("low_text_yield")
    if s.parser_confidence < THRESHOLDS["confidence"]:
        reasons.append("low_parser_confidence")
    if s.doc_class in HIGH_VALUE:
        reasons.append("high_value_class")
    return ("vision" if reasons else "parser"), reasons

Two details of that function are decisions rather than consequences. The comparisons point in opposite directions — area is tested with >= and the other two with < — because area is a signal that rises when escalation is warranted while yield and confidence fall, and getting one of those backwards produces a router that escalates exactly the pages it should not. And the four tests are combined by accumulating into reasons and escalating if the list is non-empty, which is an OR: any single signal is enough. An AND would require every signal to agree, and since these signals fire on different page classes — area on dense tables, yield on scans, confidence on multi-column text — an AND would fire on almost nothing. Returning reasons alongside the verdict, rather than just the verdict, is what makes the record later in this chapter possible at all.

Thresholds come out of the benchmark#

Every number in THRESHOLDS was chosen by sweeping it against the 200-page benchmark, where the label for each page is whether the vision path beat the parser by more than 8 points on that page's governing score — 8 being the width that survives the sample's confidence interval.

That label definition is a design decision sitting on top of a requirement, and they are worth separating. The requirement is to escalate pages where the vision path is materially better. The decision is to fix "materially" at the width the sample can actually resolve, so that a page counted as a win is one the benchmark could distinguish from noise. Set the width lower and the labels include differences the sample cannot support, and every threshold swept against them inherits that noise.

The cost column is not six separate measurements. Every cell in it is reproduced, to within a rounding of the per-page constants, by one two-line model: take the escalation share of four hundred thousand pages a month and charge them at the vision path's per-page cost, take the remainder and charge them at the parser's, and add. Check it on the first row — a shade over three per cent of the monthly volume at two and a half cents or so, plus the other ninety-seven per cent at two-thirds of a tenth of a cent — and the total lands on the cell. Check it on the oracle row and it lands there too.

What that licenses is the point of building the table rather than admiring it. If every cell follows from the share, then any threshold anyone proposes can be priced from its escalation share alone, with no extra run and no new measurement, and an argument about where to set a knob becomes arithmetic. It also means a cell that failed to reproduce would be evidence of a cost the model does not contain — a per-call overhead, a different retry rate on escalated pages — which is a more useful thing to learn from a table than a confirmation.

The oracle row is the one that makes the rest legible. A perfect router would escalate 6.0% of pages for $884 a month; the shipped rule escalates 8.1% for $1,101, which is 25% more money to buy 94% of the available gain. Moving from 0.88 recall to 0.94 costs $113 a month, and moving from 0.94 to 0.97 by escalating whole document classes costs $331 — still cheap in absolute terms, and the point of the table is that the trade is visible and priced rather than argued about.

Those two costs of moving are exact differences between cells in the last column, which is worth noticing because it says the marginal price of recall is computable from the same table and needs no separate model. It also exposes what the table is really showing, which is two columns coming apart. Escalation share and cost rise together in a straight line, since cost is a fixed markup on share. Recall does not: it climbs steeply while share is small and then flattens, so the same increment of share buys progressively less. That divergence is the entire economics of routing. If the two columns moved together there would be no threshold to choose, only a budget to spend.

The row that would settle every argument is the one the table cannot contain. No buildable rule reaches recall 1.00 at 6.0% escalation; the oracle does, and it does so by consulting the answer, which is the thing established above as unaffordable. Its absence from the buildable rows is the finding rather than a gap in the experiment.

The sweep also disciplines the thresholds against intuition. Nobody in a meeting proposes 0.12 for table area; the round numbers people reach for are 0.25 or 0.50, and both of them sit at the bad end of that table. Nor is the knee where anyone would guess: recall climbs steeply from 0.25 down to 0.12 and then flattens, so pushing the area threshold to 0.08 would take escalation to 9.6% and recall to 0.95, which is not worth it.

Re-sweeping is a scheduled job, not a one-off. Corpus composition moves, and a threshold set against a 6% escalation population will drift as sources change — the per-source OCR-rate monitoring from the previous topic is exactly the alarm for that. Quarterly re-sweeps against a refreshed sample, with the threshold version bumped, keeps the rule honest.

The rule, its fallbacks, and what happens when a path fails#

Two things in that graph are more important than the threshold arithmetic.

The parser-output sanity check is the fallback for the router's false negatives. A page the router sent to the parser, whose parser output then comes back with a zero-cell table, a text yield near nothing, or cells with no resolved column labels, escalates after the fact. It costs one extra path on a small number of pages and it converts the router's 6% miss rate from silent damage into a second chance.

The reason that check can exist at all is that it runs on evidence the router never had. Before extraction there is only a picture of a page; afterwards there is a structured object that either has bound column labels or does not. The router's proxies are guesses about what extraction will produce, and the sanity check is an observation of what it did produce — which is why a signal too weak to route on can still be strong enough to catch a mistake, and why placing the same check before the call would be worthless.

The hold state is the fallback for the vision path's own failures, and it is deliberately not "fall back to the parser". Chapter 2 showed why: when the vision output fails its gates because the page was unreadable, the parser's output on that page is unreadable too, and quietly substituting one for the other produces an indexed page that nobody knows is broken. A held page is a visible gap; a substituted page is an invisible error. Chapter 9 takes up what holding costs.

Recording which path ran#

Every indexed page carries a provenance record, written by the router rather than reconstructed later.

{
  "page_id": "filing_88213_p05",
  "path": "vision",
  "router": {
    "rule_version": "r7",
    "reasons": ["table_figure_area", "high_value_class"],
    "signals": {"table_figure_area": 0.61, "text_yield": 0.31,
                "parser_confidence": 0.79, "doc_class": "financial_statement"},
    "thresholds": {"area": 0.12, "yield": 0.40, "confidence": 0.86}
  },
  "render": {"dpi": 300, "crops": ["full", "table_0"], "encoder": "png"},
  "model": "claude-sonnet-4-5-20250929",
  "prompt_version": "extract_page_v4",
  "gates": {"schema": "pass", "legend_count": "pass", "range": "pass"},
  "usage": {"input_tokens": 2617, "output_tokens": 1204},
  "extracted_at": "2026-03-12T09:41:07Z"
}

That record is page 5 of the filing, and the signals block explains its own verdict: a table-and-figure area well above the threshold beside it, a text yield below its threshold, a parser confidence below its threshold. Any one of the three would have escalated the page on its own. The reasons list names two of them because the fourth signal, document class, also fired.

Storing the signal values alongside the decision, not just the decision, is what makes the record useful. With the values you can replay any historical page against a proposed new threshold and see how the routing would have changed, which turns threshold changes into something you can evaluate before shipping rather than after. Without them you know a page went to the vision path and cannot say why.

The reason the values are enough for a replay, and the decision is not, comes straight from the shape of the route function. That function is a pure map from signal values and thresholds to a verdict, holding nothing else. Keep the inputs and you can recompute the output under any thresholds you like, for any page in history, at the cost of a query. Keep only the output and the inputs are unrecoverable, because many different signal values produce the same verdict and nothing records which set produced this one. Storing the thresholds in force at the time closes the loop, since a replay has to know what it is being compared against.

This is also where an appealing but empty defence lives. A team that watches overall answer quality believes it will notice a bad routing change, and the belief is reasonable — quality is, after all, the thing being protected. What makes it empty is that an aggregate quality number moves the same way whatever caused the move: a worse model, a new customer with harder documents, a threshold edit, a regression in the parser. It returns one number for all of them, so it cannot tell you which, and a signal that gives the same reading for every cause carries no information about the cause. Escalation share broken out by rule reason does vary with the cause, which is the property that makes it a monitor rather than a mood.

The limits of a routed design#

Replacing the hand-set thresholds with a small learned model is a reasonable proposal and it measures well. A logistic regression over the same four signals plus page aspect ratio and block count, trained on the benchmark's page labels, reaches recall 0.97 at an escalation share of 7.4% — better than the hand rule on both axes, because it can trade the signals off against each other rather than OR-ing them.

The part to refuse is "no fallback". A classifier has false negatives, and its false negatives are not random: they concentrate on the pages whose layout is unusual enough that the training sample contained few of them, which is the same population where the parser does worst. So the pages the classifier misjudges are precisely the pages where sending them to the parser causes the most damage, and with no fallback the damage is silent — the parser produces something, the something gets indexed, and the only evidence that anything went wrong is an answer that is wrong six weeks later.

That correlation is the whole of the argument and it is easy to read past, so it is worth stating as a chain. A page is unusual, therefore few pages like it were in the training sample, therefore the classifier has little basis for it and is likelier to be wrong about it. Separately: a page is unusual, therefore its layout is far from what a parser's ordering and table heuristics assume, therefore the parser is likelier to be wrong about it too. One property drives both, which is why the two error sets overlap instead of cancelling. A router whose mistakes were independent of the parser's would be a much safer component than this one, and nothing about a classifier makes them independent.

A learned router also raises the auditing bar rather than lowering it. A threshold of 0.12 can be read, argued with, and changed by one person; a set of learned coefficients cannot be inspected in the same way, its behaviour shifts when it is retrained, and "why did this page go to the parser" becomes a question that requires the stored feature values to answer at all. Everything in the provenance record above becomes more necessary with a classifier, not less. The reasonable position is a classifier plus the sanity-check fallback plus the record, and the unreasonable one is any router without the second and third.

There is also an approach that deletes the routing question by deleting the text pipeline. ColPali embeds page images directly as multi-vector patch representations and retrieves over them with late interaction, meaning each query token is scored against every patch vector of every candidate page at search time rather than against a single pooled embedding per page, so no parser and no transcription ever runs — the retrieved unit is a page image, handed to a vision model at answer time. The ColPali paper and Hugging Face's Multimodal RAG using Document Retrieval and VLMs notebook are the places to see it end to end.

It genuinely removes this chapter's problem and replaces it with a different bill. A page becomes roughly 1,030 patch vectors at 128 dimensions, about 264 KB per page at 2 bytes per component, against roughly 32 KB per page for eight text chunks at 1,024 dimensions — an 8× index, or 105 GB a month at this corpus's volume, growing forever. That per-page figure is a plain product and worth following: a count of vectors, times the components in each, times the bytes a component occupies, which is the same multiply-out as any array size and lands on the quoted figure exactly. Multiply it by a month of pages and the monthly figure follows in one more step. It also produces no extracted text, so anything else that consumed the text pipeline — keyword search, exports, redaction, analytics over table values — has to be rebuilt or kept alongside. For a corpus that is mostly figures and scanned forms and whose only consumer is question answering, the trade is often worth it. For a corpus that is 94% prose feeding several downstream systems, it is not.

A router is also a component that can be wrong, and it doubles the number of ways a page can end up badly extracted: the parser can fail, the vision path can fail, and the router can send a page to the wrong one. It needs its own monitoring — escalation share by source and by rule reason, tracked daily — and its own regression tests, because a change to the layout detector silently changes the routing of every page.

It is also specifically a per-page decision. Routing whole documents to one path is much simpler and gives most of the cost back: a 40-page filing with three table pages escalates entirely, so the escalation share on this corpus jumps from 8.1% to about 34%, and the monthly bill goes from $1,101 to roughly $3,800. The per-page granularity is where the saving lives. That last figure comes from the same two-parameter model as the sweep table, which is how a design question about granularity gets answered in one line of arithmetic rather than in a trial.

A router is a rule you can state, price and replay: four cheap signals, four thresholds swept against labelled pages, an OR that escalates on any of them, a sanity check behind it for the pages it misses, and a hold state for the pages that defeat both. None of those parts is clever. What makes the arrangement work is the last one, which is not a decision procedure at all but a record — the signal values, the thresholds in force, the reasons that fired, the path that ran — because every one of the others is a number somebody will want to change, and a number nobody can replay is a number nobody can change safely.

What the provenance record above does not contain is the page itself. It names a model version, a render setting and a prompt version, all of which are pointers to things that may not exist in six weeks, and it stores no pixels. The next chapter is about a wrong answer traced to a page that was extracted correctly according to every gate in this pipeline, and about what you needed to have kept.