Vision locates the table; structured data supplies the figures

The pricing appendix was a nine-column table at roughly 6-point type, 34 rows, printed as a PDF page. The extraction came back with the structure perfect: every row present, every SKU string correct, the column headers in order, the merged cells under "Enterprise" handled properly. Two columns of numbers had swapped places. list_price held the contract price and contract_price held the list price, on 34 of 34 rows, consistently — which is what made it survive review, because a spot check of three rows found three internally coherent rows.

Rewriting the prompt did not fix it. Naming the columns explicitly did not fix it, nor did asking for the header text alongside each value, nor did requesting the table one column at a time, which produced the same swap on the two columns concerned and correct output on the other seven. The reason is in the image rather than the instructions: after the page was rasterised — rendered from the PDF's vector description into a grid of pixels — at 150 dots per inch and downscaled to fit the provider's long-edge cap, the two numeric columns were 41 pixels apart with x-heights of about 4 pixels, the x-height being the height of a lower-case letter without ascenders and the measure that legibility actually tracks, and the visual association between a digit and its column header had thinned to almost nothing. Earlier work on small text established where legibility gives out; what this chapter is about is what to do at that boundary rather than how to detect it.

Three responses, and one of them is usually right#

Rejecting on low confidence is the response teams reach for first and the weakest of the three as normally implemented, because the model's own confidence on this failure was high. It had every reason to be: the layout was right, the numbers were real numbers from the page, and nothing in its output was internally inconsistent. Self-reported confidence detects the cases where the model already knows it is struggling, which are the blurry scan and the cut-off column, and it does not detect a clean read of the wrong column.

Structural invariants detect it, and they cost nothing to compute. Cross-footing is the strongest single check on financial tables: if the table carries a total row or a total column, the extracted cells must sum to it, and a transposition of two columns whose values differ will break the column totals immediately. Domain invariants do similar work where totals are absent, because a contract price above list price on all 34 rows is a claim about the world that a pricing table would not make.

def audit_table(rows, totals, tolerance=0.005):
    problems = []
    for col, stated in totals.items():
        got = sum(r[col] for r in rows)
        if abs(got - stated) > tolerance * max(abs(stated), 1):
            problems.append(f"{col}: cells sum to {got:,.2f}, table states {stated:,.2f}")

    inverted = [r["sku"] for r in rows if r["contract_price"] > r["list_price"]]
    if len(inverted) > 0.2 * len(rows):
        problems.append(f"contract > list on {len(inverted)}/{len(rows)} rows — columns may be swapped")
    return problems

Asking for a cleaner capture is the second response, and it is often the cheapest real fix. The 6-point text was unreadable because of decisions made before the model saw it, all of which are yours to change: rasterise the PDF page at 300 DPI instead of 150, crop to the table region and send the crop at native resolution rather than the whole page downscaled, or split a nine-column table into three overlapping four-column crops so each one survives the resize with usable x-heights. On the appendix above, cropping to the two disputed columns and re-asking resolved them correctly on all 34 rows, at the cost of a second call and about 1,100 extra input tokens.

The third response is the one to design for when it is available, and it inverts the roles. Vision is excellent at finding the table on the page and telling you its bounds, its headers and its shape, and that is a structural reading rather than a character-level one. The exact figures then come from a source that has them as data — the PDF's own text layer, which is the characters and their positions that the file stores underneath the rendered page, the CSV behind the dashboard, the API the screenshot was rendered from, the database the report was generated against.

import pdfplumber

# Model returns the table's normalised bounds and header order; it never reads a digit.
region = {"page": 12, "bbox": [0.08, 0.31, 0.94, 0.78],
          "headers": ["SKU", "Description", "List price", "Contract price", "Uplift %"]}

with pdfplumber.open("appendix.pdf") as pdf:
    page = pdf.pages[region["page"] - 1]
    x0, top, x1, bottom = (
        region["bbox"][0] * page.width, region["bbox"][1] * page.height,
        region["bbox"][2] * page.width, region["bbox"][3] * page.height,
    )
    table = page.crop((x0, top, x1, bottom)).extract_table()

# Characters come from the PDF's text layer; column order comes from the model's headers.
assert [h.strip() for h in table[0]] == region["headers"]

The characters are now exact, because they were never estimated, and the model has been used for the job it does well. Where this applies it should be the default, and it applies far more often than people check — most PDFs generated from a spreadsheet or a reporting tool carry a full text layer, most dashboards have a CSV export or a JSON endpoint one network tab away, and most vendor reports exist as a database row somewhere before they exist as a picture.

Ranking the three against the pricing appendix makes the order clear. Rejection would have shipped the swapped columns, because confidence was high and nothing looked wrong. The cleaner capture fixed it and cost a second call plus the engineering to decide when to re-crop. Routing the digits to the PDF's text layer fixed it permanently, removed the failure mode from the system rather than detecting it, and left the model doing the part no parser can do, which is knowing that this particular rectangle on page 12 is the pricing table rather than one of the four other bordered blocks on the page.

When the number exists only as a picture#

Finance asks for revenue figures from chart images in PDF investor decks, the underlying data is genuinely unavailable, and the pipeline lands within 2% of truth on 94% of values in a test set. The output feeds a screening model that shortlists companies. The tempting reading is that 94% within 2% is comfortably inside the noise of a screening decision, so it ships with a monthly spot check.

That reading uses the wrong statistic twice. It treats a per-value accuracy as though the unit of consequence were the value, when the unit of consequence is the company that gets shortlisted or does not. And it summarises the 6% as though those errors were small, when the checks in the previous chapter showed that the errors which escape are drawn from a different distribution than the ones inside the band. A misread unit, a log axis interpolated linearly, a value taken off the secondary y-axis: each is wrong by a factor rather than by a percent.

Put numbers on it. A screening run over 500 companies pulling 12 values each is 6,000 values, so 6% is 360 values that are not within 2%. Sampling the tail from the same corpus, roughly a fifth of those — around 70 values — are wrong by more than 20%, and a value wrong by 20% or more on a growth rate or a margin is enough to move a company several places in any ranking built on it. Whether that matters depends entirely on where the company sits relative to the cut. A company ranked 300th whose revenue is overstated by 40% moves to 210th and is still not shortlisted, so the error costs nothing. A company ranked 55th against a top-50 cut moves to 44th on the same error, and a real company is displaced by a false one.

The design that follows is per-value abstention rather than per-document acceptance, abstention meaning that the pipeline returns no number for an item it cannot stand behind instead of returning its best guess. Every value carries its bracket, and any value whose checks fail, whose band is wider than the tolerance the decision needs, or whose series attribution could not be confirmed, is not returned as a number at all. It goes to a queue where a person reads that one chart, which takes about forty seconds. At an abstention rate of 8%, that is 480 values across the 500-company run, or roughly five and a half hours of human reading for a job whose alternative is not doing it.

The second half of the design is decision-aware, and it is what makes the answer to finance defensible. Once each value carries an interval rather than a point, you can propagate the intervals through the screening model and ask which companies change side when every value moves to the unfavourable end of its band. Those companies get reviewed regardless of confidence, because they are the only ones where the error can change an outcome.

def boundary_sensitive(companies, score, cutoff_rank=50):
    base = sorted(companies, key=lambda c: -score(c, "point"))
    shortlisted = {c.id for c in base[:cutoff_rank]}
    flips = []
    for c in companies:
        lo, hi = score(c, "low"), score(c, "high")           # bands, not points
        rank_lo = sum(1 for o in companies if score(o, "point") > hi)
        rank_hi = sum(1 for o in companies if score(o, "point") > lo)
        if (rank_lo < cutoff_rank) != (rank_hi < cutoff_rank):
            flips.append(c.id)                              # band straddles the cut
    return flips, shortlisted

On a run like the one above this typically flags 20 to 40 companies out of 500, which is another two to five hours of review and turns a statistical claim into a bounded one: every company whose shortlist status could have been changed by extraction error was read by a person. That is shippable. "94% is inside the noise" is not, because it never establishes that the 6% is distributed harmlessly, and the whole point of the failure modes in the last chapter is that it is not.

What this does not resolve#

Abstention has a cost that grows the wrong way. The abstention rate is highest on exactly the documents that are worst prepared — scanned decks, screenshots of screenshots, charts drawn without gridlines — so the human queue fills with the hardest reading, and a pipeline that abstains on 8% of a clean corpus can abstain on 30% of a poor one. Measuring the abstention rate per source, and telling the requester before you build that a badly sourced corpus turns this into a mostly-manual process, is more honest than discovering it in month two.

There is also a limit on invariants. Cross-footing catches transpositions when totals exist, and a table without totals or a chart without a stated sum gives you nothing structural to check against. In those cases the only real controls are the ones that make the reading easier — a higher-resolution capture, a tighter crop — and the willingness to return nothing.

All of this has assumed the pixels are the only channel available. That assumption is worth testing directly, because the case for driving anything by image rests on an integration being unavailable or not worth building, and that is a claim with an arithmetic behind it.