A chart's quantities appear nowhere on the page as text

A text layer is the character data a page file carries alongside its visual marks: the actual codepoints, retrievable as strings without looking at pixels. It is what makes most extraction checkable. Chapter 2's invoice fabrication — where a vision model emitted a line item the page did not contain — was caught because the invoice printed a grand total the fabricated row had to agree with, and even when a table prints no totals, the digits it does print exist as characters somewhere: in a text layer, in a parser's output, in a higher-resolution re-run. A fabricated table cell can therefore be caught by asking whether its string is anywhere on the page.

A chart breaks that arrangement, and the break is structural rather than occasional. A fabricated table cell and a fabricated data point part company exactly here: the quantities a plot conveys are carried by the positions of marks, and positions are not characters. A chart prints its axis ticks, its legend and its caption, and depicts everything else. The page depicts the values and never states them, so there is no second source anywhere on that page to check the model's numbers against, and the checks that work everywhere else in extraction return nothing here.

Where the number comes from#

The derivation is geometric and worth doing by hand once, because seeing the arithmetic tells you exactly which errors are possible.

The model locates two tick labels on the y-axis and their pixel rows, which calibrates a linear map from vertical position to value. Then it locates the data mark and applies the map.

# Observed in the rendered page image (943 x 1220 px after resize)
tick_2000_row = 812     # pixel row of the "2,000" gridline
tick_4000_row = 498     # pixel row of the "4,000" gridline
point_row     = 553     # pixel row of the FY2022 Cloud marker

units_per_px = (4000 - 2000) / (tick_2000_row - tick_4000_row)   # 6.37
value = 2000 + (tick_2000_row - point_row) * units_per_px        # 3650.3

Each line is doing something a reader should be able to redo. Two gridlines are labelled, so two pixel rows have known values attached to them. The difference between those two values, divided by the difference between their two pixel rows, is how many units of the quantity one pixel of vertical travel is worth — which is the slope of a straight line through two known points, and it is the right operation only if the axis is linear. That condition is doing real work: a log axis, a broken axis or an axis that does not start at zero all make the same division return a number that is meaningless outside the interval between the two ticks used to compute it. The subtraction runs tick-two-thousand's row minus tick-four-thousand's row rather than the other way round because pixel rows increase downward while values increase upward, and taking the difference in that order cancels the sign flip instead of introducing one.

The second line applies the map. Start at the value of a tick whose position is known, count how far the marker sits above that tick in pixels, convert those pixels to units with the ratio just computed, and add. Every term in that calculation is an estimate produced by the same decoder that produces the text. There is no measurement subsystem inside the model; the pixel rows are not read off with a ruler, they are the model's belief about where things are.

What the arithmetic does show is the resolution floor, which is a hard physical limit independent of model quality. At 6.37 units per pixel, a one-pixel error in locating the marker is a 6.37 unit error in the answer, and at the effective resolution a full-page render survives — chapter 2 put that at about 111 DPI — a plot area 314 pixels tall spanning a 2,000-unit range cannot resolve better than about ±13 units even in principle. That figure is not an independent estimate: divide it by the units-per-pixel above and it comes to two pixels. The floor is simply the statement that the top of a bar or the centre of a marker cannot be located to better than about a pixel either side, converted into the units of the axis. For a chart whose y-axis runs to 5,000 on a quarter-page figure, the floor is closer to ±25.

The consequence is that a chart-derived quantity has a precision set by the rendering, and that precision travels with the number without being written next to it. A model can be flawless and still be wrong by that much, and no amount of prompting recovers information the raster does not contain.

Values printed on the marks themselves change this completely, because then the number is a string on the page and the operation is transcription rather than measurement. It is worth checking whether your figures are data-labelled before treating chart extraction as a hard problem, since a surprising share of business decks label every bar.

This is not a new technique. Digitising published figures by clicking on axis ticks and then on data points is what meta-analysts have done for thirty years — WebPlotDigitizer and its predecessors exist because printed papers report results only as plots — and the workflow has always included a human confirming the calibration before any point is trusted. The vision model automates the clicking and removes the confirmation, and the confirmation was doing most of the work. That is a precedent rather than a picture, and the place the two part company is worth stating: a person operating a digitiser sees the calibration fail, because the tool draws the axis it inferred back over the image. A model returns the same shape of output whether its calibration was right or wrong.

Series identity is bound by colour, and colour degrades#

Getting the value right is the smaller half. Each extracted point also carries a series name, and that name comes from matching the visual style of the plotted mark — colour, dash pattern, marker shape — against the legend swatches, then reading the text next to the matched swatch.

Three ways that binding breaks, all of which leave every number in the output plausible:

A colour collision after print and scan. Two series rendered as #2E75B6 and #4A90C2 in the original deck are separated by about 12 units of lightness; through a greyscale fax and a 200 DPI scan they differ by two or three levels of grey, and the binding becomes a coin flip. The output then attributes Cloud's trajectory to Infrastructure and vice versa. Every value is correct, every value is attached to the wrong name, and the derived answer — "Cloud overtook Infrastructure in FY2022" — is exactly backwards.

Ordinal binding when colour is uninformative. Given a monochrome chart distinguished by dash patterns the model cannot resolve, the fallback is to pair the first legend entry with the topmost series, which is right about as often as legend order matches plot order.

A secondary axis. A combination chart with revenue in £m on the left and margin in percent on the right presents two calibrations, and applying the left one to the right-axis series produces values off by two orders of magnitude while remaining internally consistent. This is the one failure of the three that a range check catches for free.

The first two share a property that decides how they can be caught. A colour collision and an ordinal misbinding both produce output that is correct in every respect except which name sits on which series, so nothing internal to the figure object is inconsistent. The counts agree, the values are in range, a second run makes the same match for the same reason. Only evidence from outside the figure separates them, which is why the gate that catches them is the one that reads the body text.

The checkable surround#

Nothing above can be verified against the values, so the checks attach to everything around them, all of which does exist as text or as a countable structure.

Each gate uses a different kind of evidence, which is why they compose rather than duplicate.

The axis and unit cross-check works because the axis title, the tick labels and the caption are printed text, so the parser or an OCR pass over the figure's bounding box produces them independently of the vision model. If the model reports y_units: "$m" and the strings inside that box include £m, the figure's facts are wrong about the most consequential attribute they have. This check found four of the errors in the chart stratum described below, and it is the cheapest of the five.

The legend-to-series count check is arithmetic on the schema from chapter 3, which is why legend_entries was given its own list. Three legend entries and four emitted series means the model split one series or invented one; three entries and two series means one was dropped. Either way the binding is unreliable across the whole figure, not just the mismatched part. The same arithmetic extends to the point count on a chart with a categorical x-axis: a figure with three series over five years must yield fifteen points, so a count that is not the product of the series count and the tick count is reporting a structural failure before any value is examined.

The range check compares every extracted y value against the tick range extended by 5%. It is trivially cheap, it catches the secondary-axis failure almost every time, and it catches nothing else, which is worth knowing before anyone claims it as validation. That last clause is the one to dwell on, because a range check is the gate teams most often present as evidence that chart extraction is being validated. It is tempting: it runs on every value, it costs nothing, and it passes. The reason it passes is the reason it is not a check. A misread bar top lands somewhere on the axis by construction — the model read it off that axis — so a value corrupted by the resolution floor, by a colour collision or by an ordinal misbinding is inside the tick range every time, and the gate returns the same verdict for a correct figure and a wrong one. It discriminates exactly one failure, the one that throws a value off the axis entirely, and the table below gives it two catches out of twenty.

Cross-run disagreement is the broadest signal and the most expensive. Run the figure twice — a second sample from the same model, or a second model — and compare the extracted points pairwise. On this corpus, agreement within 3% of the axis range on all points was the criterion, and disagreement flagged nine of the twenty known-bad figures while flagging four good ones. It costs a second call on figure pages only.

The prose consistency check is the highest-yield of the five and the one teams build last. Body text near a figure very often states one of its numbers: "Cloud revenue grew 14% to £2.9bn". Extracting numeric claims from the two paragraphs around a figure and comparing them against the corresponding extracted point caught six of the twenty. It also catches the colour-collision failure specifically, since the prose names the series.

The catch column does not add up to its own total, and the shortfall is the most useful number in the table. Add the five individual counts and the sum is twenty-four; the union is sixteen. Eight catches are therefore duplicates — figures that more than one gate found — which is what "compose rather than duplicate" means in practice and also what it does not mean. The gates use independent kinds of evidence and they still overlap on a third of their combined catches, because a badly extracted figure tends to be wrong in several ways at once. The practical consequence is that no gate's column is its contribution: removing the cross-run check does not cost nine figures, it costs however many of its nine no other gate also found. The same shortfall appears on the false-flag column, where the five counts add to six against a union of five, so one good figure is being flagged twice.

Sixteen of twenty, over 63 figures carrying 412 extracted assertions from the chart stratum of the previous chapter's benchmark. The four that survive every gate are the ones the geometry predicts: unlabelled bars whose values are wrong by 4–8% of the axis range, consistent across runs because both runs make the same misjudgement about where the bar top sits, inside the tolerance band, and not mentioned in the surrounding prose. There is no check for those. They are the residual, and the only honest response is to make the pipeline's consumers aware that chart-derived quantities carry an error bar rather than a value.

That residual is not a gap in the gate set that a sixth gate would close. Walk the four surviving figures back through the columns and each gate fails on them for a stated reason: the printed surround is correct, so the axis check has nothing to disagree with; the counts are right, so the schema checks pass; the error is a few percent of the axis range, which is inside both the range tolerance and the cross-run band; and the prose does not mention the figure, so there is nothing to compare against. Every one of those is a property of the page rather than of the checking, which is why the residual is a floor.

Because the chart stratum is a small share of any real corpus — 1.2% of pages here — the 2.1× cost of running all five gates on figure pages adds under 1% to total vision spend. Cheap checks that only apply to rare pages are the easiest quality wins available in this pipeline.

Not all chart claims are equally fragile#

Grouping the 412 assertions by kind separates what is usable from what is not:

The first four rows are a ladder, and what orders them is how much of the calibration each claim has to be right about. A directional claim needs only the sign of a difference between two marks, which survives any calibration error that affects both marks alike. An ordinal claim needs the marks ranked, which survives a wrong slope but not a wrong axis. An approximate-magnitude claim needs the slope roughly right. A precise claim needs the slope and the offset both right to within a couple of percent, which is the same couple of percent the resolution floor already spends. The accuracies fall monotonically down that ladder because each rung demands strictly more of the same measurement.

The fifth row breaks the pattern, and the break is the informative part. Series attribution sits at 78%, above precise values and below approximate ones, which puts it in the middle of a ladder it does not belong to. It is not a magnitude claim at all — it asks which name goes with which line — so its failures come from the colour-and-legend binding rather than from the calibration, and they are uncorrelated with resolution. Re-rendering the page at higher DPI moves every other row up and leaves this one where it is. That is why the two rows whose best check is prose consistency are the directional row and this one: both are claims the body text can contradict, and neither is a claim more pixels can fix.

The Best check column has one empty cell, and it is the finding rather than an omission. The only claim kind with no available check is also the least reliable of the five. Every other row has a gate that bears on it; precise values have none, because there is no evidence on the page against which a precise value could be tested, which is where this chapter started.

The shape of that table is the practical conclusion of the chapter. A vision model reading a chart is a good instrument for shape and a poor one for magnitude, and a pipeline that indexes "Cloud FY2022 revenue: 2551" as a fact is taking the 61% column and presenting it with the confidence of the 94% one. Indexing the caption, the axis labels, the legend entries and a qualitative description — with the figure's page image linked — puts the reliable claims in the index and leaves the unreliable ones where a human can look at them. Vendors building on this pattern say the same thing in their own terms; LandingAI's Agentic Document Extraction keeps the visual context attached to every extracted element precisely so the extracted claim can be checked against the region it came from.

Charts still go to the vision path, and these gates stay on charts#

None of this argues for skipping charts. The parser's chart-fact recall in the previous chapter's benchmark was 0.02, which is the number you get from a path that only sees the axis labels, and a corpus where analysts ask about trends has a real gap that only the vision path fills. The argument is about what to write into the index and with what hedge, not about whether to run the model.

It is also not a claim that these checks generalise to tables. Prose consistency works on figures because figures are discussed in the surrounding text; a forty-row table is not. Cross-run agreement is weaker on tables too, since two runs that read the same smudged digit will agree on the same wrong reading more often than two runs agree on the same misjudged bar height. The check set is figure-specific and was built by looking at figure failures.

The gate counts carry their own conditions as well. Sixteen of twenty is a result on 63 figures from one corpus, with that corpus's conventions about how figures are captioned and how often the body text quotes them. A corpus of figure-heavy decks where nothing is discussed in prose loses the highest-yield gate entirely, and its union would sit well below sixteen.

A chart is the one place in document extraction where the usual recourse is unavailable. Everywhere else, a suspicious value can be taken back to the page and looked for; here the page does not contain it, and never did. That is why the working method inverts: instead of checking the extracted quantities, check everything printed around them, count the structures that must agree, sample the model twice, and ask the body text what it says about the figure. Sixteen of twenty bad figures fall out of that, and the four that do not are the ones whose errors are the size of the rendering's own precision — which is a reason to attach an error bar to every chart-derived number rather than a reason to build a sixth gate.

Quality now has five dimensions with numbers attached and a set of gates for the flakiest of them. The benchmark that produced those numbers is about to be summarised into a single headline for a decision meeting, and the way that summary is computed decides whether the pipeline gets routed or replaced.