Chart values are estimates until the axes bracket them

A four-bar revenue chart from an investor deck, y-axis running 0 to 20 with gridlines every 5, labelled "€m". The extraction returned 12.4, 14.1, 15.9 and 18.2. The true series, recovered later from the company's filed accounts, was 12.9, 13.8, 16.4 and 17.1. Every extracted value was within 6% of truth, every value was in the right order, the growth story the numbers told was the same growth story the chart told, and nothing about the output invited a second look. The analyst who received them had no way to know which of the four to check, because they were all equally reasonable.

Reading the height of a bar against a scale is estimation, in the same sense that a person without a ruler estimating a shelf as "about 80 centimetres" is estimating. The model is producing a continuous quantity from pixels, which is the same weakness that made raw coordinates unreliable two chapters ago, appearing here in a form where nothing downstream will bounce. A misplaced click fails visibly. A misread bar becomes a row in a spreadsheet.

Make the model read what it is good at#

What models do reliably on a chart is read text and describe structure: the tick labels, the axis title, the legend entries, which bar is taller than which, roughly where a line crosses. What they do unreliably is interpolate between two gridlines. The check that follows from this is to demand both, and to use the reliable part to constrain the unreliable part.

For each value, require the two axis ticks it sits between, quoted as they appear on the image. A bar whose top is between the 10 and 15 gridlines has to come back with a value in [10, 15], and anything outside that interval is rejected before a human ever sees it.

{
  "chart_type": "vertical_bar",
  "y_axis": { "title": "Revenue (€m)", "ticks": [0, 5, 10, 15, 20], "scale": "linear" },
  "values": [
    { "x_label": "Q1 FY24", "value": 12.4, "tick_below": 10, "tick_above": 15,
      "position_in_band": "just below middle" },
    { "x_label": "Q2 FY24", "value": 14.1, "tick_below": 10, "tick_above": 15,
      "position_in_band": "upper third" },
    { "x_label": "Q3 FY24", "value": 15.9, "tick_below": 15, "tick_above": 20,
      "position_in_band": "just above the gridline" },
    { "x_label": "Q4 FY24", "value": 18.2, "tick_below": 15, "tick_above": 20,
      "position_in_band": "upper third" }
  ]
}

def validate(extraction):
    ticks = extraction["y_axis"]["ticks"]
    problems = []
    for v in extraction["values"]:
        lo, hi = v["tick_below"], v["tick_above"]
        if lo not in ticks or hi not in ticks:
            problems.append((v["x_label"], "quoted a tick that is not on the axis"))
        elif not (lo <= v["value"] <= hi):
            problems.append((v["x_label"], f"{v['value']} outside [{lo}, {hi}]"))
        elif hi - lo > 0.25 * (max(ticks) - min(ticks)):
            problems.append((v["x_label"], "band too wide to read to this precision"))
    return problems

On the deck above, all four values pass containment, which is the honest outcome: the bracket was never going to catch a 6% error inside a 5-unit band. What it does catch is the class of error that is an order of magnitude rather than a few percent, and that class is more common than teams expect once they start logging rejections. Running this over 800 charts from a real deck corpus, the rejections cluster into a small number of causes. A value of 1,240 quoted between ticks 10 and 15 means the model read the axis in units and the data labels in thousands. A value of 47 quoted between ticks 10 and 100 on a chart whose gridlines are 1, 10, 100 means it treated a log axis — one where each gridline is a fixed multiple of the one below it rather than a fixed amount above it, so the same vertical distance means ten times as much near the top as near the bottom — as linear and interpolated linearly inside the decade — the true value at that position is nearer 30, and the failure is invisible without the tick pair, because 47 is a perfectly plausible number. A value quoted between ticks that do not appear in the axis array at all usually means it read the secondary y-axis, a second vertical scale drawn on the right-hand side for a series measured in different units, while answering a question about the left.

The band width in the third branch of that validator is the part people skip and should not. Ticks 5 apart mean the finest honest reading is roughly ±1, so a returned value of 14.1 is claiming a precision the image cannot support, and passing it downstream as 14.1 rather than 14 ± 1 launders an estimate into a measurement. Carrying the bracket forward as the value's uncertainty, instead of discarding it once validation passes, is what lets a consumer decide whether the precision is adequate for what they are doing with it.

Let the model measure and do the arithmetic yourself#

There is a stronger version of the same idea, which is to stop asking for the value at all. The mapping from a bar's height to a number is arithmetic once you know three pixel positions: the top of the bar, and the y positions of two labelled gridlines. Asking the model for those three positions and computing the value in code moves the hard part out of the model and into a linear interpolation you can inspect — straight-line arithmetic that places the bar top proportionally between two known values.

def value_from_pixels(y_bar, y_lo, y_hi, v_lo, v_hi, scale="linear"):
    frac = (y_lo - y_bar) / (y_lo - y_hi)      # image y grows downward
    if scale == "log":
        import math
        return 10 ** (math.log10(v_lo) + frac * (math.log10(v_hi) - math.log10(v_lo)))
    return v_lo + frac * (v_hi - v_lo)

# Model reported: bar top at y=311, gridline "10" at y=402, gridline "15" at y=268
value_from_pixels(311, 402, 268, 10, 15)      # -> 13.4, not the 12.4 it estimated

The gain is concentrated where estimation is worst. On a linear axis with a zero baseline the arithmetic and the model's own guess usually agree to within a percent or two, so the extra structure buys little. On a log axis it is the difference between a right answer and one that is wrong by a factor of two, because linear interpolation inside a decade is what a model does by default and is never correct. On a chart whose y-axis starts at 40 rather than 0 — common in decks, since it exaggerates growth — the same applies, as a model that anchors on bar height rather than on the labelled scale will overstate every value in proportion.

What this does not remove is the dependence on pixel localisation, which chapter 3 established is not a strong suit. A gridline reported three pixels off in a band 134 pixels tall introduces a 2% error, which is tolerable; a bar top reported at the wrong bar is not, and that is why the bracket stays even when the arithmetic is doing the work. Requiring the model to state the tick pair as well as the pixel positions gives you two independent readings to disagree with each other, and a disagreement of more than half a band is a signal to abstain rather than to average.

The two blue lines#

A chart with two series in adjacent shades — a corporate palette where the primary and secondary blues differ by about 15% lightness — produces a failure that the bracket check waves straight through. The model reads a coherent set of values, correctly bracketed, correctly increasing, all sitting on real marks in the image. They belong to the other line. Internal consistency is exactly what you would expect, since the values were read off a real series; they answer a question nobody asked.

The safeguard for this is to require the model to quote the legend entry and the colour of the marks it actually read, as text and as a hex value sampled from the marks themselves rather than from memory of what the legend said.

{
  "series_requested": "Recurring revenue",
  "series_label_as_printed": "Recurring revenue",
  "series_colour_hex": "#1f4fd8",
  "legend_swatch_position": "bottom centre, second of three",
  "distinguishing_note": "darker of the two blues; the lighter line sits above it after Q2"
}

That answer is checkable without another model call. The legend swatch has a position in the image, and the colour of the marks has a position too, so you can sample both from the pixels and compare them against what the model claimed:

from PIL import Image
import numpy as np

def sample(img, box, exclude_near_white=True):
    px = np.asarray(img.crop(box)).reshape(-1, 3)
    if exclude_near_white:
        px = px[px.sum(axis=1) < 720]          # drop background and gridlines
    return np.median(px, axis=0)

claimed = np.array([0x1f, 0x4f, 0xd8])
swatch  = sample(chart, legend_swatch_box)      # the swatch beside "Recurring revenue"
marks   = sample(chart, plot_area_box)          # where the model says it read

if np.linalg.norm(swatch - claimed) > 40:
    raise SeriesMismatch(f"legend swatch is {swatch}, model claimed {claimed}")

A Euclidean distance of 40 in RGB — the straight-line distance between two colours treated as points in a three-dimensional red, green and blue space — is loose enough to survive JPEG artefacts and anti-aliasing, the blending of edge pixels with the background that keeps a thin line from looking jagged, and tight enough to separate two blues a designer intended to be distinguishable. Where the two shades are genuinely closer than that — which happens, and is a legitimate accessibility complaint about the chart rather than about your pipeline — the fallback is to make the model state a relational fact that does not depend on colour, of the kind in distinguishing_note above: which line is on top in a named region, which one starts higher, which one has the marker shape. A relation between two series is a structural reading rather than an estimate, and it is checkable against the values themselves, since a series claimed to sit above the other after Q2 must have larger values there.

Laid out against each other, the two checks cover different failures and neither covers everything:

What bracketing does not do#

The bracket bounds error; it does not reduce it. A value that is 6% wrong inside a correctly quoted band passes every check in this chapter, and the four numbers in the opening survived it. If the consumer of the data needs three significant figures, no arrangement of prompts and validators over an image will get you there, and the answer is a different data source rather than a better prompt.

It also assumes the axis is readable. Charts with no gridlines, no tick labels, or an axis that starts at a non-zero value the deck neglected to mark, give the model nothing to bracket against, and the correct output there is a refusal rather than an estimate with invented ticks. Requiring ticks to be a non-empty array that the model claims to have read from the image, and rejecting extractions where two consecutive quoted ticks are not evenly spaced on a linear axis, catches most of the cases where it fabricated a scale.

Stacked bars defeat it differently. In a stacked chart the top of a segment is a cumulative total, so a model reading the visible top edge of the third segment and reporting it as that segment's value produces a number that is correctly bracketed and semantically wrong by the sum of the two segments below it. The only reliable guard is to make the chart type an explicit field, as in the schema above, and to require segment boundaries rather than tops when the type is stacked.

None of this addresses the question the finance team will actually ask, which is whether an extraction pipeline that lands within 2% on 94% of values is good enough to put into production. That number is a document-level average, and the decisions downstream are made per value.