What you ask the model to emit decides what can go wrong

Asking a vision model to describe a page requires deciding what it should write, and that decision is usually made by default. Somebody asks for markdown because markdown is readable, or for JSON because JSON is parseable, and the choice is treated as packaging around an answer that exists independently of it. It is not packaging. The target format fixes the set of statements the output is capable of making, so anything the page contains that the format cannot express is lost at the moment the format is picked, before the model has looked at anything.

Two terms are worth fixing first. A schema here is a machine-readable declaration of the shape an output must take — which fields exist, what type each holds, which values an enumerated field may contain. Constrained decoding is the mechanism that holds a model to one: rather than generating freely and being checked afterwards, the sampler is restricted at each step to tokens that keep the emerging output valid against the schema. The distinction between those two matters later, because one of them is a guarantee about shape and neither of them is a guarantee about truth.

Four decisions before the first call#

The call is not "send image, get text". Four parameters are being chosen whether or not anyone chooses them, and each one moves measurable quality.

One page per call. Batching eight pages into one request is tempting for the same reason batching anything is, and it degrades output in a specific way: the model produces a document-level narrative rather than eight page-level transcriptions. Over 120 statutory reports processed both ways, single-page calls scored 0.81 on table-structure F1 — the harmonic mean of precision, the share of emitted cells that are right, and recall, the share of printed cells that were emitted, so a run cannot score well by being cautious or by being prolific alone — and eight-page calls scored 0.63, and 11% of the batched calls emitted fewer table objects than there were pages containing tables — two adjacent tables with similar headers were merged into one. Batching also destroys retry granularity, since a malformed response means re-running eight pages, and it makes cost attribution per page impossible, which matters a great deal by chapter 7.

The harmonic mean is doing something specific in that definition and it is worth a sentence, because the choice of average is what makes one number usable here. A harmonic mean sits close to the smaller of the two quantities it combines. A run that emits a single cell and gets it right has precision at its maximum and recall near the floor: an ordinary average would place it about halfway and reward extreme caution, while the harmonic mean keeps it near the floor. A run that emits every cell it can imagine inverts those two and is punished the same way. That is the property being bought — a score that cannot be raised by emitting less or by emitting more, only by emitting the right cells.

A render resolution you chose and recorded. The previous chapter showed what a fixed 150 DPI does to small print and why the number that matters is effective resolution after the provider's resize rather than the DPI you passed to the renderer. The point here is the recording: the resolution is an input to the extraction, so a stored extraction without its render settings is an experiment without its method. Providers also expose a fidelity control — OpenAI's Low or High Fidelity Image Understanding documents the low-detail mode that downsamples aggressively for a flat token cost — and that setting belongs in the same record.

A document-kind hint. Telling the model what it is looking at is a single sentence and it moves numbers. On the statutory report corpus, "This is one page of a UK statutory annual report" raised table-structure F1 from 0.74 to 0.83 against the bare instruction, and cut the rate of currency figures emitted with no unit from 22% to 4%, because the hint supplies the convention that the column heading £m at the top of the table applies to every figure beneath it. The hint has to come from something you know about the source — the ingestion route, the customer, the folder — rather than from the model's own guess, or it is circular.

The emitted structure. This is the decision the rest of the chapter is about.

Only one of those four is a requirement. What the work has to deliver is that a downstream consumer can tell what each extracted figure refers to — which column, which period, which units — without going back to the page. Everything above is a design decision serving that requirement, and each can be chosen badly without any error appearing. A batched call still returns valid output. A 150 DPI render still returns a full table. A missing document-kind hint still returns figures. The failure of a design decision here is not an exception; it is a slightly worse artefact that looks exactly like a good one.

Three formats, and what each one can hold#

Take the first row on its own, because it is the one that shows how a format loses information without anything appearing to go wrong. A markdown table's header is a single row of cells, one per column, and the grammar admits exactly one string per column with no relation between those strings. A header that spans two columns is a relation — one label governing several columns at once — and there is nowhere in the grammar for a relation to go. So the model, asked for markdown, has to map what it read onto the constructs available, the only available construct is a flat label per column, and the spanning label is dropped. The emitted table then parses, renders, carries the right number of cells and holds correct values. Nothing about it looks incomplete, which is what makes the loss undetectable rather than merely unfortunate. No wording of the instruction repairs this, because the failure is in the target and not in the reading.

The interesting thing about the middle column is where it stops. Markdown and HTML give identical answers on five of the seven capability rows — coordinates, reading-order index, empty against unreadable, legend entries, and what can be validated before indexing are all exactly the same in both. They come apart on two rows, and both of those rows are about headers. So the upgrade from markdown to HTML is a header-structure upgrade and nothing else, bought at 20 to 30% more output tokens. If the defect you are actually chasing is a dropped sidebar, a cell nobody could read, or a chart legend, moving from markdown to HTML costs real money and changes nothing at all about your ability to see it.

The word resolved in the third column needs its meaning fixed before that column can be read. A resolved column label is the label a cell's column carries once every row of the header above it has been folded in: where a fiscal-year label spans two columns and each of those columns carries its own sub-heading, the resolved label of one of them is that fiscal year together with that sub-heading, written throughout this chapter as FY2023 · Revenue. It is a single string, it belongs to the cell rather than to the header, and it is the form in which the relation markdown's grammar had nowhere to put survives at all.

HTML is the interesting middle for the row it half-closes as well. It can express a colspan, so the merged header is representable, and pipelines that emit HTML tables are meaningfully better off than pipelines that emit markdown. What it does not give you is the resolution — knowing that a header spans two columns is not the same as knowing that the third data cell in row two belongs to FY2022 · Revenue, and every consumer downstream has to redo that inference. Doing it once, at extraction, in the place where the model can still see the page, is strictly better than doing it repeatedly in code that cannot.

The schema column reads Yes down every capability row, and that constancy is worth reading correctly rather than as a verdict. It is a column about what can be said, not about what will be true. Every entry in it is a statement that the format has somewhere to put a particular fact, and none of them is a statement that the fact arriving there was read off the page. Holding that distinction is what stops the rest of this chapter from being an argument that schemas make models accurate.

The cost column is real and should be weighed rather than waved away. A schema carrying coordinates on every block runs 35–50% more output tokens than plain markdown for the same page, and output tokens are the expensive half of a transcription call. Chapter 7 puts money on that.

The schema, and the field that does the most work#

from typing import Literal
from pydantic import BaseModel, Field

class BBox(BaseModel):
    x0: float; y0: float; x1: float; y1: float   # fractions of page width/height

class Cell(BaseModel):
    row_label: str
    column_label: str          # fully resolved, e.g. "FY2023 · Revenue"
    value: str
    legibility: Literal["read", "inferred", "unreadable"]

class Table(BaseModel):
    caption: str | None
    bbox: BBox
    reading_order: int
    column_labels: list[str]   # resolved; duplicates are a defect, not a style
    cells: list[Cell]

class Figure(BaseModel):
    bbox: BBox
    reading_order: int
    caption: str | None
    chart_type: Literal["bar", "line", "scatter", "pie", "other"]
    x_axis_label: str | None
    y_axis_label: str | None
    y_units: str | None
    legend_entries: list[str]
    series: list["Series"]

class TextBlock(BaseModel):
    bbox: BBox
    reading_order: int
    role: Literal["body", "heading", "caption", "sidebar", "footnote",
                  "header", "footer", "pull_quote"]
    text: str

class Page(BaseModel):
    schema_fit: Literal["full", "partial", "none"]
    blocks: list[TextBlock] = Field(default_factory=list)
    tables: list[Table] = Field(default_factory=list)
    figures: list[Figure] = Field(default_factory=list)
    unstructured_remainder: str | None = None

Several of these fields exist to make a specific failure representable rather than to carry content. column_labels as a list of resolved strings turns the merged-header defect into a duplicate check, which is three lines of code and runs before anything is indexed. role on a text block turns the dropped-sidebar failure from chapter 1 into something a layout comparison can catch, since a page with a detected shaded region and no block of role sidebar is a page worth re-running. legend_entries as its own list, separate from the series, allows a count mismatch to surface — chapter 5 is entirely about why that particular mismatch matters.

The duplicate check is worth following through, because the mechanism is less obvious than the rule. Two columns that sit under different spans but carry the same sub-heading resolve to two different strings, since the span is part of what gets folded in. If the model failed to carry the span down, both columns resolve to the sub-heading alone and the list contains the same string twice. So a defect about a header that spans columns surfaces as a repetition in a list that has no business repeating, and it does so without any comparison against the page. That is the pattern to look for in schema design generally: a field whose internal consistency is disturbed by the failure you cannot otherwise see.

legibility is the field people are most sceptical of, correctly. Asking a generative model to report whether it could actually see something is asking it to generate another field, and that field is subject to the same pressures as the rest of the output. Measured against hand-marked ground truth on the degraded invoice set from the previous chapter, self-reported unreadable had recall of 38% and precision of 71%. It catches roughly two of every five cells that were genuinely illegible and is right about seven of every ten it flags. As a guarantee it is worthless. As a triage signal costing nothing extra, a field with 71% precision that routes pages to a higher-resolution re-run is well worth having, provided nobody writes documentation claiming that unflagged cells were read.

Those two numbers answer different questions, and which one you read depends entirely on which job you are asking the field to do. Recall is about the cells that were genuinely illegible: the three in every five it misses come back unflagged, so an unflagged cell tells you nothing whatever, and any guarantee would have to rest on recall. Precision is about the cells it flags: a triage process only spends money on those, so it only ever meets the stronger of the two numbers. The same field is worthless for one purpose and worth having for the other, and no improvement to the field changes that — the two purposes read different numbers.

schema_fit and unstructured_remainder are the escape hatch, and a schema without one is worse than no schema. A rigid structure applied to a page that does not fit it — an org chart, a hand-annotated form, a full-bleed photograph with a caption — forces the model to map the page onto categories that do not apply, and it will comply, because complying is what the constraint makes it do. Giving it a way to say partial and dump the remainder as text keeps the failure visible instead of pushing it into a Table object that never should have existed.

Making the model actually produce it#

Providers expose schema constraints as tool definitions, and the mechanism matters because it is the difference between a schema you validate after the fact and a schema the decoder cannot violate.

import json
from anthropic import Anthropic

client = Anthropic()
TOOL = {
    "name": "emit_page",
    "description": "Emit the structured contents of exactly one document page.",
    "input_schema": Page.model_json_schema(),
}

def extract(png_b64: str, doc_kind: str, render: dict) -> dict:
    msg = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=8000,
        tools=[TOOL],
        tool_choice={"type": "tool", "name": "emit_page"},
        messages=[{"role": "user", "content": [
            {"type": "image", "source": {"type": "base64",
             "media_type": "image/png", "data": png_b64}},
            {"type": "text", "text":
                f"One page of a {doc_kind}. Emit every region: body text, "
                f"headings, sidebars, footnotes, tables and figures. For each "
                f"table cell give the fully resolved column label, composing "
                f"multi-row headers (e.g. 'FY2023 · Revenue'). Mark any cell "
                f"you cannot actually read as unreadable rather than "
                f"estimating it. Coordinates are fractions of page size."},
        ]}],
    )
    block = next(b for b in msg.content if b.type == "tool_use")
    return {"page": block.input, "render": render,
            "model": msg.model, "usage": msg.usage.model_dump()}

tool_choice forcing the tool is what turns the schema from a suggestion into a constraint on decoding: the sampler is restricted at each step to tokens that keep the emerging JSON valid against the schema, so a response that omits column_labels or invents a legibility value outside the enum is not merely rejected, it is unreachable. Grammar-constrained decoding of this kind is where the field has settled over the last two years, and it is why "ask for JSON and retry on parse errors" is now a fallback rather than a design.

What the constraint does not do is make anything true. A schema that requires column_labels to be present guarantees four strings and guarantees nothing about whether those strings are the labels printed on the page. Structured output moves defects from invisible to representable, which is a precondition for measurement rather than a substitute for it.

That gap is where the confident wrong sentence gets written, and it is worth setting out plainly because the belief is a reasonable one. Constrained decoding really does deliver a hard guarantee, of a kind that is rare enough to be striking: certain outputs become unreachable rather than merely unlikely. Having seen a real guarantee, a reader slides from the shape is guaranteed to the output is validated, and then treats a schema-valid extraction as a checked one. What actually happened is that a validator was installed whose verdict does not change with the thing that matters. Feed it a cell transcribed off the page and it returns valid; feed it a cell the model invented and it returns valid. The verdict is constant while correctness varies, so a passing run carries no information about correctness, and the reassurance it produces is manufactured by the check rather than measured by it. Every rule that does carry information in this chapter compares a field against something outside itself — a duplicate against the rest of its list, a legend count against a series count, a bounding box against a detector's box.

The stored record carries the render settings, the model identifier and the usage counters alongside the page object, and that is deliberate. A page object without its render resolution and model version cannot be reproduced later, which chapter 9 pays for in full.

Coordinates you did not ask for do not exist#

bbox on every block is the field most often left out, because it inflates output and it is the one field whose value is not obviously content. It earns its place by being the only thing that lets two independently produced descriptions of the same page be compared spatially. A layout detector says there is a table occupying the region from 0.12 to 0.88 horizontally and 0.31 to 0.64 vertically; the vision model says it emitted a table; without coordinates there is no way to ask whether those are the same table, or whether the model's table covers a region the detector saw as two.

The coordinates are not precise. Measured against hand-drawn ground-truth boxes on 400 pages, the vision model's block boxes had median IoU of 0.62 against 0.94 for the layout parser's — intersection over union, the overlapping area of the two boxes divided by the area they cover between them, which is 1.0 for identical boxes and 0 for boxes that do not touch — and the errors were mostly a loose bottom edge on multi-line blocks. That accuracy is adequate for asking "is there a region here the model did not report" and inadequate for cropping an image to show a user the exact sentence a citation came from. Knowing which of those two jobs the coordinates can do stops a team from building the second one on them.

A loose bottom edge costs more than it looks like it should, and the reason is in the shape of the measure. Extending a box past the true region adds no area to the intersection, since there is nothing there to overlap, while adding all of that area to the union. The numerator holds still and the denominator grows, so the score falls purely on the denominator. A box that is correctly placed and merely too tall therefore scores far below what its placement deserves, which is why a median of 0.62 is consistent with boxes that are all in roughly the right place — and why it still cannot be used to crop, since a crop is judged by its edges rather than by its placement.

Error rate against error visibility#

Constraining the output does not reduce hallucination rate. In a controlled comparison over 300 table pages, cell values were wrong at almost identical rates under markdown and under the schema — 4.1% against 3.9%, a difference inside the noise. What changed was that 61% of the schema run's errors were detectable by a rule that ran before indexing, against 8% for markdown, because the schema errors showed up as duplicate column labels, coordinate boxes overlapping impossibly, or a legend entry count that did not match the series count.

Those two pairs of figures look as though they ought to move together and do not, and that decoupling is the result. One pair is flat: the error rate is effectively the same under both formats, so the format did not touch how often the model was wrong. The other pair multiplies by more than sevenfold: the share of errors a pre-indexing rule could catch goes from a small minority to a clear majority. Reading them as one quantity produces the wrong conclusion in either direction — that schemas make models accurate, or that since accuracy did not move the schema bought nothing. Error rate and error visibility are separate axes, and only the second is under the control of the format.

Nor does a schema help if you never compare its fields against anything. Coordinates that are stored and never checked, legibility flags that nothing routes on, and resolved labels that no rule tests for duplicates are pure cost. Every field in the schema above exists because some check downstream consumes it, and a field with no consumer should be deleted.

What you ask for is therefore a decision about the visibility of your own defects, taken before any page is read and not revisitable afterwards. A format with nowhere to put a fact will return output that looks complete and is not, and the loss leaves no trace in the artefact for anybody to find later. A format with somewhere to put it will sometimes put the wrong thing there, and a wrong thing in a named field is something a rule can be pointed at.

The output is now shaped so that two paths over the same page produce artefacts that can be compared field by field. That is the missing ingredient in the way most teams decide this question, which is to run both, paste both into a document, and ask which one reads better. The next chapter is about what a real comparison costs and what it measures.