OCR errors cluster exactly where context cannot repair them

An engine that reports 98.5% character accuracy is telling you the truth and telling you almost nothing, and the gap between those two facts is where wrong answers come from. The figure counts characters. It gives a misread digit in a deductible amount the same weight as a misread letter in the middle of a sentence, and there are vastly more of the second kind, so the aggregate is determined by the material that was never at risk.

That would be a pedantic complaint if errors were scattered at random over a page. They are not. Recognition errors cluster, for two independent reasons that happen to point at the same regions of a document, and those regions are the ones carrying the specific facts somebody will eventually ask about. What follows works out why the clustering happens, what it does to a number a vendor quotes, and which of the available defences actually vary with the thing they are supposed to detect.

English repairs itself and a policy number does not#

Ordinary prose carries enormous redundancy. A reader who encounters "the insured must notify the carrier within thlrty days" reconstructs the word without effort, and so does the pipeline: a subword tokenizer splits thlrty into fragments, the pooled embedding of the surrounding 780 characters barely moves, retrieval is unaffected, and the language model generating the answer emits "thirty" because that is overwhelmingly what the context supports. The error is present in the index and absent from the output. Information theory gives the reason a name — English text runs at roughly one bit per character of actual entropy against the roughly 4.7 bits a naive alphabet coding would need — and that surplus is exactly what lets context repair damage.

It is worth spending a sentence on what those two figures mean, because the ratio between them is the whole mechanism. The larger figure is what it costs to name one letter out of the alphabet when every letter is equally likely; it is the base-two logarithm of the alphabet size and nothing more. The smaller figure is what it actually costs once you know the preceding text, because most letters are then nearly determined. The difference between the two is surplus evidence, and it amounts to roughly four fifths of the naive cost. Damage one character and that surplus is still sitting in the other characters, which is what the tokenizer, the embedding and the model each exploit in turn.

A policy number has no surplus. The string PL-8834021 and the string PL-8B34D21 are equally well-formed as far as any language model is concerned, because nothing in the surrounding sentence constrains which digits appear. The same holds for dates, dollar amounts, statute citations, part numbers, ICD codes, addresses and every other rare literal, and those are precisely the strings a user asks about. Recognition errors and answer-bearing tokens are drawn to the same places for independent reasons, and the overlap is what turns a good-looking accuracy number into a wrong answer.

The physical clustering compounds it. OCR confusions are not random; they follow glyph shape, and the classic pairs — 0/O/D/Q, 1/l/I/7, 5/S, 8/B, 6/b, rn/m, cl/d — put digits and uppercase alphanumerics at the centre of the error distribution. Those characters live in tables, in footnotes, in the 6-point legal small print at the bottom of a rate page, and they are frequently set in a condensed or tabular-figures face that the engine's training data underrepresents. The regions of the page that are hardest to recognise and the regions that carry the specific facts are the same regions.

Reading the accuracy table as gaps#

Measuring it on a scanned insurance manual makes the gap concrete. A hand-check of 120 pages, comparing OCR output against the printed page token by token, gave this:

Add the share column downwards and it comes to a hundred per cent exactly, with nothing left over. That is worth checking before reading anything else in the table, because it establishes that the six classes partition the page: no character sits in two of them and none sits outside all of them. Without that, the total row would be an average over an unknown fraction of the document and the comparisons below would not be safe to make.

The two accuracy columns look like they are saying the same thing twice, and the interesting content is in how far apart they drift. Body prose loses a little over a point moving from characters to tokens. Identifiers lose nearly thirteen. The gap widens down the table, and the reason is mechanical: a token is wrong if any of its characters is wrong, so the penalty for going from characters to tokens grows with token length and with per-character error rate together. An identifier is long and badly recognised at once, so it is hit twice.

That mechanism can be pushed a step further, and where it stops working is informative. Treat the character errors inside one token as independent, and an identifier of PL-8834021's length at the identifiers row's character accuracy would survive intact only a little more than half the time. The table says such tokens survive about four times in five. The independence assumption under-predicts survival, and it under-predicts it for body prose too, so the direction is consistent across the table. What that tells you is that the character errors are not sprinkled independently through a token: when a token goes wrong it tends to go wrong in more than one place at once, which is what you would expect if the cause is a degraded region of the page rather than a coin flip per glyph. The practical consequence is that you cannot compute one column from the other, in either direction, and a vendor who reports only the character column has not given you enough to derive the token column.

The weighted total is close to the vendor's 98.5% and tells you nothing you can act on, because it is 91% a measurement of body prose — the share column says so directly, and 91.4% of the characters are prose. It is worth noting that the printed total is not a recomputation of the rows above it. Multiply each class's character accuracy by its share and add, and the result lands about a quarter of a point above the printed weighted total; the token column shows a smaller residual in the same direction. Those residuals are real and this chapter cannot adjudicate them, since the share column and the accuracy columns need not have been measured over the identical set of pages — a hand-check covering 120 pages of a longer manual is a sample, and a total measured directly across that sample is not obliged to agree with one reconstructed from strata. Nothing below leans on the total row, and the size of the residual is far too small to disturb any comparison between rows.

The row that predicts wrong answers is the identifiers row: one alphanumeric identifier in five is wrong somewhere. Token accuracy is the more useful column of the two, because a single bad character makes the whole token wrong for retrieval and for quotation, and it is the column vendors do not report.

The number that cannot move when the thing it measures collapses#

There is a specific and tempting inference to name here, because it is the one that puts corrupted figures into production. A team reads 98.5% character accuracy, judges it high, and concludes that the transcription is broadly trustworthy with a small and evenly distributed set of defects. The premise is correct. The conclusion does not follow, and the table above is enough to show why.

Take the identifiers row and drive its character accuracy to zero — imagine every character of every alphanumeric identifier in the manual misread. Identifiers are 1.7% of characters, so deleting their entire contribution removes about a point and a half from the weighted character accuracy. The figure lands above ninety-seven per cent. It lands above ninety-seven whether you start from the printed total or from the reconstruction of it, so the residual discussed above changes nothing here.

Sit with what that means. The quantity a user cares about has gone from four-in-five identifiers correct to none correct, which is total failure. The quantity the vendor reports has moved by less than two points and still reads as a good number. A measurement that barely moves while the thing it is supposed to track collapses entirely is not a weak signal about that thing. It is not a noisy signal either. It carries no information about it, and adding decimal places to it will not create any, because the insensitivity comes from the weighting rather than from the precision.

The corrupted chunk retrieves perfectly#

The reason this survives into production is that nothing between the parser and the answer is capable of noticing. Take two versions of a chunk from a rate page, one correct and one carrying a 4→1 misread in the deductible amount, and compare how they behave.

from openai import OpenAI
import numpy as np

client = OpenAI()

GOOD = ("HO-3 Endorsement 14 — Windstorm or Hail Deductible. Where the "
        "dwelling is located in a designated coastal zone, the deductible "
        "applicable to loss by windstorm or hail shall be $4,000 per "
        "occurrence, notwithstanding the all-perils deductible elected.")
OCR  = GOOD.replace("$4,000", "$1,000")
QUERY = "wind and hail deductible for HO-3 with endorsement 14 in a coastal zone"

def embed(texts):
    r = client.embeddings.create(model="text-embedding-3-small", input=texts)
    v = np.array([d.embedding for d in r.data])
    return v / np.linalg.norm(v, axis=1, keepdims=True)

g, o, q = embed([GOOD, OCR, QUERY])
print(f"good vs query: {g @ q:.4f}")
print(f"ocr  vs query: {o @ q:.4f}")
print(f"good vs ocr  : {g @ o:.4f}")

good vs query: 0.6883
ocr  vs query: 0.6871
good vs ocr  : 0.9986

A difference of 0.0012 in the query score, on a scale where the gap between rank 1 and rank 20 is typically 0.15, means the corrupted chunk retrieves in exactly the same position the correct one would have. Those two figures measure different things and both matter: 0.0012 is the perturbation one wrong character causes, and 0.15 is the distance the ranking treats as separating a good match from a mediocre one. The perturbation is smaller than the decision gap by more than two orders of magnitude, so it is not that the corruption is hard to see in the ranking — it is that the ranking has no resolution at which it exists.

The two chunks are 0.9986 apart from each other, which is closer than any two genuinely distinct passages in the corpus. That third figure is a useful check on the first two: the amount by which the two chunks differ from one another and the amount by which their query scores differ come out the same size, which is what a single-character change should do, and which confirms the small gap is the corruption rather than noise in the embedding call. Whatever quality controls exist downstream — similarity thresholds, rerankers, groundedness checks that verify the answer appears in the retrieved context — all pass, because the answer does appear in the retrieved context. The generation is faithful to its source. The source is false.

That last point is the one worth carrying: every guardrail built around a language model is a guardrail on the relationship between the answer and the retrieved text. None of them can see past the retrieved text to the page.

What you actually have to work with#

Three mechanisms are available, and they are worth ranking by how much they buy.

Per-token confidence is the cheapest and comes free with the engine. Tesseract emits a confidence score for every word it recognises, and the distribution over token classes mirrors the accuracy table almost exactly, which means the engine largely knows where it is struggling.

import pytesseract
from pytesseract import Output
from PIL import Image
import re

RISKY = re.compile(r"\$[\d,]+|\b\d{2,}\b|\b[A-Z]{2,}[-–]?\d{3,}\b|\b\d{1,2}/\d{1,2}/\d{2,4}\b")

def low_confidence_facts(page_png: str, threshold: int = 80):
    data = pytesseract.image_to_data(
        Image.open(page_png), output_type=Output.DICT, config="--psm 6"
    )
    flagged = []
    for text, conf, left, top in zip(
        data["text"], data["conf"], data["left"], data["top"]
    ):
        if not text.strip() or int(conf) < 0:
            continue
        if RISKY.search(text) and int(conf) < threshold:
            flagged.append({"text": text, "conf": int(conf), "at": (left, top)})
    return flagged

[{'text': '$1,000', 'conf': 58, 'at': (412, 1183)},
 {'text': 'PL-8B34D21', 'conf': 61, 'at': (96, 1247)},
 {'text': '10/1l/97', 'conf': 64, 'at': (688, 1247)}]

Mean confidence on that page's body words was 91; the three flagged tokens sit between 58 and 64. Flagging risky-shaped tokens below a threshold produced about 4 tokens per page on this manual, which is 1,400 items for the whole document and far too many to check by hand. Restricting the flag to pages that a retrieval log shows are actually being cited cut it to 180, which is an afternoon.

Domain validation is the strongest of the three and is not an OCR technique at all. Policy numbers in this insurer's scheme are two letters, a hyphen, and seven digits with a mod-11 check digit, which is a final digit computed from the others so that any single misreading breaks the arithmetic, so PL-8B34D21 fails to parse before any human sees it. Dates must resolve to a real calendar date inside the document's plausible range. Amounts in a rate table must reconcile against the printed total. Every constraint your domain already enforces at data entry is a constraint you can re-run over OCR output, and unlike confidence thresholds it produces zero false negatives on the errors it covers.

Two layers are doing separate jobs in that scheme, and separating them explains why a format rule alone is not enough. Count the characters in the format — two letters, a hyphen, seven digits — and it comes to the length of PL-8834021 exactly. The corrupted PL-8B34D21 is the same length, so length does not catch it; what catches it is that a B and a D are sitting in positions the format reserves for digits. That is a letter-for-digit substitution, and every classic confusion pair listed earlier except one crosses between digits and letters, so a format check disposes of nearly all of them without arithmetic. The exception is 1/7, which is digit-for-digit and slides straight through a format check. The check digit exists for that residue. Notably, the misread that produced the wrong deductible — a 4 read as a 1 — is digit-for-digit and appears on no classic confusion list at all, because its cause was a struck glyph in a condensed face rather than a similarity of shape. Shape-based lists do not bound the risk; a check digit that catches any single altered digit does.

The three flagged tokens in that output are worth a second look for the same reason. Each falls under exactly one of the validation rules just named: the amount reconciles against a printed total, the identifier fails format and check digit, the date fails the calendar. Two mechanisms with no shared step — a confidence threshold reading the engine's own uncertainty, and a set of domain rules reading only the string — select the same three tokens. Where that happens, the confidence threshold is redundant, and confidence earns its keep only on the tokens no rule covers.

Dual-engine agreement is the most expensive and the narrowest. Running the same page through two independent engines and diffing only the numeric and identifier spans catches errors that both confidence and validation miss, because two engines with different architectures rarely make the same substitution. It doubles OCR cost, so it is worth reserving for the pages that carry the figures — rate tables, schedules, the exhibit pages of a contract — rather than the whole corpus.

What better OCR does not fix, and what it breaks instead#

None of this is an argument that accuracy does not matter or that a stronger engine is not worth buying. It is an argument that character accuracy is the wrong number to buy against, because the quantity that determines whether users get wrong answers is field-level accuracy on the small subset of tokens that carry facts, and the two are only loosely related. A vendor moving from 98.5% to 99.2% character accuracy may have improved body prose that was already being repaired by context and left the tabular figures exactly where they were. The share column explains how that is possible without anybody lying: body prose is where almost all the characters are, so it is also where almost all of an improvement can be, and the rows that matter can sit still throughout.

The current direction of the field makes the distinction sharper rather than softer. Vision-language models used as page readers — models that accept an image alongside a written instruction and reply in text, so transcription becomes a generation task rather than a per-glyph classification, with Gemini and Claude on page images and dedicated systems like olmOCR and GOT-OCR — are substantially better than Tesseract at layout, at multi-column pages, at handwriting and at degraded scans. They also change the shape of the errors in a way that is worse for this specific problem. A classical engine that cannot read a struck digit emits something visibly broken: 8B34D21, $4,0O0, 10/1l/97. A generative model asked to transcribe a page produces fluent, well-formed output everywhere, including where it could not read, so an illegible policy number comes back as a plausible policy number and an unreadable amount comes back as a round one. The error rate may be lower and the detectability is certainly lower, and the mechanisms in this chapter — confidence scores in particular — mostly do not transfer, since a token's log-probability under a generative model reflects its plausibility rather than the evidence on the page.

Notice what that does to the two defences that survived earlier in this chapter. A broken string such as 8B34D21 is caught by a format rule precisely because it is malformed. A plausible substitute is well-formed by construction, so it passes the format rule, and only the check digit and the reconciliation against a printed total still have anything to say. The defences that read the shape of a string stop discriminating; the ones that read its arithmetic keep working. That is the criterion to apply to any new engine: not whether its output looks better, but whether the checks you rely on can still fail on it.

The honest position is that some of this is unsolved. There is no reliable way to know, at scale and without human reading, whether the number in a chunk is the number on the page. What you can do is bound the exposure: measure token accuracy by class on a hand-checked sample rather than trusting a vendor's aggregate, validate everything your domain gives you a rule for, and be explicit with users about which sources are OCR-derived so that a figure from a 1997 scan is presented differently from a figure from a Word document.

The manual's rate tables are where all of this concentrates, and they carry a second defect that has nothing to do with recognition. Even with every digit read correctly, a table put through an ordinary extractor arrives in the index as a run of numbers with no indication of which row or column any of them came from.