Explicit no-match outcomes

Replacing an LLM step with patterns is usually justified on a cost line and verified on an evaluation set, and both of those can be right while the replacement is unsafe. The property that goes missing is not accuracy. A model asked for a field it cannot find, under schema-constrained generation, returns null, and an ingestion layer that treats null as a failure parks the document for review. A regex that does not match returns nothing, and nothing becomes an empty string, and an empty string is a value.

That last sentence compresses four separate steps, and each of them is a place somebody could have intervened and did not. A pattern that fails to match returns None in Python, which is a distinct object meaning "no match object exists" — at this point the information is still intact. The extractor then reads a group off it or substitutes a default, and the conventional default for a string field is the empty string, which is where the distinction is destroyed: None and "" are different objects, but "" is what gets written. That empty string is then serialised into whatever record goes downstream, and it passes validation, because a schema that says the IBAN field holds a string is perfectly satisfied by a string of length zero. Finally a consumer branches on it — "if there is no IBAN, use the vendor master" — and executes a rule written for a document that has no bank account, on a document whose bank account it simply could not read. No step in that chain is a bug. Each is the ordinary thing to write, which is why the chain is worth tracing rather than the individual lines — and why a team can retire a model while inheriting none of its safety, and turn a coverage gap into misdirected payments, without anybody having written anything they would recognise as wrong.

The difference is that one system can say "I do not know" and the other can only say "nothing here". Downstream systems are built against defaults, and a default chosen eighteen months ago for the case where a field is genuinely absent will be applied to the case where the field is present and unreadable, silently, at whatever rate the coverage gap runs at. That is what this chapter is about: giving the replacement a vocabulary large enough to distinguish those two, and sizing the human capacity that the third outcome requires.

Three outcomes, not two#

"Did not match" covers two situations a payment system must never confuse: the field is genuinely absent from this document, and the field is present but written in a form the pattern does not recognise. The replacement has to carry the distinction in its return type, so that no caller can accidentally discard it.

from dataclasses import dataclass
from enum import Enum

class Status(Enum):
    FOUND = "found"
    ABSENT = "absent"          # positively established: no IBAN on this doc
    UNDETERMINED = "undetermined"   # no pattern covered it; we do not know

@dataclass(frozen=True)
class Field:
    status: Status
    value: str | None = None
    evidence: str | None = None   # the matched span, for review
    rule: str | None = None       # which pattern fired

def extract_iban(doc: Document) -> Field:
    if (m := IBAN_RE.search(doc.body)) and iban_checksum_ok(m.group(0)):
        return Field(Status.FOUND, m.group(0), m.group(0), "iban_body_v3")
    if doc.attachments:
        return Field(Status.UNDETERMINED, rule="iban_body_v3")
    if PAYMENT_TERMS_RE.search(doc.body) and not IBAN_HINT_RE.search(doc.body):
        return Field(Status.ABSENT, rule="terms_present_no_iban_v1")
    return Field(Status.UNDETERMINED, rule="iban_body_v3")

The field being extracted there is an IBAN — the International Bank Account Number, the string a payment run pays into. ABSENT is the expensive outcome to establish and the one worth being strict about, because claiming a field is genuinely missing is a positive assertion and needs positive evidence: here, a payment terms block present in the body with no bank details anywhere near it. Everything that is not FOUND and not provably ABSENT is UNDETERMINED, and the default in that function is UNDETERMINED rather than ABSENT because the failure that costs money runs in that direction.

The branch order encodes that asymmetry and repays reading in sequence. A checksum-passing match returns FOUND immediately, because a positive match is the only thing that licenses a positive answer. The next branch is the interesting one: if the document has attachments, the function gives up straight away and returns UNDETERMINED, without even looking for the absence evidence below it. That is not laziness. The patterns only read doc.body, so an attachment is a region of the document the extractor has not examined, and no amount of evidence found in the part it did read can establish that a field is missing from the part it did not. Only when the body is the whole document does the ABSENT test get to run, and even then it demands two things at once: a payment terms block present, and no hint of bank details anywhere near it. The final line catches everything that fell through, and it returns UNDETERMINED because falling through means no rule made a positive claim about this document either way.

The reason to arrange it that way rather than the other is that the two mistakes cost different amounts. Calling a present-but-unreadable IBAN ABSENT sends the document down the vendor-master path and can move money to the wrong account. Calling a genuinely-missing IBAN UNDETERMINED sends the document to an analyst who spends half a minute confirming there is no IBAN. One error is a payment and the other is a half-minute, so the default belongs on the side of the cheap one — and that reasoning is worth writing in a comment, because the next person to touch this function will see a queue full of documents that turned out to have no IBAN and will be tempted to flip the default to make the queue shorter.

The routing that follows from three outcomes is what actually makes the replacement safe.

The loop from the review queue back into the patterns is what makes this affordable over time rather than a permanent tax. Every document an analyst resolves is a labelled example of a format the extractor does not cover, and after two weeks of production traffic the queue has told you exactly which three layouts account for most of the gap.

Numbers that decide whether the queue is viable#

Coverage is the measurement an evaluation set cannot give you, because it is curated and production is not. Run the new extractor over one month of live traffic in shadow mode and count statuses per field. On a stream of 18,000 vendor emails a month:

The last column is the percentage applied to the stream, and it is worth computing once so the rest of the paragraph is checkable. Nine-tenths of one percent of eighteen thousand documents is 162. Two and four-tenths of a percent is 432. Four and a half percent is 810.

Those overlap, so the union is 1,044 documents a month rather than 1,404. At roughly 30 seconds of analyst time each, that is 8.7 hours a month, which at an analyst-hour of $45 is $390 — against the $198 a month the model was costing at $0.011 a document. The replacement is not cheaper until the review queue shrinks, and it shrinks because the loop in the diagram feeds new patterns back in. Once the PDF-attachment layout was covered, IBAN UNDETERMINED fell from 4.5% to 1.1%, which takes the union to roughly 430 documents a month and the review cost to about $160 — cheaper than the model, and not by much.

Three arithmetic steps sit inside that paragraph and each one is a place a proposal goes wrong. Adding the three counts gives 1,404, and that number is not the queue: a document with two unreadable fields arrives in front of an analyst once, not twice. Subtracting the union from the sum says that three hundred and sixty of the reviews were double-counted, which is another way of saying several hundred documents a month are hard in more than one way at the same time. The second step converts documents into hours: 1,044 reviews at half a minute each is a bit over half a thousand minutes, which is the 8.7 hours. The third converts hours into money at the analyst rate, giving a little under $390. Every one of those steps changes the unit, and a comparison that skips even one of them ends up putting a count of documents next to a monthly bill.

The model's side of the comparison is the same shape and takes one line. Eighteen thousand documents at $0.011 each is $198 a month. Setting $390 against $198 is the honest form of the claim, and it says the replacement began life at roughly twice the cost of the thing it replaced.

The improved figure is worth deriving too, because it is the one a proposal will quote and its derivation carries a condition. IBAN UNDETERMINED falling from 4.5% to 1.1% removes about three and a half percent of eighteen thousand documents, which is a little over six hundred. Take those off the union of 1,044 and roughly 430 are left, at half a minute each and $45 an hour, which is the $160. The condition is in the subtraction: it assumes each of those six hundred documents was in the queue only because of its IBAN. Any of them that also had an unreadable issue date is still in the queue, so 430 is a floor and the true figure sits somewhere above it. That is the right direction for an estimate that is about to be used to argue for a migration, and it is worth saying out loud rather than presenting 430 as measured.

Presenting it that way is uncomfortable and correct. A replacement's cost includes the human capacity to absorb what it cannot do, and a proposal that reports only the token saving has hidden the same kind of line item a one-sided cost comparison always hides.

The lookup table has its own version of the problem and it is easier to miss, because a table lookup feels exact in a way a regex does not. The table maps a vendor's sender domain to the invoice layout its emails use, and a domain absent from the table is an UNDETERMINED for the same reason an unmatched pattern is. The failure that costs money is a domain present in the table with the wrong entry, which happens when a vendor is acquired and keeps its domain while switching to the parent company's billing system. The extractor then applies the old layout confidently, finds a plausible field in the wrong position, and produces a FOUND with evidence attached. Entries therefore need a last-verified date and a rule that a layout unconfirmed for six months drops to UNDETERMINED on its next use, which converts a stale entry into a review item rather than a wrong payment.

What the shadow run adds that the evaluation set cannot#

A curated set is assembled from documents somebody could find and label, which biases it toward the formats that were common when it was built. Field-for-field parity on 1,200 rows containing no PDF-only invoices was a true statement about a population that was not production.

That parity is the most persuasive evidence in the whole pull request and it is worth being precise about what it can and cannot support, because "identical to three decimal places" sounds like the end of an argument. Two systems agreeing on every row of a set establishes that they behave the same way on the formats in that set. It establishes nothing whatever about formats not in it, and the strength of the agreement is no help: perfect agreement on covered formats is exactly what a coverage gap looks like from inside a curated sample, because a format nobody thought to include is a format nobody labelled either. The failure mode is worse than a blind spot, in fact. The set was assembled by people who could find and read the documents, which selects against the formats that are hardest to read, which is the same property that makes a regex fail on them. The sampling bias and the coverage gap have a common cause, so they are correlated rather than independent, and the evaluation set is systematically least informative exactly where the replacement is weakest.

Running both systems over live traffic for two weeks and diffing field by field surfaces the gap directly. Here the diff showed disagreement on 4.1% of documents, decomposing into three groups worth handling differently: cases where the model found a value and the patterns returned UNDETERMINED, which is the coverage gap at 3.4%; cases where both produced values that differed, at 0.5%, where reading forty showed the model right in most of them, all involving a second IBAN in an email footer; and cases where the patterns found a value and the model returned null, at 0.2%, where the model was wrong. Only the last group is good news, and it is small.

Those three add to the 4.1%, which is the check worth doing before reading any further: a decomposition that does not sum to the total has a fourth group in it that nobody has looked at. What makes the decomposition useful rather than decorative is that the three groups have different owners. The 3.4% is a patterns problem and is fixed by writing patterns. The 0.5% is a disambiguation problem — both systems produced a value and they disagreed — and it is fixed by deciding what the field means when a document contains two candidates. The 0.2% is the only group where the replacement is the better system, and quoting an aggregate disagreement rate would have folded all three into one number that suggests one remedy.

Running the shadow costs the model bill twice for the duration, which is $198 for the two weeks here and is the cheapest insurance in the migration. The arithmetic behind that figure is worth a moment, because it is more reassuring than it first sounds: two weeks is half a month, and paying twice over half a month comes to exactly one month's bill. The whole price of finding out whether the replacement works is one further month of the thing being switched off.

A shadow run also needs the old system left switched on and switchable back, with the cutover done per field rather than per document — invoice number and issue date crossed over in the first week, the IBAN a fortnight later once the attachment layout was covered — so that a single field's coverage gap cannot take the whole step down. A migration that removes the model on the day the new code merges has no way to distinguish "the replacement is worse" from "the replacement is different", because there is nothing left to compare against.

The recall requirement travels with the field rather than with the system. If the payment process needs 99% of the IBANs present in incoming mail to be extracted, then FOUND at 94.2% does not meet the bar on its own, and the replacement meets it only because UNDETERMINED is routed to a human who recovers the rest. Writing that down as "recall 99% at 4.5% referral" rather than "94.2% accuracy" is what lets the next person tell whether a change to the patterns broke the guarantee.

The two phrasings differ in their denominator, and that is the whole of the difference. "94.2% accuracy" counts every document in the stream, including the ones the extractor correctly established have no IBAN at all. Recall as the payment process means it counts only the documents that have an IBAN, so the 1.3% established as ABSENT leave the denominator entirely — which pushes the patterns' unaided figure to a shade over ninety-five in a hundred, still short of the bar. The referral then closes the gap, because a document routed to a human is not a miss; it is a slower success. So the guarantee has two parts and both have to be stated: the recall the process gets, and the fraction of traffic that has to pass through a person for it to get that. Quote the first without the second and a later change to the patterns can hold recall at 99% while quietly doubling the referral rate, which is a staffing problem arriving as a quality metric that has not moved.

Validation, fallbacks and abstain paths#

Validation is not the same thing as a no-match outcome, and having one does not give you the other. The iban_checksum_ok call runs mod-97 over the candidate string and rejects malformed ones, which catches truncation and the character-level noise that optical character recognition leaves behind when a number was read off a scanned page. It does not catch a well-formed IBAN belonging to the wrong party, and the 0.5% disagreement group above is exactly that: a customer's own IBAN, quoted in a footer about direct debits, matching the pattern and passing the checksum. A confident wrong answer is worse than an UNDETERMINED, because it does not reach the review queue and there is no metric on which it looks unusual. Where two candidates match in one document, the safe behaviour is UNDETERMINED with both spans recorded, and the tempting behaviour is to take the first.

The reason a checksum cannot be asked to do more is worth stating, because "it passed validation" is the single most common sentence protecting a wrong value. Mod-97 is a function of the account number's own characters: it recomputes a remainder from the digits and compares it to the two check digits carried inside the string. Every input it needs is in the string, and nothing about the invoice, the vendor or the amount enters the calculation. It can therefore detect that a string has been damaged since it was written — a dropped character, a transposition, a mis-read digit — and it can detect nothing at all about whose account the undamaged string identifies. That is a property of what the function is given, not a limitation somebody could fix with a stricter version. Any check that could catch a customer's own IBAN quoted in a direct-debit footer would have to compare the string against something outside it, which means the vendor master record, the amount, or the position in the document, and none of those is what a checksum does.

An explicit no-match is also not a fallback. The ABSENT branch routes to the vendor master record, which is the behaviour that caused the incident, and it is safe there only because it is reached by positive evidence and written to an audit log where somebody can count how often it fires.

Nor does any of this apply only to regex replacements. A trained classifier substituted for a model has the same hazard in a different shape: it always returns a class, so an input it has never seen comes back as a confident label rather than a refusal, and a calibrated probability threshold is the mechanism that gives it an UNDETERMINED at all. Whatever replaces the model needs an abstain path, and the technique that supplies it differs by arm.

The honest limitation is that the review queue puts a floor under what the replacement can save, and the floor is set by how much of your traffic is genuinely unusual. A stream with a long tail of one-off vendor layouts may sit at 8% UNDETERMINED permanently, and at that rate the human cost exceeds what the model cost, at which point retiring the model was the wrong call — and the shadow run will have said so before you shipped.

The obvious way to make the queue cheaper is to stop sending those documents to a person and send them to the model instead: patterns first, model on whatever the patterns could not resolve. That arrangement looks like it keeps most of the accuracy at a fraction of the cost, and it changes what your accuracy numbers mean in a way the arithmetic used to justify it does not survive. The model in that arrangement no longer sees the traffic its standalone number was measured on; it sees only what the patterns could not handle, which is the hardest fraction of the stream, and its score on that fraction is a different quantity with the same name.