Ship the workflow first, then instrument the fallback rate

A recurring shape of design disagreement has no facts in it at all. Somebody proposes a fixed graph for a job whose inputs are well understood today, and somebody else objects that the inputs will change — a new supplier, a new document layout, a customer who does something nobody anticipated — and asks for a design that can cope with cases nobody has enumerated. Both people are reasoning honestly, and neither can win, because the thing they disagree about is the fallback rate: the proportion of inputs that will fail to match anything you built for. That number does not exist yet, and no amount of arguing produces it.

What breaks the deadlock is noticing that one of the two designs measures the number and the other one hides it. A deterministic graph that routes unmatched inputs into an explicit branch and increments a counter produces the fallback rate as a byproduct of running. A design that copes with unfamiliar inputs by improvising over them produces no such number, because coping and failing look the same from outside. So the cheapest way to settle an argument about future inputs is to ship the design that will report on them, which happens to be the design one side was already proposing.

Building the branch you are arguing about#

Take a concrete job to make that non-abstract. Fifty thousand invoices land in an S3 prefix every night between 22:00 and 02:00. Six fields have to come out of each one — supplier VAT number, invoice number, issue date, net total, VAT total, currency — and go into a table that finance reconciles the next morning. VAT is value-added tax, the sales tax charged across the UK and the EU, and a supplier's VAT number carries a check digit that a validator can recompute, which is why the validation step later in this chapter has something to check against. Three suppliers account for all of them, and each supplier's PDF layout has been stable for two years: Acme is 61% of volume, Northwind 27%, Globex 12%.

Those three shares sum to exactly one hundred, and that is the load-bearing fact rather than a tidy coincidence. A breakdown that closes on its total leaves no room for a fourth term, so the absence of a fourth supplier is a measured absence rather than an unlisted category quietly holding the remainder. That is precisely what the argument is about: one side is claiming that a zero which is genuinely zero today will stop being zero, and the other is claiming it will not. Neither claim is checkable against a total that already closes, which is why the argument has to be settled forward in time rather than in the room.

The workflow has five nodes and one of them exists purely to make the unknown countable:

LAYOUTS = {"acme": AcmeTemplate, "northwind": NorthwindTemplate, "globex": GlobexTemplate}

def process(pdf: bytes, doc_id: str) -> Outcome:
    text = ocr(pdf)                                   # retried, 4 attempts, jittered
    layout = detect_layout(text)                      # regex on header block + issuer VAT
    if layout not in LAYOUTS:
        metrics.increment("invoice.fallback", tags={"reason": "layout_unmatched"})
        fallback_store.put(doc_id, text, reason="layout_unmatched")
        return Outcome.NEEDS_HUMAN

    fields = LAYOUTS[layout].extract(text)            # deterministic, positional
    fields = fill_gaps_with_model(text, fields)       # one call, only for empty fields
    problems = validate(fields)                       # VAT checksum, totals arithmetic, date sanity
    if problems:
        metrics.increment("invoice.fallback", tags={"reason": problems[0].code})
        fallback_store.put(doc_id, text, reason=problems[0].code, partial=fields)
        return Outcome.NEEDS_HUMAN

    ledger.write(doc_id, fields)
    return Outcome.WRITTEN

Fifteen lines of that file are the instrument. fallback_store keeps the OCR text and whatever partial extraction existed, tagged with a reason code, and metrics.increment puts a counter behind a dashboard split by reason. Every input the pipeline cannot handle now leaves a labelled trace instead of an exception in a log, and the disagreement in the design review acquires a value that will exist by Friday.

Note where the model already appears. fill_gaps_with_model handles the fields that positional extraction misses — a supplier who prints the VAT total in words, a scanned page where the totals block shifted. That is one call with a bounded output shape, inside a node, and it is why the choice here was never between "rules" and "AI". Both designs use a model; they differ in who decides what runs next.

Making the fallback branch honest#

An instrument that under-reports is worse than no instrument, because it produces a number that gets quoted. Three things have to be true of the fallback branch for its rate to mean what people will assume it means.

It has to fail closed. The tempting shape is a catch-all that does its best — takes the fields it managed to extract, leaves the rest null, writes the row and logs a warning — and that design produces a fallback rate near zero while quietly degrading the ledger. Anything that did not fully succeed has to be a fallback, so validate returning even one problem routes the whole document out rather than writing a partial row. The first version of this pipeline wrote rows with null VAT totals and reported a 0.2% fallback rate; the real rate was 1.3%, and the gap was discovered by finance rather than by the dashboard.

That gap is the stumble this chapter exists to name, and it is worth stating in the form that makes it hardest to fall for, because nothing about it looks like a bug. The counter was correct. It was incremented, faithfully, every single time the branch it sat behind was taken, and an engineer reading the code would find nothing wrong with it. What varied underneath that constant was the definition of the thing being counted. A branch that fails open moves the boundary between success and fallback without touching the counter, so the quantity stays honest while the category it names quietly shrinks. The reason the tempting design is tempting is that it reads as resilience: writing a partial row rather than rejecting the document keeps the job moving and keeps the error log quiet. What is actually true is that a partial row is a failure that has been recorded as a success, and an instrument that classifies failures as successes reports a number that is not merely imprecise but pointing the wrong way. The direction of the error is the dangerous part — a fallback rate that fails open is always too low, and too low is the direction that ends arguments.

It has to store enough to replay. fallback_store.put(doc_id, text, reason=...) keeps the OCR text, not merely the reason, because the question you will want to answer in two weeks is "what do these 650 documents have in common", and that question needs the documents. Storing only a counter tells you how big the problem is and nothing about what it is, which is the difference between knowing the rate and being able to act on it.

And the reason codes have to be a closed set. The chain from a free-text reason to a lost finding is short enough to walk. A reason derived from an exception message carries whatever the exception carried, which usually includes a file path or a line number. Two occurrences of the same underlying fault therefore produce two different strings. A dashboard groups by the string it was given, so those two occurrences land in two groups. Repeated across a night's traffic, one fault becomes hundreds of groups of size one, and a cluster large enough to act on is invisible as a cluster. If reason is a free-text string derived from an exception message, the dashboard groups by strings that differ in a file path or a line number, and the 71% cluster that made the Northwind fix obvious never appears as a cluster. A dozen codes defined as an enum, with one of them called unknown, keeps the cardinality low enough to group and makes the growth of unknown its own signal.

There is a fourth practice that is optional and cheap. Running the new pipeline in shadow for the first week — processing everything, writing nothing to the ledger, comparing against the existing manual process on a 500-document sample — costs a week of compute and settles whether the 98.7% that does not hit fallback is actually correct. A fallback rate measures coverage, and coverage is not accuracy; a template that extracts the wrong total confidently never touches the fallback branch at all.

That last sentence names a residual the rest of the chapter cannot close, and it should be read as a limit on everything downstream of it. The counter can only see documents the pipeline declined to handle. It is structurally blind to documents the pipeline handled wrongly, because a confident wrong answer takes the same path as a confident right one. So the fallback rate bounds the work you know you cannot do and says nothing whatever about the work you did badly, and the shadow comparison is the only instrument here that looks at the other half.

Reading the number that comes back#

After two weeks — 700,000 invoices — the counter says something specific, and what it justifies depends heavily on where it lands. These are the bands that have actually driven decisions on jobs of this shape:

Only one relation is at work in the first two columns, and it is worth naming so the table stops looking like four separate observations. A nightly count is a fallback rate multiplied by a nightly volume. The table evaluates that single product at each band edge against this job's fifty thousand a night, which is why the second column can be read off the first without any further information: the smallest edge lands on a hundred a night, the next on a thousand, the next on four thousand, and above that the count grows in step with the rate. Because the relation is a product rather than a lookup, two things follow. A rate that falls between two edges lands between the two counts, so no interpolation and no re-running is needed to place it. And a job with a different nightly volume gets its own version of the table by substituting one number, which means the bands transfer to other work while the counts do not.

The rate alone is not sufficient, because a fallback rate is a count of things you did not handle and says nothing about whether they resemble each other. The two weeks of fallback_store contents answer that. On this job the rate came back at 1.3% — 650 invoices a night — and the reason codes split as 71% vat_checksum_failed, 22% layout_unmatched, 7% everything else.

Those three shares also sum to one hundred, and the closure means something weaker here than it did for the supplier volumes, which is exactly why it is worth separating them. The supplier breakdown closes because three named suppliers exhaust the traffic. The reason-code breakdown closes because its last term is everything else — a remainder that absorbs whatever the first two did not claim. A remainder always makes a column add up, so the sum carries no information about whether the categories are complete. The honest reading is that two reasons are named and identified, and the rest of the fallback population is unexamined. That matters later, because the residual is not an afterthought in this case: it is what the job is left with.

Both named reasons turned out to be fixable, and the arithmetic of what remained was available before either fix was attempted. Seventy-one per cent of six hundred and fifty is a little over four hundred and sixty a night, and twenty-two per cent of the same figure is one hundred and forty-three. Remove both and what is left is the seven per cent remainder, which is about forty-five a night, or nine hundredths of a per cent of fifty thousand. That is a prediction, not a retrospective: the composition of the fallback population fixed the post-fix rate before anybody wrote a line of the fix, which is the practical reason for insisting on a closed set of reason codes in the first place. The composition is what converts a rate into a plan.

The checksum failures turned out to be Northwind printing the VAT number with a space after the country code, which is a two-character fix to a regex and took thirty minutes. The unmatched layouts were 143 invoices a night from a single new supplier acquired in a merger, which is a fourth template and took a day. After both, the rate sat at 0.09%, and the argument about a fourth layout had been settled by the fourth layout actually arriving and costing one engineer-day.

That is the general shape. A future-proofing argument is a claim about a distribution, and the deterministic version with an instrumented fallback is the cheapest apparatus for observing that distribution, because it processes everything it can and files everything it cannot with a label attached. An agent built for the same job would have handled the Northwind spacing silently, which sounds better and is worse: the space would never have been fixed, the cost of coping with it would have been paid on every invoice forever, and nobody would have known the fourth supplier existed until someone asked why spend went up.

What the fixed graph is doing in the meantime#

While the counter fills up, the workflow is not merely a placeholder. Four things are explicit in it that a model-chosen sequence of actions makes implicit, and each one has a name that appears in a compliance conversation eventually.

Retries are attached to the steps that can have them, with policies chosen per dependency: four attempts for OCR, one attempt and no retry for ledger.write, because that write is not idempotent and a duplicated invoice row is a worse outcome than a failed batch. A single global retry policy could not express that difference, and a model deciding when to try again cannot be told it in a way that binds.

Tests exist per node. AcmeTemplate.extract has 40 fixture PDFs — sample documents committed next to the test, so the test needs nothing from the outside world — and runs in 900 milliseconds with no network and no model, so a change to the header regex is verified in CI rather than by observing the ledger the next morning. validate has its own table of arithmetic cases. The model call has an eval set of 300 gap-fill examples scored on exact match, an eval set being a fixed collection of inputs with known correct answers that is rerun whenever the prompt or the model changes.

Approvals sit on edges rather than inside judgment. Any invoice with a net total above £25,000 routes to NEEDS_HUMAN regardless of confidence, which is one comparison in one place, auditable by reading it.

Failure handling is total in the mathematical sense: every input reaches WRITTEN or NEEDS_HUMAN, and both are terminal and both are counted. There is no third thing that happens.

Totality is what makes the counter trustworthy, and the link is worth stating rather than leaving as a property of the diagram. If every input reaches one of two terminal outcomes, then the count of one outcome and the count of the other add to the count of inputs. A fallback rate computed that way cannot be diluted by a third destination that nobody is counting — no silent drop, no exception path, no row that went nowhere. The reason a rate from a total function can be quoted and a rate from a partial one cannot is that only the first has a denominator you can defend.

Three ways to answer the proposal, and why two of them lose#

The first response is the tempting one and it is a refusal dressed as a judgment. It offers no way for the person proposing the agent to be right, which means it does not resolve the disagreement so much as postpone it, and it costs credibility when the fourth supplier does turn up. The third response concedes the entire substantive point — that unanticipated inputs are coming — and disagrees only about when to pay for handling them.

Where the rate stops being the decision#

What the approach does not do is settle the question in advance for every job, and there are two ways it can mislead. A fallback rate measured over two weeks says nothing about seasonality; a job whose inputs shift in January will report a comfortable number in October. And the rate is not the cost. Six hundred and fifty fallbacks a night at 90 seconds of human review each is 16 hours of work per night, which is two full-time people, and that arithmetic can make a 1.3% rate unaffordable while a 4% rate on a job where the fallback is "skip it and log" is fine. The number to carry into the decision is the fallback rate multiplied by what a fallback costs, and the second factor is a property of the business rather than the pipeline.

Working that product for this job makes the point concrete. Before the two fixes, 650 fallbacks a night at 90 seconds each is 16.3 hours. After the fixes, 45 fallbacks a night is 68 minutes, which one person absorbs alongside other work. The rate fell by a factor of fourteen — the ratio of one and three tenths of a per cent to nine hundredths — and the staffing question disappeared entirely, and neither of those outcomes required the design to change. Had the rate stayed at 1.3% with no clustering — 650 documents a night that resembled nothing — the same arithmetic would have made the case for something else, and that is the situation the next two chapters take up.

Two properties of that product are easy to conflate and worth separating, because only one of them is a quantity of work. The hours are fixed by the rate and the per-item review time, and nothing about scheduling changes them. When those hours can be worked is a separate constraint set by the job: the nightly window closes at 08:00 when finance starts reconciling, so the review cannot be spread evenly across a day, and a queue that has to be cleared between 06:00 and 08:00 is a queue whose headcount is set by the width of the window rather than by the daily total. Compressing the same hours into a shorter window does not reduce the work; it raises the number of people who have to be present while the window is open. A staffing conversation that quotes only the daily total is answering the easier of the two questions.

The third factor that belongs in the product, and is usually left out, is what a fallback costs the person on the other end. Sixteen hours of invoice review is expensive and it is also bounded, predictable, and nobody's emergency. A fallback that means a customer waits three days for a refund has the same rate and a different cost, and pipelines whose fallback lands on a customer rather than on an internal queue need a much lower threshold before the flexibility argument becomes worth funding.

There is also a class of work where this whole method is inapplicable, because the fallback branch would catch everything, and that limit belongs on the method rather than after it. The method applies where the work can be enumerated in advance and the instrumentation is measuring how good the enumeration is. The invoice job met that condition — three layouts, six fields, one validation rule set — which is why counting the exceptions to the enumeration was informative. Some tasks cannot be enumerated at all, because the steps depend on things discovered during the run. That is a property of the work rather than a sign that nobody has tried hard enough. Drawing that boundary honestly is what keeps the criterion from hardening into a rule that produces bad designs.

The move this chapter argues for is not a preference for fixed graphs. It is a preference for designs that report on the thing being argued about, and the fixed graph happens to be one, provided its fallback branch fails closed, keeps the input, and files under a closed set of reasons. Get any of those three wrong and the number still arrives, still gets quoted, and is wrong in the direction that ends the argument prematurely.

The person proposing the agent is conceded everything except the timing. Unanticipated inputs are coming, they will need handling, and nobody knows their rate or their shape. Two weeks and fifteen lines produce both, and a rate with a composition attached tells you whether what arrived is a template to write or a tail to iterate over — which is a different decision, taken with evidence, at a point where being wrong costs an engineer-day rather than six weeks.