The next model release is not a reason to keep spending

The number came in at 156 of 200, which is 78.0%, with a 95% interval of 71.8% to 83.2%. The bar was 85%, agreed in writing on 2 March. This is one of the rare results that the machinery in chapter 3 resolves cleanly, because the upper end of the interval sits below the threshold — the experiment did not merely fail to demonstrate 85%, it produced positive evidence against it.

The sponsor's response was not an attempt to move the bar. She said that eighteen months ago the same task scored 61% on an earlier model, that the generation after that reached 71%, that the current one reached 78%, and that a fourth release is expected within the year. Three data points, roughly plus eight and plus seven percentage points, extrapolating to something in the mid-eighties. She may well be right, and a flat refusal on the grounds that the criterion was precommitted is an answer that treats process as more important than the actual question.

What the argument gets right and what it leaves out#

The trend is real and the mechanism behind it is real: document extraction has improved substantially with each frontier release, meaning each new flagship model from the handful of labs working at the leading edge, and most of that improvement arrived without anyone changing their prompts. Betting against that trend over an eighteen-month horizon has been a losing bet more often than not, which is why the argument feels strong in the room.

What it leaves out is who pays for the interval and what they get in return. Extrapolating a per-release improvement assumes the improvements land on your failure modes, and there is no reason they must — the gains from the last release were concentrated in reasoning over long documents, and this project's failures are concentrated somewhere else entirely. It also assumes a release date, which nobody outside the labs has, and the two intervals between the three data points were seven months and eleven months.

The claim is also more testable than it sounds. Of the 44 failures on the frozen 200, 27 were hand-written annotations on scanned pages where a printed quantity had been struck out and rewritten, 9 were multi-page invoices where the true total sat on page three and the model took the page-one subtotal, 5 were currency confusions on suppliers billing in USD, and 3 were supplier-name aliasing against the master data. So the sponsor's belief reduces to a specific question — does the next model read ballpoint annotations on 200-dpi scans — and that question can be answered on 27 documents in an afternoon rather than by keeping a team employed.

Three responses, with prices#

The middle row costs nothing today and is the most expensive of the three over a couple of years, because a threshold that moved once after results were seen is not a threshold any more, and the next PoC's criteria will be written by people who know that.

The first row is the plausible-sounding option and the one the shallow answer reaches for: keep a small team on it, re-evaluate when the next model ships. It fails on the ratio between what it costs and what it buys. Two engineers for sixteen weeks buys prompt engineering and preprocessing work with a historical yield of a few points on this task, which is real but not seven, and the actual event the sponsor is waiting for — a new model — arrives on a schedule the team cannot influence by working. Meanwhile the code grows, an S3 bucket fills with copies of supplier invoices, a scheduled job starts running nightly because it was convenient, and in four months the question "should we build this" has been replaced by "should we turn off the thing that is already running", which is a question with a completely different default answer.

There is a subtler cost to the first option that rarely gets mentioned, which is that continued iteration degrades the evidence it is meant to improve. A team working against the frozen 200 for sixteen weeks will run that set several dozen times, and every prompt revision is selected on its performance against those exact documents. The result is ordinary overfitting through model selection: the set stops being an estimate of performance on invoices and becomes an estimate of performance on these 200 invoices. On tasks of this kind, forty or fifty iterations against a fixed set typically buys two to four points that do not transfer, which means a team that iterates its way from 78% to 85% has not necessarily crossed the bar in any sense that production will honour. Keeping a held-out set of 100 further invoices, sealed and unread, is the standard guard, and it is worth budgeting for at labelling time precisely because the situation where you need it is the situation where you failed.

The retest is the concession, and it should be cheap#

Running the frozen 200 costs about $7.40 in model spend at current per-page pricing and forty minutes of wall-clock at concurrency 8. Adding an engineer's half-day to read the results and write two paragraphs brings the whole retest to roughly £300. That is the entire cost of finding out whether the sponsor was right, which makes the position easy to hold: the belief is worth testing and is not worth a team.

For the retest to actually cost £300 in eight months' time, it has to be runnable by someone who was not on the project, which means the repository has to be left in a state where one command reproduces the result.

# retest.sh — must run green on a clean checkout, no tribal knowledge
set -euo pipefail

# 1. verify the frozen set has not moved since 2026-03-04
sha256sum -c evals/invoices-200/manifest.txt

# 2. pre-screen: the 27 handwritten-annotation failures only, ~$1, 6 minutes
python -m poc.run --set evals/invoices-200 --filter tag=handwritten-annotation \
  --model "$MODEL" --out runs/prescreen.jsonl
python -m poc.score --run runs/prescreen.jsonl --report prescreen

# 3. full set, three runs, only if the pre-screen clears 60% on that stratum
python -m poc.run --set evals/invoices-200 --model "$MODEL" --repeat 3 \
  --out runs/full.jsonl
python -m poc.score --run runs/full.jsonl --bar 0.85 --report full

The pre-screen in step 2 is what keeps the retest honest and cheap at the same time. It targets the stratum the sponsor's argument depends on, costs about a dollar, and either supports continuing to the full run or ends the retest in six minutes. Precommitting that gate means the retest cannot quietly become "have another go at the whole thing".

A decision record is what makes the pause reversible#

A decision record is a short dated file, kept with the code rather than in someone's mail, that states what was decided, by whom, on what evidence, and what would reopen it. Stopping without a record is indistinguishable from the project being forgotten, and the difference matters in eight months when the new model ships and nobody remembers what was tried. The record does not need to be long; it needs to name what failed, under which conditions, what would change the answer, and who decides.

# decisions/2026-03-20-invoice-extraction-poc.yaml
decision: stop
date: 2026-03-20
decided_by: M. Ortiz, VP Finance Operations
result: 156/200 = 78.0% invoice-level exact match, 95% CI [71.8%, 83.2%]
bar: ">= 85%" (set 2026-03-02, unchanged)
runs: 78.0 / 77.5 / 78.5 across three passes

failed_because:
  handwritten_annotation_on_scan: 27   # struck-out and rewritten quantities
  total_on_later_page: 9               # page-1 subtotal taken as invoice total
  currency_confusion: 5                # USD-billing suppliers
  supplier_alias_mismatch: 3

conditions: see poc/conditions.md (channel mix 62/31/7, operator in the loop,
  standard invoices only, no rate limits reached)

not_disproved:
  - a future model may close the handwriting gap; historical per-release gains
    on this task: 61% -> 71% -> 78% over 18 months
  - a deskew + 300 dpi rescan pipeline was never tried and may be cheaper

retest:
  trigger: new frontier model with document/vision improvements, OR 2026-11-01
  owner: J. Reyes (engineering)          # named person, not a team
  cost: ~0.5 engineer-day, ~$8 model spend
  procedure: ./retest.sh, pre-screen gate at 60% on handwritten stratum
  same_set: evals/invoices-200 @ sha256:4f1c…, unchanged and unseen since

between_now_and_retest:
  engineering_spend: 0
  running_jobs: none                     # nightly extraction job disabled 03-20
  credentials: PoC service account expires 2026-04-03

The between_now_and_retest block is the part that gets skipped and the part that makes the rest true. A stop that leaves a scheduled job running and a service account alive is not a stop; it is an unstaffed system in production, which is the worst of the available states.

The state the diagram adds beyond the obvious ones is Closed, and giving dormancy an exit prevents a graveyard of projects that are technically awaiting retest forever. A dormant PoC whose labels are twelve months old is measuring a distribution that no longer exists, and reviving it means re-labelling before retesting, at which point the honest description is a new experiment rather than a retest.

What to actually say in the meeting#

The answer to a sponsor who says the next model will close the gap has four parts, and none of them is a defence of process for its own sake. You agree that she may be right, and say which specific claim that reduces to — whether a future model reads ballpoint annotations on 200-dpi scans, since 27 of the 44 failures are that one thing. You state the result against the bar as a fail, because the interval's upper bound is below 85% and the criterion was signed on 2 March. You offer the retest as the way to buy the same information: a named owner, a date or a release trigger, one script, half a day, roughly £300, against the same frozen set with nothing changed. And you say what stops in the meantime, which is the whole of the engineering spend, the nightly job and the credentials, because a project that is waiting for evidence should not be accumulating a production footprint while it waits.

The tone of that conversation decides whether it works. A sponsor who hears the criterion quoted back at her as a rule is being told that the process outranks her judgement, and she will reasonably resist; a sponsor who hears that her hypothesis is worth testing and that there is a cheap way to test it in November is being offered what she asked for, at a hundredth of the price. The precommitment is doing its work either way, because it is what makes 78% a fail rather than an opening position, and the negotiation is only over what happens next.

If she wants more than that, the shape of the ask changes: continuing is a new experiment with its own criteria, its own budget and its own date — a deskew and 300-dpi rescan pipeline, or a fine-tune, which is further training of an existing model on your own labelled examples, or splitting handwritten annotation into a separate path — and it gets authorised on its own merits rather than as the continuation of something that failed. That distinction between a pivot and a drift is the whole content of the disagreement. A pivot names what it is testing now; a drift keeps the team and lets the goal migrate.

There is a real cost to the pause that is worth conceding out loud. The two engineers will move to other work and will not be available in November, the third of the failure taxonomy that was preprocessing rather than model capability will not get fixed, and some knowledge evaporates no matter how good the record is. The trade is that £300 in November buys the answer that £48,000 between March and November would have bought, and the £48,000 also buys a system nobody decided to build.

Which is the harder half of this. Stopping a project is a decision somebody can make; stopping a running artefact is a different problem, because prototype code reaches production without anyone deciding anything at all.