A kill criterion needs a number, a frozen set, and a date

Three weeks into the supplier-invoice prototype, the review meeting ran fifty minutes and ended with three people holding three different conclusions. The engineer had pulled sixty invoices out of the accounts-payable mailbox, run them through a prompt that returned JSON, and checked the output by hand: forty-four had all six header fields right, twelve had one field wrong, four came back with line items merged into unusable strings. The VP of Finance Operations saw a system that read invoices and asked when the AP team of six could be reduced. The engineering manager saw 73% on a sample nobody had chosen carefully. The engineer, who knew that eight of the twelve near-misses were the same date-format problem, saw a day of work. None of them was being dishonest, and nothing in the room could settle the disagreement, because the only thing agreed in advance was that the team would spend three weeks finding out whether this was viable.

What that team spent three weeks on is a proof of concept, almost always written PoC: a small and deliberately temporary build whose output is evidence about whether a proposed system is possible at all, rather than a system anyone is meant to use. Sixty invoices out of a mailbox, one prompt, results checked by hand — the code is disposable and the finding is the deliverable. A PoC exists because the cheaper ways of answering "can this be built" are unreliable, since an architecture document argues from untested assumptions and a vendor demonstration runs on the vendor's own documents, while the expensive way answers the question only after the budget is committed. Engineers meet one whenever somebody senior asks whether a new capability could take over a manual process, such as reading invoices, triaging support tickets or summarising contracts, and nobody can say until it has been tried on real inputs for a fortnight.

A proof of concept answers a question, so somebody has to write the question down first, and that is what a kill criterion is. It is the experiment's stopping rule: a threshold, agreed before any prototype code exists, that states which result ends the work. A usable one names the measurement, the set of cases the measurement runs on, the value that counts as failure, and the date the check happens. Agreeing it while nobody yet knows the answer is the whole trick, because a bar set in ignorance is the only kind that survives meeting a result somebody wanted to come out differently.

That meeting was decided before it happened, at the moment the work was authorised without a number. A proof of concept produces evidence, and evidence only settles anything when the question was written down first. Once results exist, every number is attached to a story about why it is better or worse than it looks, and those stories are usually true. Eight of the twelve failures really were one date-format bug. The sample really was unrepresentative. The people arguing are not cynics; they are looking at genuinely ambiguous data and finding the reading that fits what they already wanted.

Three parts, and each one does work#

The criterion that would have ended that meeting in four minutes has a shape: if invoice-level exact match on the frozen 200-case set is under 85% on 20 March, we stop. Each of the three parts is load-bearing, and dropping any one of them puts you back in the fifty-minute meeting.

The number has to be a number on a defined metric. "Invoice-level exact match" means all six header fields correct on the same invoice, which is a different and much harsher measure than per-field accuracy — six fields at 97% each, if the errors are independent, gives you 83% at the invoice level. Naming which one you mean is not pedantry, because the gap between them is larger than the gap between passing and failing. "Quality is good enough for the AP team" is not a metric; it is a placeholder for a metric that somebody will define later, under pressure, with the results already on the screen.

The population has to be fixed and named before you look at it. The frozen 200 is a specific directory of specific invoices with specific labels, hashed and committed, chosen and stratified in advance — sampled in fixed proportions from named groups rather than taken as they came — 62% PDF-native, 31% scanned, 7% photographed on a phone, drawn across the supplier tail rather than the top ten suppliers who send 40% of the volume. Earlier work on golden datasets covers how that set gets built and labelled; the point here is narrower. Without a frozen population, "85%" is a rate on whatever you happened to run, and the set drifts towards the cases you fixed. Somebody adds twelve invoices from the supplier whose layout you just handled, drops the four that "aren't really representative", and the number climbs without the system improving.

The date has to be on a calendar. A criterion with no date is not a stopping rule; it is a target you approach asymptotically while the spend continues. Two weeks, ending Friday 20 March, with the review already in people's calendars, means the question gets asked whether or not the answer is convenient. Dates also make the cost of the experiment explicit before you commit to it: two engineers for two weeks is roughly £12,000 of loaded salary, which is a number the sponsor can compare against the £15,400 a month the AP team currently spends keying 14,000 invoices at about £1.10 each.

Five dimensions, because quality alone does not kill anything#

A single accuracy bar is the most common version of this and the weakest, because a system can clear it and still be unusable. Extraction that is 91% accurate at nine seconds a page and eleven cents an invoice costs more than the clerks it replaces. Extraction that is accurate and cheap but invents a bank account number once in every four hundred invoices is not a quality problem to be tuned; it is a reason not to build the thing at all. Writing criteria across five dimensions forces those trade-offs into the open at a point where nobody has anything invested in the answer.

# poc/criteria.yaml — committed 2026-03-02, before any prototype code
poc: supplier-invoice-header-extraction
sponsor: M. Ortiz, VP Finance Operations   # agreed in writing 2026-03-02
decision_owner: M. Ortiz                   # the only person who may move a bar
review_date: 2026-03-20                    # invite sent, attendees accepted
frozen_set: evals/invoices-200/            # sha256 manifest in manifest.txt

criteria:
  quality:
    metric: invoices where all 6 header fields exactly match the label
    population: the frozen 200
    bar: ">= 85%"
    also_reported: per-field accuracy, failure taxonomy   # reported, not a bar
  latency:
    metric: p95 wall clock per invoice at concurrency 8
    bar: "<= 20 s"        # overnight batch, so this is deliberately generous
  cost:
    metric: mean model spend per invoice at list prices
    bar: "<= $0.04"       # ~$560/month at 14,000 invoices
  safety:
    metric: fields emitted with a value not present in the document
    bar: "<= 0.5% of fields, and exactly 0 on payee or bank account"
  integration:
    metric: engineer-days to write one extracted invoice into the ERP staging table
    bar: "<= 5 days"      # measured by a spike, not estimated

on_fail: stop at review_date, write a decision record, no extension by default
on_change: edits after 2026-03-06 require sponsor approval in the pull request

The integration row is the one teams most often leave out, and it is the row that has ended the most invoice-extraction projects I have seen described. The model reads the invoice; the ERP, the enterprise resource-planning system that holds the finance ledger and is the system of record for what has been ordered, received and paid, will not accept a posting without a purchase-order match and a cost-centre code that lives in a different system; and the six weeks nobody scoped turn out to be the whole project. Spending one of the ten prototype days writing a single invoice end-to-end into the staging table converts that from an unknown into a measurement.

What makes a bar binding is where it came from#

A threshold in an engineer's notebook is a preference. The same threshold becomes binding when three things are true about its provenance: a named sponsor agreed to it in a written artefact before results existed, one named person owns any change to it, and everyone understands that moving the bar after seeing results is itself a recorded decision rather than a conversation.

The written artefact matters more than its format. A YAML file in the repository works because it is versioned and a change shows up as a diff with an author and a date; so does a page in the project wiki, a paragraph in the ticket, or an email that says "confirming: 85% invoice-level on the 200-case set, checked 20 March, otherwise we stop". What does not work is a slide shown once in a kickoff, because slides do not have a diff. The sponsor's name matters because the argument after the demo is not technical and cannot be settled by the person who wrote the code — the engineer who says the result is a fail is making a claim about the business, and needs to be pointing at somebody else's signature when they make it.

The last part is the subtle one. Nothing here forbids changing a bar; requirements are sometimes wrong, and discovering that 85% was the wrong number is a legitimate finding. What the arrangement removes is the possibility of changing it silently. A pull request that edits criteria.yaml on 19 March, with the sponsor as reviewer and a sentence explaining why, is a decision the organisation made and can look back at. The same change made verbally in the review meeting is indistinguishable from the result being reinterpreted, and six months later nobody will remember which it was.

The one that gets argued with#

Of those five, the quality bar is the one that will be argued with, and the argument will not be about the number. Nobody stands up and says 85% should have been 78%. What happens is that the metric gets redefined underneath the number: the result comes in at 81% invoice-level, and someone points out that per-field accuracy is 96.4%, which is true, and that most failures are a single field, which is also true, and that a clerk reviewing only the low-confidence extractions would catch them, which may well be true and is a different system than the one that was tested.

Two things blunt that. The first is to precompute the alternative metrics and put them in the same document as a reported number rather than a bar, so that when someone reaches for per-field accuracy it is already on the page and already labelled as not the criterion. That removes the rhetorical advantage of arriving with a new number nobody has seen. The second is to write down, in advance, what the alternative reading would have to demonstrate to count: if the argument is that a confidence-gated human review rescues the system — the extractor attaches a self-reported confidence score to each value it produces, and a clerk checks only the ones below a chosen score — then the claim is that a threshold exists at which the flagged subset contains most of the errors and is small enough to review — and that is measurable on the frozen 200 in an afternoon. Either the confidence scores separate errors from correct extractions or they do not, and at 81% invoice-level with flat confidence, sending 40% of invoices to a clerk is not automation.

The response, when the argument comes, is neither to win it nor to concede it in the room. It is to say that the bar was set at 85% invoice-level by the sponsor on 2 March, that the result is 81%, that this is a fail under the agreed criterion, and that changing the criterion is available as a decision the sponsor can make in writing along with the reason. That converts a debate about interpretation into a decision with a name attached, which is the only form in which it can be revisited later.

What this is not#

A kill criterion is not a production service-level objective. The bar exists to answer one question at one moment: is there enough evidence to keep spending? An SLO governs a running system, is measured continuously, and has error budgets and consequences attached; the two get confused because both are numbers with thresholds, and teams sometimes lift the PoC bar straight into production as if 85% invoice accuracy were an operating target. It is not, and what a passing PoC does and does not license is worked through later in this topic.

It is also not a substitute for deciding what to prototype. The criteria file above tells you what would count as success on invoice extraction, and it says nothing at all about whether invoice extraction is the part of this system that could kill the project. A team with ten prototype days and five criteria can still spend all ten of them on the component that was never in doubt, arrive at the review with a passing quality number, and learn nothing about whether the thing can be built. Choosing where the prototype budget goes is a separate decision from choosing what counts as passing, and it is the one that determines whether the evidence you buy is worth anything.