Precedence declarations

A precedence declaration is a sentence appended to a prompt that says which instruction wins when instructions disagree.

The JSON output requirement takes precedence over all other instructions
in this prompt. Under no circumstances return anything other than a JSON
object matching the schema.

That is the whole of it, and it is the natural response to the kind of conflict described in the previous chapter — two rules that agree on most inputs and issue different orders on a narrow class — because it is one line, it takes minutes to write, and the inconsistency it was aimed at stops immediately and completely.

What it does not do is make the disagreement go away, and the gap between those two things is where the damage lives. The losing rule is still in the prompt, still consuming tokens, still exerting pull at exactly the point where its input class fires; all that has changed is which side the model comes down on. On that input class the model now has one channel available and something it cannot express through it, so the output arrives well-formed, confident and wrong, which is a considerably more expensive shape of failure than the one that was removed.

What the line changes and what it leaves alone#

Take the invoice pair from the previous chapter: a format rule at line 4 demanding JSON and nothing else, and a rule at line 61 saying that missing information must be explained. Appending a precedence line for the format rule does not remove line 61. It re-weights the branch at the first token, moving the log probability of The from around -1.09 to somewhere past -6, which at temperature 0.2 takes the prose path from roughly thirty-eight percent of the missing-field class to under half a percent. The disagreement is unchanged; the tiebreak is now decided.

The second of those figures comes from undoing the logarithm the same way the previous chapter did: raising e to the minus six gives about a quarter of a percent, which is what "under half a percent" is rounding. So what the precedence line achieved is one thing precisely — it pushed the losing candidate's score down far enough that exponentiating it produces a number too small to matter. It did not touch the rule that generates that candidate. Every request in the contested class still carries both instructions, still pays for both in tokens, and still arrives at the same branch with both of them pulling at it. One of them now loses every time.

The problem is what happens on the winning path. The model has been told it must return an object matching a schema in which invoice_number is a required string, and it has been told, forcefully, that nothing else is acceptable. It is looking at a document that does not contain an invoice number, it still has the difficult-document rule telling it that missing information is important, and it has exactly one channel available. So it fills the field. Sometimes with null, which the schema tolerates and which is genuinely indistinguishable from a field that was present and empty. Sometimes with a value assembled from context — a number that appeared elsewhere on the page, a pattern learned from the vendor's previous invoices, a plausible reconstruction. Both of those pass validation. Neither is flagged anywhere.

The two rate figures are in different units in that table and have to be brought onto the same one before anything can be compared. Parse errors are quoted per day. Fabricated required fields are quoted per three weeks, which is twenty-one days, so nineteen of them over that span is a little under one a day. Now both columns are per day and the comparison is available.

The trade was forty loud failures a day for roughly one silent one a day. Counting incidents, that looks like a thirty-fold improvement, which is why such a change closes cleanly in review. Counting cost per incident, it is not close, and the important asymmetry is the last row: the loud failures were self-reporting, so the system knew about every single one of them the moment it happened, whereas the silent ones have to be found by a human noticing an anomaly somewhere downstream.

The operation being skipped in that first reading is a multiplication. A rate is not a cost. The cost of a failure mode is its rate multiplied by what one instance costs, and the table has those two factors sitting in adjacent rows. Comparing the rates alone is comparing one factor of a product while ignoring the other, which is legitimate exactly when the ignored factor is the same on both sides — and here the two per-instance costs are a retry against a duplicate payment plus an audit. There is no threshold in the table at which the comparison tips, because the point is not a specific ratio. It is that the two per-instance costs are not within any small factor of one another, so a large improvement in the rate column can still be a loss overall.

Backend engineers have a strong intuition for this shape already, because it is catch (e) {}. An exception is being thrown, you do not have a good handler for it, and wrapping it in an empty catch makes the error go away in the only sense that the error log measures. The system continues past the point where it no longer has valid state, and the damage surfaces somewhere with no connection to the cause. A precedence line over a conflicting rule is the prompt-level version of that: it does not handle the case, it suppresses the signal that the case occurred.

That comparison is an analogy, and the place it breaks is the more expensive half of it. A swallowed exception discards an object that was constructed: the error existed for a moment, and a logger, a breakpoint or a re-run with the catch removed will recover it. A precedence declaration discards nothing, because nothing is ever constructed. The model's determination that a field was absent from the document never becomes a value anywhere in the system, so there is no equivalent of deleting the catch and reading the exception. The only route back to the signal is to re-derive it from the source document, which is what the check at the end of this chapter does, and it recovers less than all of it.

A declaration is scoped to the prompt, not to the pair#

The wording above declares its own scope, and the scope is every other instruction in the prompt. That is considerably wider than the pair it was written to settle, and a prompt maintained for a year contains other soft rules. This one contained a second, with nothing to do with missing fields.

# line 78, in the "handling difficult documents" section
Some uploads are not invoices at all — delivery notes, remittance
advices, monthly statements. Do not extract figures from these.
Say which kind of document it appears to be and stop, so the
operations team can re-file it.

Roughly six such documents reached the extractor each week, usually a delivery note attached to the wrong email thread. Before the precedence line they behaved as that rule intended: the response was one sentence naming the document type, the parser threw, and the on-call queue re-filed it in about a minute. A precedence declaration is not scoped to the conflict you found — it settles every contest in the prompt, including the ones you have never read. Narrowing the wording helps, because the JSON requirement takes precedence over instruction 61 has a blast radius of one rule rather than sixty. A precedence line narrow enough to be safe, though, is usually specific enough that you already know exactly which disagreement you are papering over and could have repaired it instead.

What the losing rule was carrying#

Before reaching for precedence, the question worth asking is what capability the losing rule provided and where that capability goes after the tiebreak. In this prompt the difficult-document rule was the only mechanism by which the model could say "I do not know". Nothing else in the system had that ability. The schema had no field for it, the parser had no branch for it, and the operations queue had no state for it. The rule was carrying the entire uncertainty channel in a paragraph of English, which is a fragile place to keep it, and precedence closed the channel.

That reframes the fix. The conflict exists because one rule needs to express something the output format cannot carry, so the durable repair is to make the format able to carry it. Add the affordance to the schema, and the two rules stop disagreeing, because there is now an output that satisfies both.

That is the previous chapter's definition of a conflict run backwards, and it is worth seeing as such. A conflict was defined there as an input class on which no output satisfies both rules. The set of satisfying outputs was empty here because the schema had nowhere to put an explanation, not because the two sentences contradict one another in the abstract — as English, "return JSON" and "explain what is missing" have no quarrel at all. Adding a field makes that set non-empty on exactly the contested class. Once it is non-empty there is nothing left for a tiebreak to decide, which is why the precedence line and the paragraph it was arbitrating both come out, rather than one of them winning. The general form is that a format conflict is a property of the output format, so the format is the cheapest place to look for a repair.

{
  "type": "object",
  "required": ["invoice_number", "vendor_tax_id", "total", "currency", "extraction"],
  "properties": {
    "invoice_number": { "type": ["string", "null"] },
    "vendor_tax_id":  { "type": ["string", "null"] },
    "total":          { "type": ["number", "null"] },
    "currency":       { "type": ["string", "null"] },
    "extraction": {
      "type": "object",
      "required": ["status", "missing"],
      "properties": {
        "status": { "enum": ["complete", "partial"] },
        "missing": {
          "type": "array",
          "items": {
            "type": "object",
            "required": ["field", "reason"],
            "properties": {
              "field":  { "type": "string" },
              "reason": { "type": "string", "maxLength": 200 }
            }
          }
        }
      }
    }
  }
}

With that schema in place, the prompt gets shorter rather than longer. The precedence line comes out, because there is nothing left to break a tie between. The difficult-document paragraph comes out too, replaced by a single sentence that points at the structure: If a required field is absent from the document, set it to null, set extraction.status to "partial", and add an entry to extraction.missing naming the field and why it could not be determined. Two paragraphs and a precedence declaration are replaced by one sentence, the uncertainty channel becomes a typed field that the parser can branch on, and the operations queue gets a partial state it can route to a human. The behaviour is now enforced by the schema, which the model sees as a hard constraint, rather than by prose competing with other prose.

Precedence is one of several things you can do with a format conflict, and it is worth seeing them against the same criteria rather than one at a time.

The two-call option deserves a mention because it is the one people reach for when the schema is owned by another team and cannot be changed this quarter. It works, and on a 340-document-a-day pipeline the extra call is cheap. On a pipeline doing forty thousand documents a day it is not, and it adds a second place where the classification and the extraction can disagree with each other.

When precedence is the right answer#

None of this makes precedence declarations wrong in general. Some constraint hierarchies are real, and stating them is exactly correct: a safety constraint genuinely does dominate a helpfulness constraint, a legal disclosure requirement genuinely does dominate a brevity preference, and a model that resolves those the other way is worse than one that resolves them consistently. The test that separates the good case from the invoice case is whether the losing rule has somewhere to go. When a safety rule wins over a formatting preference, the formatting preference is simply not honoured on that request and nothing is lost that anyone needed. When the JSON rule wins over the missing-information rule, the information the losing rule existed to surface is destroyed, and the destruction is not recorded.

So the check before shipping a precedence line is narrow enough to run in your head. Name the input class where the two rules disagree. Describe what the output looks like on that class after the losing rule is suppressed. Ask whether that output is distinguishable, downstream, from a correct output. If it is not, you are not resolving a conflict, you are hiding one, and the hiding will hold until something expensive depends on the difference.

When the schema belongs to another team and the prompt is inside a change freeze, neither of those repairs is available this quarter. Getting part of the signal back without touching either is still possible: a cheap downstream check restores some of what a precedence line suppresses. It runs on the record rather than on the model, and it asserts that every extracted value actually appears somewhere in the source text.

import re

def _squash(s: str) -> str:
    return re.sub(r"[\s.,/\-]", "", s).lower()

def ungrounded_fields(record: dict, source_text: str) -> list[str]:
    haystack = _squash(source_text)
    return [
        field
        for field in ("invoice_number", "vendor_tax_id", "total")
        if record.get(field) is not None
        and _squash(str(record[field])) not in haystack
    ]

Run over the nineteen bad records, this flags fourteen of them, because a fabricated value is usually assembled rather than copied and does not appear anywhere in the extracted text. It costs one pass over a 4 KB string per document, about 0.4 milliseconds, and it needs no model call, so it can sit in the consumer and raise on every record.

Two details in that code carry the reasoning. _squash strips whitespace, full stops, commas, slashes and hyphens and lowercases what is left, because a correctly extracted value was copied out of the document but not necessarily character for character — a tax ID printed with spaces in the source and returned without them is the same value, and a strict comparison would report it as fabricated. The is not None test excludes fields the model declined to fill, because an honest null is the opposite of the defect being hunted.

The limit follows from the same asymmetry the check exploits: it detects assembly rather than error. The five records it misses reused a number that was genuinely on the page — a purchase-order number promoted to invoice_number, a delivery date read as an invoice date — and grounding cannot tell those from a correct read. This is a monitor, not a repair: it restores the signal at partial recall and leaves the capability exactly as gone as it was. It earns its place while a schema change waits for another team's sprint, and it should not become the reason the schema change is dropped from the plan.

There is an honest limit to the schema approach too. It works when the disagreement is about what the output can carry, which covers a large share of format conflicts, and it does nothing at all for a scope conflict where two rules demand genuinely opposite behaviour on the same input — answer in English versus answer in the customer's language cannot both be satisfied by widening a type. For those, one of the two rules has to go.

The reason precedence is so attractive is that it converts a failure you can see into one you cannot, and every metric anyone was watching improves. The reason to reach past it is that the losing rule was in the prompt because somebody needed the thing it expressed, and a tiebreak does not remove that need — it removes the channel. Widening the output format is the repair that ends the disagreement rather than deciding it, and it usually makes the prompt shorter, because two paragraphs of prose negotiating a contradiction collapse into one sentence pointing at a field.

All of this assumed the two conflicting lines had already been located, which in the invoice case took a day of looking in the wrong places and then a lucky read of the prompt. On a prompt with thirty-four numbered rules, where the failure is a case that used to work and now works four times in five, several careful readings can turn up nothing at all.