Predicting a schema's worst field

A schema review before anything ships has to be done from the schema, because that is all there is. There is no production data, no labelled sample and often no test fixtures, and the reviewer has twenty minutes. What makes that possible is that the failure modes are structural: each one is visible in the declaration, and each one has a characteristic rate. It is also, not by coincidence, a standard interview exercise — a schema and twenty minutes is a problem with a checkable answer and nothing to look up.

The output worth producing is a ranking with numbers attached, because a prediction with no number cannot be wrong, and a review that cannot be wrong teaches nobody anything. Here is a four-field schema that arrived in a pull request titled "support ticket triage, v1":

{
  "ticket_id":         { "type": "string" },
  "is_refund_request": { "type": "boolean" },
  "sentiment":         { "enum": ["positive", "neutral", "negative"] },
  "next_action":       { "type": "string", "description": "what should happen next" }
}

Four questions that rank a schema in twenty minutes#

The reading is a small number of questions applied to each field in turn, and each of them corresponds to something an earlier chapter of this topic showed failing.

Is the set of legal answers fixed anywhere? A field with an enum has its answers fixed by the decoder, a field with a description listing values has them fixed with a failure rate, and a free string has them fixed nowhere. next_action is the extreme case: the space of legal answers is every English sentence, so two responses that mean the same thing are different strings and two that differ importantly can look identical.

Can both legal answers be honestly supported by the input? This is the is_exploitable question from chapter 1, on names that assert what the source does not establish, and it applies to any boolean or small enum that makes a claim about the world rather than about the document. A field where the source usually establishes neither value produces confident output with no relationship to truth.

Does one member's definition swallow the others? Chapter 3's severity case, where a single trigger nearly every input satisfies collapsed a three-way distinction. Read each enum member's trigger, estimate what fraction of your actual inputs satisfies it, and check that the fractions sum to something near one and do not overlap heavily.

Is the field answering one question? Chapter 5's test, applied by writing the field's question as a sentence and looking for an "and".

Two more read off the shape rather than the semantics: how deep the field sits, which chapter 6 quantified, and whether its description names a literal value the type does not have, which chapter 3 called the tell.

Applied to the triage schema, the ranking falls out and so does a predicted rate for each:

ticket_id is copied verbatim from the input and fails loudly at the next database lookup, which is why a low rate there costs almost nothing. is_refund_request asks about something the ticket text usually establishes explicitly, so both values are grounded in the document. sentiment has a fixed vocabulary of three and a boundary problem only between neutral and negative. next_action is a free string whose description states neither who acts, nor by when, nor from what set of possible actions, so every response is an unconstrained sentence and there is no definition of correct to check it against.

Where those ranges come from is what makes the review teachable rather than a matter of taste. A predicted failure rate is two estimates multiplied together: the share of inputs falling in a field's ambiguous region, and the chance of getting one of those wrong. For is_refund_request the ambiguous region is partial refunds, credits and goodwill gestures, which is a small and nameable slice of support mail, and inside that slice the model is close to a coin flip — a small share times a large error rate gives the single-digit prediction. For sentiment the ambiguous region is everything sitting on the neutral-to-negative boundary, which on a support queue is a wide slice, and the error inside it is again roughly even, which is what lifts the prediction into the fifteen-to-twenty-five band. For ticket_id there is no ambiguous region at all, because the value is copied rather than decided.

next_action breaks that method, and noticing that it does is worth more than the number attached to it. There is no ambiguous region because there is no region: with no fixed answer set, "correct" has no definition, so what is being predicted is not an error rate but a usability rate — how often a sentence written with no stated actor, no stated deadline and no stated set of options turns out to be actionable anyway. A field whose predicted failure rate cannot be assembled out of a share and an error rate is a field with no specification, and that is a stronger finding than any percentage.

The reflex answer, and why it loses#

Give an engineer a schema and ask which field will be wrong most often, and the reflex is to name the only free-text field. That instinct gets next_action right above, and it goes wrong as soon as free text is the least of the schema's problems.

Take the security-triage schema from chapter 5: severity as an enum of low, medium, high and critical described as impact if exploited; is_exploitable as a boolean; affected_component as a free string; recommended_action as an enum of patch, mitigate, accept and duplicate. affected_component is the free-text field, and it is the field least worth worrying about. A component name is copied out of the report, where the researcher has usually named it precisely because their whole point depends on it, and it is consumed by a lookup against a service registry. When the model writes auth-service and the registry knows auth-svc, the lookup misses, something logs, and a human sees it the same day. Measured against a registry of 340 components, exact matches came back on 71 percent of reports and the remaining 29 percent were visible as unmatched strings, of which roughly two thirds were near-misses a fuzzy match resolved.

Following that through is what demotes the field, rather than the 71 percent on its own. Twenty-nine percent of reports did not match the registry exactly. About two thirds of those are spelling and abbreviation variants that a fuzzy match against 340 known component names resolves, which leaves something under a tenth of reports naming a component nobody can identify. And that tenth is loud: an unmatched string is logged, a human reads it the same day, and the repair is a rename. A field whose worst case is a same-day rename is not competing for the top of this list with a field whose worst case is a wrong escalation that nothing detects.

The two fields that matter are worse in the specific way that produces no signal at all. is_exploitable demands a verdict about a deployed system from a document written by someone with no access to the deployment, so both values are unsupported and the model picks the one the report's tone leans towards. recommended_action folds a workflow state into a vocabulary of actions and destroys the same axis on every response. Neither of them errors, neither of them looks unusual in aggregate, and both of them feed decisions.

Ranking them: is_exploitable first, because a wrong value there changes whether a report is escalated and nothing downstream can detect it; recommended_action second, because the loss is systematic rather than random and therefore survives averaging; severity third, since impact if exploited gives no per-member definition and inherits is_exploitable's uncertainty on top of that; affected_component last.

A prediction you cannot falsify is a preference, so the second half of the review is the important one. For is_exploitable, the evidence that would prove the ranking wrong is inter-annotator agreement, meaning how often two people labelling the same inputs separately arrive at the same answer: have two security engineers independently label 100 reports from the report text alone, with no access to the deployment. If they agree at Cohen's kappa above about 0.8 — kappa scores that agreement after subtracting the amount two people would hit by chance, so 1.0 is exact agreement and 0 is no better than guessing — the reports do establish exploitability, the field is answerable, and the objection collapses, at which point the model's agreement with the human consensus becomes the real measure. If the engineers agree at 0.4, no schema change will save the field and the number was never obtainable. For affected_component, the falsifier is the registry match rate: at 40 percent rather than 71, it would be the worst field in the schema and the ranking would be wrong. Both tests are a day's work and neither requires running the model.

The subtraction inside kappa is the part worth understanding, because raw agreement on a boolean is a misleading number on its own. Two annotators who both answer true on nearly every report will agree on nearly every report without either of them having read one carefully. Kappa handles that by working out how often two people with those answering habits would coincide by accident, taking that away from the agreement actually observed, and dividing what is left by the room the chance figure had left available. A high kappa therefore means the two agreed on the cases where agreement was not free, which is the only agreement that says the question has an answer.

Running it on people before running anything on the model is what bounds the rest of the exercise. If two security engineers reading the same report reach different verdicts, there is no ground truth for is_exploitable to be scored against, and any accuracy figure computed for the model is measuring its agreement with whichever annotator happened to label the set. That is the sense in which a low kappa means the number was never obtainable: not that it is hard to measure, but that there is nothing there to measure.

The change to recommended_action is chapter 5's split into a disposition and an action, with per-member definitions replacing impact if exploited on severity. The change to is_exploitable is chapter 1's rename to report_claims_exploitable with a description that says what counts as a claim, which converts a question the document cannot answer into one it can.

Running the read backwards#

The same six questions work as a diagnostic when you already have a bad response and need to find the line of schema that caused it. The value of doing it this way is that the alternatives are worse: raising the temperature, adding "be accurate" to the system prompt, or switching models are all things teams try first, and none of them addresses a field that is unanswerable as specified.

Most of that table can be checked mechanically before anything ships, which is worth automating because a schema review is exactly the kind of task that gets skipped under deadline:

from typing import Any

SENTINEL_WORDS = ("unknown", "n/a", "none", "tbd", "use 0", "-1", "empty string")
AMBIGUOUS_UNITS = ("pct", "rate", "amount", "time", "size", "duration", "value")

def review(schema: dict[str, Any], path: str = "", depth: int = 1) -> list[str]:
    findings: list[str] = []
    for name, field in schema.get("properties", {}).items():
        here = f"{path}.{name}" if path else name
        desc: str = field.get("description", "")
        ftype = field.get("type")

        if ftype == "object":
            findings += review(field, here, depth + 1)
            continue
        if depth >= 4:
            findings.append(f"{here}: depth {depth}; flatten or prefix")
        if ftype == "boolean" and name.startswith(("is_", "has_")):
            findings.append(f"{here}: asserts a fact; can the source establish both values?")
        if ftype == "string" and "enum" not in field and "pattern" not in field:
            findings.append(f"{here}: unconstrained string; who acts, from what set?")
        if any(u in name for u in AMBIGUOUS_UNITS) and ftype == "number":
            findings.append(f"{here}: unit not fixed by the name")
        if any(w in desc.lower() for w in SENTINEL_WORDS):
            findings.append(f"{here}: description names a literal the type lacks")
        if "enum" in field and len(desc) < 40:
            findings.append(f"{here}: {len(field['enum'])} members, no per-member definitions")
    return findings

Run against the support-ticket schema, that emits four findings and puts next_action at the top on the unconstrained-string rule. Run against the security-triage schema it flags is_exploitable, affected_component, recommended_action's missing definitions and severity's three-word description, which is nineteen characters and well under the forty the last rule asks for.

What the review cannot see#

The checks are syntactic, and the most expensive failures in this topic were semantic. Nothing in that function knows that impact if exploited is under-defined while the highest severity whose definition the ticket satisfies is not, because both are prose of a plausible length. Nothing in it knows that duplicate belongs to a different dimension from patch, because all four members are lowercase identifiers in a list. Chapter 3's severity case would have passed a length check comfortably: the description was a full sentence with a concrete rule in it, and the rule was the problem.

The general reason a syntactic check cannot reach these is worth stating once, because it also says what would be needed to reach them. Every failure in this topic is a relation between a schema and a population of inputs rather than a property of a schema on its own. Whether one enum member's trigger swallows its siblings depends on what fraction of your tickets satisfy it. Whether a boolean is answerable depends on what your documents establish. Whether a vocabulary is complete depends on what arrives. A function holding only the schema holds one half of each of those relations, so it can flag the shapes that are suspicious in any population and none of the ones that are wrong in yours. That is the boundary, and it is also the argument for automating everything on the near side of it: those checks are the half that comes free, and spending review time on them is spending it on the cheap half.

The coverage question is the one that stays open. Whether your eight-member vocabulary covers 91 or 97 percent of real inputs cannot be derived from the schema, cannot be derived from the model's outputs, and is the single number that determines how much silent damage the enum does. It comes from a couple of hundred hand-labelled examples and it goes stale as the input distribution moves, which makes it an ongoing cost rather than a one-time review item. That is the honest state of schema design for models right now: the structural failures are predictable from the schema and largely preventable, and the semantic ones need labelled data that most teams do not have.

What the review does buy is a ranked list produced before anything runs, which turns an evaluation from "sample 300 responses and look for problems" into "check these three fields first". The next thing to do with it is a schema in your own repository — pick the extraction call that most often produces a support ticket, run the four questions over its fields by hand, write down a predicted failure rate for the worst one, and then label fifty inputs to find out whether the prediction holds. Either result is worth having: a confirmed prediction gives you a repair with a known payoff, and a wrong prediction tells you the field you were not worried about is the one carrying the risk.