Splitting the failure counter

An extraction pipeline that reports one number for structured-output failures can tell you that something is wrong and nothing else. The four classes an individual response can be sorted into — a parse failure, a schema failure, a refusal, a truncation — are already distinguishable on a single response from the stop reason, the validator's error path and whether a complete JSON value was found. Counting them apart over a population is what turns them into a diagnosis.

That is a cheap change with an uneven payoff. It reduces the failure rate by exactly zero documents, costs some code and some metric cardinality, and returns nothing at all on a week when nothing has broken. Its whole value lands on the morning a number moves, and what it buys then is the ability to rule three hypotheses out before touching anything.

What each of the four numbers rules out#

Each row is a different afternoon's work, and the exclusions are as useful as the inclusions. A rise concentrated in schema says nothing about the prompt, because a prompt that had stopped producing parseable output would show in parse first. A rise concentrated in truncation says nothing about the model's instruction following, because the grammar of the response was fine right up to the ceiling. A rise in refusal cannot be a formatting problem, because the model produced exactly the response it intended to.

The counters also need dimensions, because the class alone tells you what broke and not which change broke it. Four labels cover most investigations:

def record_outcome(
    outcome: str,          # ok | parse | schema | refusal | truncated | provider_error
    *,
    provider: str,
    model: str,
    prompt_version: str,
    schema_version: int,
    detail: str | None = None,   # bounded: field path, or rejection reason
) -> None:
    metrics.increment(
        "extraction.outcome",
        tags=[
            f"outcome:{outcome}",
            f"provider:{provider}",
            f"model:{model}",
            f"prompt_version:{prompt_version}",
            f"schema_version:{schema_version}",
        ],
    )
    if detail is not None:
        metrics.increment(f"extraction.detail", tags=[f"outcome:{outcome}", f"detail:{detail}"])

Cardinality is the constraint on that label set, meaning the number of distinct time series the labels produce, since a metrics backend stores and bills for one series per combination of label values. It is easy to get right if you check the arithmetic before shipping. Six outcomes, three providers, six live prompt versions and three schema versions is 324 time series, which any metrics backend handles without noticing. Adding supplier_id, with 17,000 distinct values, makes it 5.5 million series and will either cost a fortune or get your metrics dropped at the agent. Supplier belongs in the log line and in the stored sample, and the query that groups failures by supplier is a log query rather than a metrics query.

The operation there is a multiplication rather than an addition, and that is the whole reason cardinality surprises people. A series exists for every combination of label values, so six outcomes times three providers is eighteen, times six live prompt versions is a hundred and eight, times three schema versions is the 324. Adding a label does not add its values to the total. It multiplies the total by them. That is how 17,000 suppliers turn 324 into 5.5 million, and it is why trimming the label's values is never the fix — a supplier list cut to a tenth of its length still multiplies. The fix is to move the label somewhere the storage model is one record per event rather than one series per combination, which is what a log line is.

detail is worth carrying separately and worth keeping bounded. Validator error types are a small set — missing, decimal_type, extra_forbidden, literal_error — and so are field paths for a fixed schema. Concatenating the two gives something like total.decimal_type, which is a few dozen values and is the single most useful string in an incident, because it turns "schema failures are up" into "schema failures are up and they are all issue_date.string_pattern_mismatch".

Keeping the evidence without keeping the PII#

A counter tells you what class of failure rose. Reproducing the failure needs the response that caused it, and this is the part that gets deferred, because the responses contain the contents of customer invoices: names, addresses, bank details, occasionally an entire medical document that was misfiled into the stream. That is personally identifiable information, usually written PII: data that can be tied back to a named individual, and therefore data that a debug bucket has to answer for in the same way the production database does.

The first move is to notice how much of the problem is self-inflicted. You almost certainly already store the source documents somewhere with a retention policy, an access-control model and an audit log that legal has already looked at. Storing the model's input again in a debug bucket creates a second copy of regulated data under weaker controls, which is the thing to avoid; storing a reference to the document you already have costs nothing and preserves reproducibility, since the input can be re-fetched by anyone who is entitled to see it.

That leaves the response, and what you need from the response depends on the class. A parse failure or a truncation is a question about structure — where the object started, whether it closed, what the surrounding text looked like — and the values are irrelevant. A skeleton is enough:

import json

def skeletonise(value: object) -> object:
    """Preserve shape and types; discard every leaf value."""
    if isinstance(value, dict):
        return {k: skeletonise(v) for k, v in value.items()}
    if isinstance(value, list):
        return [skeletonise(v) for v in value[:3]] + (
            [f"<+{len(value) - 3} more>"] if len(value) > 3 else []
        )
    if isinstance(value, str):
        return f"<str:{len(value)}>"
    if isinstance(value, bool) or value is None:
        return value
    return f"<{type(value).__name__}>"

Applied to a raw response that failed to parse, the same treatment runs over the text rather than a parsed object: keep the first and last 200 characters, replace every run of digits with #, and replace anything matching an email, IBAN or card pattern with a token. What survives is enough to see that the response began with a sentence of preamble, or that it stopped mid-string at character 4,096, which is the whole question.

Schema failures need slightly more, because the offending value is the evidence. Storing the value's shape — its length, its character classes, whether it matched a numeric pattern — resolves most cases without storing the value itself. total rejected with decimal_type, length 8, matching ^[0-9,]+\.[0-9]{2}$ is a complete diagnosis of a comma-separator incident, and it contains no customer data at all.

Refusals are the class where the actual text matters most, since the decline explains itself and no derived signal substitutes for reading twenty of them. Refusal texts are also model-authored prose rather than document contents, which makes them lower-risk than they look, with the important exception that a model explaining why it declined will sometimes quote the document. The workable arrangement is a short TTL, a time-to-live after which the stored object is deleted automatically — fourteen days is enough for any investigation that is going to happen — a restricted bucket with access logging, and a redactor — the step that finds identifying spans in the text and replaces them with placeholders — that runs before the write rather than after.

FIXTURES = [
    ("truncated_line_items.json", Outcome.TRUNCATED),
    ("fenced_with_preamble.json", Outcome.PARSE),
    ("comma_in_total.json",       Outcome.SCHEMA),
    ("declined_discharge.json",   Outcome.REFUSAL),
]

@pytest.mark.parametrize("filename,expected", FIXTURES)
def test_captured_response_classifies(filename, expected):
    raw = (FIXTURE_DIR / filename).read_text()
    assert classify(json.loads(raw)) == expected

Four files in a directory is what it takes to reproduce each failure class from a captured raw response, and the value of it is not the assertion. It is that the next person to change the parser, the validator, or the refusal heuristic finds out immediately which of the four classes they broke, using responses that a real provider actually produced rather than ones someone invented while writing the test.

Where the split is a bad investment#

Four counters instead of one does not reduce the failure rate by a single document. On a week where nothing has broken, the split costs cardinality, some code, and a slightly busier dashboard, and returns nothing. Its entire value is realised on the morning the number moves, which for a mature service is a handful of mornings a year.

That makes it a genuinely bad investment at small volume. A pipeline processing 500 documents a day with a 4% failure rate produces twenty failures, and reading twenty raw responses over coffee is faster and more informative than any dashboard, because you see the actual text. The threshold where the split starts paying is roughly the point where nobody can read all the failures by hand any more, which lands somewhere around a few hundred failures a day, and it arrives earlier if there is more than one provider or more than one prompt in flight, since those are the dimensions that make eyeballing ambiguous.

The four counters are also not a root cause. They narrow four hypotheses to one and hand you a place to look; the actual answer is always in a response, a diff, or a provider changelog. A dashboard that says truncation does not tell you that an eleven-day-old prompt edit added a justification paragraph, and someone still has to make that connection.

What the split settles is which lever to reach for, and it settles it before any money is spent rather than after. Three of the four classes point at some form of trying again — truncation retried with a larger budget can work, a parse failure retried can work, a schema failure retried with the error fed back can work — and one of them, the refusal, points away from it entirely.

Each of those retries has a cost, a latency, and a cap that somebody has to choose. Choosing that cap by watching a recovery-rate number go up is how a pipeline ends up spending its p99 budget on the documents least likely to be recovered.