The words a decision hangs on are not weighted like the rest

The scheduling product from chapter one shipped a voice agent that took appointment changes end to end: the caller spoke, the recogniser produced text, an LLM extracted the requested date, time and clinic and called a booking tool. Transcription measured 8.4% WER on a properly stratified set of the company's own telephony, which is a defensible number for 8kHz audio. Support tickets said the agent booked the wrong slot on something close to one call in five, and the first three engineering theories were all about the LLM: the extraction prompt was too loose, the model needed a better date parser, someone should try a bigger model.

Both numbers were right, and they were measuring different things. The 8.4% counts words, and the booking depends on four of them.

Errors do not land uniformly, and the tokens that carry the task attract them#

Sorting the errors on that test set by the grammatical role of the reference word gives a distribution that the aggregate hides completely. One piece of vocabulary is needed to read it: a word is slot-bearing when it supplies one of the named values the task has to fill before it can act, a term borrowed from dialogue systems, where this booking has a date slot, a time slot and a clinic slot waiting to be filled from what the caller said.

Weighted by share, those class rates reconstruct the 8.4% on the dashboard. Read as a product metric they say something different: the tokens that determine the outcome carry two to four times the error rate of the corpus average, and they do so for reasons that are structural rather than accidental. Digit sequences and dates have weak language-model support because the fourth, the fourteenth and the fortieth are all grammatical in the same slot, so the decoder gets little help from context and must rely on acoustics that narrowband telephony has degraded. Proper names are worse still, since a surname the model has never seen has essentially no prior, and the decoder will substitute the nearest phonetically plausible word that does exist in its vocabulary. Negations are short, unstressed, and frequently reduced in fast speech to a schwa, the neutral uh vowel that unstressed English syllables collapse towards, so I can pay that and I can't pay that differ by a burst of about 40 milliseconds.

Compose those rates into a booking and the ticket volume stops being mysterious. A typical request in this product contains three slot tokens — a date word, an hour and a minute — plus a clinic name in about half of calls. At 21% per date-or-time token, the probability that all three survive intact is roughly 0.79³ ≈ 49%, which would imply half of all bookings are wrong. They are not, because a great many of the corrupted values cannot be booked: the fortieth of March fails a date parser, nine seventy fails a time parser, and a clinic name that matches nothing in the database triggers a clarifying question. Validity checks catch most of the damage.

What they cannot catch is an error that lands inside the valid space. The fourteenth heard as the fourth is a real date, in range, on a weekday, with availability. Nine forty heard as nine fourteen is a real time. Two hundred heard as two thousand is a real payment amount. Somewhere between a fifth and a third of slot corruptions produce a value that is well-formed and wrong, and those pass every downstream check and become confirmed bookings. Multiply that through and one call in five is exactly the order of magnitude you should expect from an 8.4% recogniser. Nothing about the LLM has to be broken for this to happen; the information was destroyed before the LLM was called.

This is why tightening the extraction prompt is a weak intervention. A language model can repair implausible errors, because the fortieth of March is recoverably wrong and a well-prompted model will ask. It cannot repair plausible ones, because there is nothing in the text to indicate that the fourth was ever anything else. The improvements available at the prompt layer are concentrated on the errors your validity checks were already catching.

Metrics that count the tokens the task depends on#

WER stays, because it is cheap, comparable across time, and the fastest tripwire for a model regression. Alongside it, score the entities.

Entity error rate applies the same alignment idea to normalised entity spans instead of words. Extract entities from the reference and the hypothesis with the same extractor, normalise them to canonical form so that nine forty and 9:40 compare equal, and count substitutions, deletions and insertions over entity slots rather than words.

from dataclasses import dataclass

@dataclass(frozen=True)
class Slot:
    kind: str      # "date" | "time" | "clinic" | "polarity"
    value: str     # canonical: "2026-03-14", "09:40", "chatham-street", "affirm"

def slot_error_rate(ref: list[Slot], hyp: list[Slot]) -> dict[str, float]:
    ref_by_kind = {s.kind: s for s in ref}
    hyp_by_kind = {s.kind: s for s in hyp}
    sub = sum(1 for k, s in ref_by_kind.items()
              if k in hyp_by_kind and hyp_by_kind[k].value != s.value)
    dele = sum(1 for k in ref_by_kind if k not in hyp_by_kind)
    ins = sum(1 for k in hyp_by_kind if k not in ref_by_kind)
    n = max(len(ref_by_kind), 1)
    return {"SER": (sub + dele + ins) / n, "sub": sub, "del": dele, "ins": ins}

reference = [Slot("date", "2026-03-14"), Slot("time", "09:40"),
             Slot("clinic", "chatham-street")]
hypothesis = [Slot("date", "2026-03-04"), Slot("time", "09:40"),
              Slot("clinic", "chatham-street")]

print(slot_error_rate(reference, hypothesis))
# {'SER': 0.333, 'sub': 1, 'del': 0, 'ins': 0}

That utterance scored 5.9% WER — one substitution in seventeen words — and 33% slot error rate. Both numbers are correct, and only one of them predicts a support ticket. Track three variants of this in production dashboards: entity error rate over all slot types, keyword recall over a critical lexicon of the terms your business cannot afford to lose, and a polarity flip rate counting how often an affirmative reference became a negative hypothesis or the reverse. The polarity metric is small and boring most of the time, and it is the one that catches a model update that has started swallowing unstressed not.

The extractor has to be identical on both sides, which is what makes this honest. If you extract entities from the reference by hand and from the hypothesis with an LLM, you are measuring the extractor and the recogniser together and cannot separate them.

The per-class table at the top of this chapter came out of the alignment operations stored in chapter one, not out of a separate experiment. Once every scored utterance has its substitutions, deletions and insertions written down as reference-word and hypothesis-word pairs, the class breakdown is a tagging pass and a group-by, and the ranked list of confusion pairs falls out of the same table. The pairs that appear at the top are rarely surprising in retrospect and are almost never what an engineer would have guessed in advance: at the scheduling company the most frequent substitution across 19,000 reference words was fifteen → fifty, at 41 occurrences, followed by can't → can at 28. Both are cheap to enumerate and expensive to overlook, and both become injection operators below.

An obvious response to all of this is to stop weighting words equally and compute a weighted WER, charging five or ten times as much for a slot-bearing token. It is easy to implement and it does track the product better than the flat number. The reason to keep the two metrics separate anyway is that a weighted WER is no longer comparable to anything — not to a vendor's figure, not to your own number from before the weights changed, and not to another team's. Report the flat WER for comparability, the entity metrics for the product, and resist the urge to fuse them into one score that answers neither question.

Injecting errors on purpose to see what moves#

Slot error rate tells you which values were corrupted. It does not tell you what the agent then did, and the relationship between the two is not obvious — some corruptions are absorbed by a confirmation turn, some are caught by a database lookup, some go straight into a tool call. The way to find out is to hold the recogniser constant and perturb the transcript, then run the real agent on the perturbed text and compare the tool calls it emits against the ones it emitted on the clean reference.

Start with the ceiling. Run the agent on the hand-made references from chapter two and record the tool calls; that is the best the agent can do with perfect transcription, and any gap between it and production that you were blaming on ASR is actually the agent's own extraction error. At the scheduling company the ceiling was 96.1% correct bookings on perfect text, which meant roughly four points of the failure rate belonged to the agent and the rest belonged to the audio.

Then perturb, using operators drawn from the confusion pairs your own error analysis produced rather than random noise:

import random

DIGIT_CONFUSIONS = {          # observed in the 8kHz error analysis
    "fourteenth": ["fortieth", "fourth"],
    "fifteen": ["fifty", "sixteen"],
    "nine forty": ["nine fourteen", "nine forty five"],
    "two hundred": ["two thousand"],
}
NEGATIONS = ["not", "no", "don't", "can't", "won't", "never"]

def drop_negation(text: str, rng: random.Random) -> str:
    words = text.split()
    hits = [i for i, w in enumerate(words) if w.lower().strip(",.") in NEGATIONS]
    if not hits:
        return text
    del words[rng.choice(hits)]
    return " ".join(words)

def confuse_number(text: str, rng: random.Random) -> str:
    for surface, alts in DIGIT_CONFUSIONS.items():
        if surface in text:
            return text.replace(surface, rng.choice(alts), 1)
    return text

def merge_turn_boundary(turns: list[str], rng: random.Random) -> list[str]:
    """Simulate a lost boundary: two speakers' words arrive as one turn."""
    if len(turns) < 2:
        return turns
    i = rng.randrange(len(turns) - 1)
    return turns[:i] + [turns[i] + " " + turns[i + 1]] + turns[i + 2:]

Sweep each operator at a controlled rate, run the agent, and diff the resulting tool call — name and arguments — against the golden call, meaning the one the same agent emitted on the uncorrupted reference text, which is the only thing you can legitimately compare against because it already contains the agent's own extraction mistakes. The output is a table that says what a class of error costs, which is the thing you can actually act on.

The last column is the one that decides where engineering effort goes. Proper name errors are frequent and mostly harmless here because the agent fuzzy-matches clinic names against a table of thirty-one real clinics and asks when the match is weak. Dropped negations are rare and almost never detected, so nearly every one becomes a wrong action. Merged turn boundaries — the agent's own confirmation text arriving inside the caller's turn — are both common and quiet, and they were not visible in any word-level metric at all, which is why the same failure will reappear from a different direction in chapter five.

Where this stops being the right lens#

None of this argues for retiring WER. It is the metric that will tell you a vendor's model update degraded your audio overnight, and it needs no task-specific machinery to compute, so it belongs on the dashboard beside the entity metrics rather than underneath them. Entity and slot metrics are narrow by construction: they measure only the values you thought to enumerate, and they are blind to a transcript that is fluent, entity-perfect and wrong about the caller's actual intent.

Fault injection has a matching limitation, which is that it can only simulate error patterns you have already observed. The operators above were built from a confusion analysis of real output, so they reproduce the failures of that recogniser on that channel; a different vendor mangles different things, and a synthetic sweep will happily reassure you about a failure mode your model does not have while missing the one it does. Refresh the operator list whenever the error analysis is refreshed.

The larger assumption in this chapter is that a transcript exists to be measured. Everything so far — the alignment, the test set, the entity scoring, the injection sweep — operates on finished text, and a batch pipeline transcribing yesterday's recordings can afford that. A live voice agent cannot, because it has to decide what to do while the sentence is still arriving.