What an 8% word error rate counts, and what it can exceed

A nightly job at a healthcare scheduling company sampled 500 calls, ran them through the transcription vendor, compared the output against a stored reference set, and posted one number to a dashboard. For four months that number sat between 7.6% and 8.4%, which everybody had agreed was fine. Then one Tuesday it read 118%, and the engineer on call spent an hour convinced the scoring script had a division bug, because a percentage of words that were wrong could not exceed the number of words that were said.

That nightly job is the ordinary shape of a production speech pipeline, so it is worth naming its parts before taking the number apart. Speech to text, also called automatic speech recognition and abbreviated ASR, is the conversion of an audio signal of somebody talking into the words they said, in order, as a string. It exists because nearly everything that wants to act on a human voice — a search index, a compliance rule, a CRM field, a language model deciding which tool to call — operates on text, and a waveform is not something any of them can read. The output is already familiar even where the machinery is not: the live captions running under a video call, the transcript panel beside a recorded meeting, the sentence a voice assistant repeats back before it acts, the searchable text a support platform attaches to every recorded call. In each of those a model has turned pressure variations in air into tokens, and something downstream has then made a decision on the strength of them.

Measuring whether the model is any good at that is what the 500 sampled calls were for, and the procedure is the same everywhere. Take audio for which a person has written down by hand what was actually said — the reference, sometimes called the ground truth — send the same audio through the recogniser, and compare the two strings. Word error rate, universally shortened to WER, is what that comparison produces: in plain terms, the share of words the machine got wrong, measured against how many words were really spoken. It is the figure on every ASR vendor's landing page, the figure research papers compete on, and the figure the dashboard had been showing for four months. Almost everything in this topic is an argument with it, so the arithmetic underneath it has to be exact, and the 118% is where that arithmetic stops matching intuition.

The script was correct. One call in the sample had a 94-second stretch where the caller put the phone down on a desk with the line still open, and the recogniser, handed a minute and a half of room tone and a distant television, emitted "thank you for watching, please subscribe" eleven times. The reference for that stretch was empty. Eleven repetitions of a five-word phrase is 55 words of transcript against 0 words of truth, and those 55 words were added to the numerator of a metric whose denominator counts only what was actually spoken. That single call moved the fleet-wide average by more than 100 percentage points, and the reason it could do that is the same reason the 8% on the other 499 calls means less than it appears to.

The alignment underneath the number#

Word error rate is computed by aligning the hypothesis — the string the recogniser produced, called that because it is the machine's best guess rather than a fact — against the reference with a minimum-edit-distance algorithm over words rather than characters, then counting the three kinds of edit the alignment used. Substitutions are positions where both strings have a word and the words differ, deletions are reference words the hypothesis skipped, and insertions are hypothesis words with no reference counterpart. The sum of the three is divided by N, the number of words in the reference, and nothing in that formula bounds the insertion count by N.

Take a real utterance from that scheduling product — one continuous stretch of speech from a single speaker, bounded by pauses rather than by grammar, which is the unit these systems and their scoring harnesses work in. The reference, transcribed by hand from the audio, is i need to move my appointment to the fourteenth of march at nine forty. The recogniser returned i need to move my appointment to the fortieth of march at nine forty five. The alignment that minimises the edit count looks like this:

One substitution, no deletions, one insertion, against a 14-word reference gives (1 + 0 + 1) / 14 = 14.3%. Note what the alignment does and does not know. It found the cheapest sequence of edits, which is not necessarily the sequence of things that went wrong acoustically — the recogniser did not "decide to insert a word", it produced a token sequence, and the alignment reverse-engineered a minimal explanation. When a recogniser splits cannot into can not, the alignment charges one substitution and one insertion, and there is no way for the metric to know that the transcript is perfectly readable.

The computation is short enough to write out, and worth writing out once so the ops are not a black box:

def align(ref: list[str], hyp: list[str]) -> list[tuple[str, str, str]]:
    """Word-level Levenshtein with backtrace. Returns (ref_word, hyp_word, op)."""
    n, m = len(ref), len(hyp)
    d = [[0] * (m + 1) for _ in range(n + 1)]
    for i in range(n + 1):
        d[i][0] = i          # delete everything
    for j in range(m + 1):
        d[0][j] = j          # insert everything

    for i in range(1, n + 1):
        for j in range(1, m + 1):
            cost = 0 if ref[i - 1] == hyp[j - 1] else 1
            d[i][j] = min(
                d[i - 1][j] + 1,        # deletion
                d[i][j - 1] + 1,        # insertion
                d[i - 1][j - 1] + cost, # substitution or match
            )

    ops, i, j = [], n, m
    while i > 0 or j > 0:
        if i > 0 and j > 0 and d[i][j] == d[i - 1][j - 1] + (ref[i - 1] != hyp[j - 1]):
            op = "correct" if ref[i - 1] == hyp[j - 1] else "substitution"
            ops.append((ref[i - 1], hyp[j - 1], op)); i, j = i - 1, j - 1
        elif j > 0 and d[i][j] == d[i][j - 1] + 1:
            ops.append(("—", hyp[j - 1], "insertion")); j -= 1
        else:
            ops.append((ref[i - 1], "—", "deletion")); i -= 1
    return list(reversed(ops))


def wer(ref: list[str], hyp: list[str]) -> float:
    counts = {"substitution": 0, "deletion": 0, "insertion": 0, "correct": 0}
    for _, _, op in align(ref, hyp):
        counts[op] += 1
    return (counts["substitution"] + counts["deletion"] + counts["insertion"]) / len(ref)

Run the hold-music call through it and the arithmetic stops being surprising. The reference for the segment is yes that works for me, five words. The hypothesis is those five words followed by thank you for watching thank you for watching thank you for watching subscribe, thirteen words with no reference counterpart. Thirteen insertions over a five-word reference is 260%. The denominator is a property of the truth, and the numerator is a property of the model's output, so the two are free to diverge without limit. This is not a pathological edge case invented to make a point: hallucinated repetition on silence, music and non-speech noise is a documented failure mode of large weakly-supervised transcription models, and the Whisper paper — Robust Speech Recognition via Large-Scale Weak Supervision — discusses both the behaviour and the long-form decoding heuristics used to suppress it.

In practice this changes how you aggregate. Averaging per-utterance WER across a corpus lets a single 260% utterance drag the mean by an amount unrelated to how much audio it represents, which is why the standard NIST scoring tools — sclite and its descendants, which have been the reference implementation since the 1990s DARPA evaluations — accumulate the error counts and the reference lengths across the whole corpus first and divide once at the end. That corpus-level ratio is the number you should report. Keep the per-utterance figures too, because the distribution is where the useful signal lives, but do not average them.

The split between the three edits is the diagnostic

Reporting only the ratio throws away the most actionable part of the computation, which is how the total divides between substitutions, deletions and insertions. Three recognisers can all sit at 8% and be broken in three different ways. A substitution-dominated profile — say 6.4% substitutions, 1.1% deletions, 0.5% insertions — is a lexical and acoustic problem: the model hears speech, decodes speech, and picks the wrong words, which points at accents, channel bandwidth or vocabulary. A deletion-dominated profile points at the voice activity detector — the component that decides which stretches of the incoming stream contain speech at all and are therefore worth passing on — rather than at the acoustic model, because the audio never reached the decoder that turns frames into words; over-aggressive endpointing that clips the first 300ms of a turn removes exactly the words that open a sentence, and those tend to be the ones that carry the intent. An insertion-dominated profile is the hallucination signature from the hold-music call, and it is the one that will not show up in a mean that has been computed over utterances instead of over words.

So store the counts, not the ratio. A row per utterance with sub, del, ins, ref_len and the audio identifier costs nothing and lets you re-aggregate any way you need later — by accent group, by trunk, by time of day, by call duration — without re-running the recogniser. The corpus number is then a sum over that table, and the interesting queries are the ones that slice it.

-- Where the 8% actually lives.
SELECT
  channel,
  SUM(sub + del + ins)::float / SUM(ref_len) AS wer,
  SUM(sub)::float / SUM(ref_len)             AS sub_rate,
  SUM(del)::float / SUM(ref_len)             AS del_rate,
  SUM(ins)::float / SUM(ref_len)             AS ins_rate,
  COUNT(*)                                   AS utterances
FROM asr_scores
WHERE run_id = '2026-03-11-nightly'
GROUP BY channel
ORDER BY wer DESC;

What the metric is quietly assuming#

Before any of the edits get counted, both strings pass through a normaliser, and that normaliser has as much influence on the reported number as the acoustic model does. The reference says fourteenth; the recogniser might emit 14th. It says nine forty; the recogniser might emit 9:40. Neither difference is an error a human would care about, but a naive string comparison charges a substitution for each. Scoring the same 500-call sample two ways at that scheduling company gave 8.1% with a normaliser that lowercased, stripped punctuation, expanded contractions and mapped digit strings to words, and 14.6% with raw strings compared as-is.

# The same audio, scored two ways.
raw_ref  = "I need to move my appointment to the 14th of March at 9:40."
raw_hyp  = "i need to move my appointment to the fortieth of march at nine forty five"

# Without normalisation: "14th" vs "fortieth", "9:40." vs "nine",
# "March." vs "march" — six edits on a 14-word reference, 42.9%.
# With normalisation (lowercase, strip punctuation, digits -> words):
#   "fourteenth" vs "fortieth" and one insertion — 14.3%.

Nothing about this is dishonest, and every vendor and every research paper does it. It does mean that a WER quoted without its normalisation scheme is not a measurement, and that comparing your 8% against a vendor's published figure is comparing two numbers that may have been produced by different rules. The Whisper repository ships its English normaliser as a separate file precisely so that comparisons can be made on equal terms, and using someone else's normaliser is the cheapest way to make your numbers mean something to someone outside your team.

The other habitual mistake is reading 8% WER as 92% accuracy, in the sense of 92% of transcripts being right. Errors are not spread evenly, one per twelve and a half words, like sultanas in a cake; they cluster on hard words and hard moments. Even under the generous assumption that they are independent and uniform, an 8% error rate over a 15-word utterance leaves 1 - 0.92^15 ≈ 71% of utterances containing at least one wrong word. Measured on real call audio the figure is usually worse than that, because the errors correlate — the same accent, the same noise burst, the same unfamiliar surname produces several errors in the same sentence. An 8% WER product is one where roughly three utterances in four have something wrong in them somewhere.

What WER does not tell you#

It does not tell you whether the transcript is usable, because every word costs one. The alignment above charged the same single substitution for fourteenth → fortieth as it would have charged for the → a, and one of those two changes the appointment date. It does not tell you whether the errors are recoverable by a downstream reader who has context. It does not tell you anything about who was speaking, or when, since it scores a flat sequence of words with no timing or speaker attached.

Given how many things it ignores, it is fair to ask why the field kept it. WER won because it is cheap, unambiguous once the normaliser is fixed, and requires no judgement from the person running it, which matters enormously when a metric has to arbitrate between competing systems submitted by competing labs. The alternatives that carry more meaning — human ratings of transcript usefulness, task-completion measures, semantic similarity scores — all require either people or a second model, both of which drift over time and cannot be re-run identically in two years. A metric that a script computes deterministically from two text files, in milliseconds, over any corpus anybody assembles, is a metric that can anchor thirty years of comparable results. That property is real and worth keeping; it just does not extend to telling you whether your product works.

It is also not the right metric for every language. Word error rate assumes words are separated by spaces, which makes it unstable for Mandarin, Japanese and Thai, where the segmenter you choose changes the denominator; character error rate is the convention there for the same reason that character-level comparison is stable when word boundaries are a modelling choice rather than an orthographic fact.

The deeper limitation is that a WER is not a property of a recogniser. It is a property of a recogniser paired with a specific set of audio and a specific set of hand-made references, and swapping either one moves the number substantially. That 8% describes 500 sampled calls from one product's telephony, transcribed by two contractors following one style guide, normalised one way. The next chapter is about what happens when you take that fact seriously and compare it against the number printed on a vendor's landing page.