A vendor's 4% describes the vendor's audio, not yours

A procurement bake-off at a debt-collection call centre ran three transcription vendors against forty of the company's own recorded calls. All three vendors published a headline figure between 3% and 5% on their marketing pages. Measured on the forty calls, with the same normaliser applied to all three so the comparison was fair, they came in at 19.3%, 17.8% and 21.1%. The room's first theory was that the vendors were inflating their numbers, and the second was that the scoring harness was broken. Neither was true, and the gap between 4% and 19% is worth taking apart carefully, because almost all of it is explainable and none of it is going away.

Where a 4% comes from#

The number on the landing page is almost always measured on a public read-speech corpus, and LibriSpeech test-clean is the usual one. It is built from LibriVox audiobook recordings: volunteers reading published prose aloud, alone, into a decent microphone, in a quiet room, at 16kHz, from a script they can see. The speakers do not stumble, do not talk over each other, do not trail off, and do not say their own surname. The reference transcript is the book, so it is exact by construction. State-of-the-art systems sit around 2% there, and a vendor quoting 4% is quoting something in that family.

Your audio shares almost none of those properties. A call arriving over a SIP trunk — the connection your telephony provider gives you into the PSTN, the ordinary public switched telephone network — has been sampled at 8kHz and squeezed through a codec, a compression scheme that trades fidelity for bandwidth, typically G.711 or, on a mobile leg, something more aggressive like AMR-NB at 12.2kbit/s. There is a car, a television, an open-plan office or a child in the background. Two people interrupt each other several times a minute. Nobody is reading. The words that matter most — surnames, street names, account numbers, product codes — are exactly the words no language model has strong priors for.

The single largest contributor, and the one engineers underestimate, is bandwidth. Narrowband telephony carries roughly 300Hz to 3.4kHz, against the 50Hz to 8kHz that wideband audio sampled at 16kHz preserves, and the energy that distinguishes the unvoiced fricatives — the consonants made by forcing air through a narrow gap rather than by vibrating the vocal folds — lives above that: /s/ has most of its energy between 4kHz and 8kHz, /f/ and /θ/ are broadband and weak. Remove that band and six and fix converge, S and F become nearly the same sound when a caller spells a reference number, and fifteen and sixteen differ mostly in a region the codec discarded. No amount of model scale recovers information that was never transmitted. Published comparisons of the same model on wideband versus downsampled narrowband versions of the same recordings typically show relative WER increases of 25% to 50% from this factor alone.

Two of those rows need a word of unpacking. Spontaneous conversation differs from read speech chiefly in disfluency — the ums, restarts, abandoned clauses and repaired words that everyone produces when composing sentences in real time and nobody produces when reading from a page — and the IVR bleed on the noisy telephony row is the recorded prompts of the interactive voice response menu, the "press one for billing" system, still audible on the line while a human is speaking over them. Read down the table and the 4% and the 19% stop looking like a contradiction. They are two rows, and your product lives on the fifth one. The vendor is not lying, and the model is not underperforming; the measurement was made on audio that has been stripped of everything that makes your audio hard.

The first thing worth doing is confirming what your audio actually is, because pipelines lie about this constantly. Audio that arrives as an 8kHz PSTN leg and gets resampled to 16kHz WAV somewhere in the recording stack is still narrowband — the file header says 16kHz, the spectrum above 3.4kHz is empty, and a recogniser that would have handled genuine wideband gets no benefit.

# What the container claims.
ffprobe -v error -select_streams a:0 \
  -show_entries stream=codec_name,sample_rate,channels,bit_rate \
  -of default=noprint_wrappers=1 call_88214.wav
# codec_name=pcm_s16le
# sample_rate=16000
# channels=1

# What the signal actually contains: mean energy in the 4-8kHz band.
ffmpeg -v error -i call_88214.wav -af "highpass=f=4000,astats=metadata=1:reset=0" \
  -f null - 2>&1 | grep -i "RMS level"
# RMS level dB: -78.4     <- effectively silent: this is upsampled narrowband

An RMS of −78dB above 4kHz on a file labelled 16kHz means the extra sample rate is decoration. Knowing that changes which vendor tier you should be evaluating and stops you paying for a wideband model whose advantage cannot reach you.

Two hours of hand transcription is the entry fee#

Nothing above tells you your own number. To get one you need references for your own audio, which means somebody sits down with the recordings and types what was said. This is the step teams skip, and skipping it means every subsequent decision — vendor choice, model upgrade, vocabulary tuning, the decision to ship — is made against a number that describes an audiobook.

Sample stratified rather than at random, because a uniform sample of production calls will be dominated by whatever your median call is and will under-represent the conditions that break things. At the collection call centre the strata that mattered were inbound versus outbound, mobile versus landline leg, the three largest regional accent groups, and calls containing a payment amount. Twenty-five to forty utterances from each cell, drawn from different days, gives a set that can answer questions rather than produce a single average.

Size follows from the precision you need. WER's uncertainty comes from the reference word count and, more importantly, from correlation within utterances, so binomial standard errors computed on word counts are optimistic. Bootstrap over utterances instead — draw a new sample of the same size from your utterances with replacement, several thousand times over, score each resample, and read the confidence interval off the spread of the results, which needs no assumption about how the errors are distributed:

import numpy as np

def bootstrap_wer(scores: list[tuple[int, int]], n: int = 5000, seed: int = 0):
    """scores: list of (errors, ref_words) per utterance."""
    rng = np.random.default_rng(seed)
    errs = np.array([e for e, _ in scores], dtype=float)
    refs = np.array([r for _, r in scores], dtype=float)
    idx = rng.integers(0, len(scores), size=(n, len(scores)))
    samples = errs[idx].sum(axis=1) / refs[idx].sum(axis=1)
    return errs.sum() / refs.sum(), np.percentile(samples, [2.5, 97.5])

point, (lo, hi) = bootstrap_wer(utterance_scores)
print(f"WER {point:.1%}  95% CI [{lo:.1%}, {hi:.1%}]")
# 200 utterances / ~2,400 ref words:  WER 19.3%  95% CI [17.5%, 21.2%]
# 1,600 utterances / ~19,000 words:   WER 19.1%  95% CI [18.4%, 19.8%]

Those two lines set expectations. Two hundred utterances, roughly twenty minutes of speech, resolves a difference of about four percentage points, which is enough to separate two vendors that are genuinely far apart and not enough to detect a model upgrade that buys you one point. Around two hours of speech, some 19,000 reference words, gets the interval down to about ±0.7pp, which is the resolution at which routine regression testing becomes meaningful.

The cost of that is real. Careful verbatim transcription from scratch runs at five to eight times realtime for an experienced transcriber on clean audio, and worse on crosstalk; two hours of audio is therefore ten to sixteen hours of work. Correcting a machine transcript instead of typing from scratch cuts it to roughly two to three times realtime, so four to six hours, and that is the approach almost everyone should take. It carries one specific hazard: a corrector reading a plausible transcript while listening to degraded audio will accept errors that sound approximately right, and the errors most likely to survive are the confusable digits and names you care about most. Two independent passes on a 10% subsample gives you an inter-annotator disagreement rate, and that rate is the floor of your measurement — if two humans disagree on 1.4% of words, no WER you report below about 1.4% is distinguishable from annotation noise.

Write down a transcription style guide before anyone starts, covering numbers, disfluencies, partial words, and what to do with unintelligible audio, then apply the same normaliser from the previous chapter at scoring time. The guide only has to be internally consistent; it does not have to match the vendor's conventions, because normalisation is where those get reconciled.

One design decision inside the test set is worth getting right the first time. It is tempting to cut the calls into short utterances, transcribe those, and score each in isolation, because it makes the harness simple and the bootstrap above assumes utterance units. The trap is that whoever cuts the audio is making segmentation decisions that the recogniser would otherwise have made itself, and clean segment boundaries hand the model an easier problem than production gives it. Transcribe whole calls, keep the reference as a continuous stream of words, and score long-form against long-form; derive utterance units for the bootstrap from the reference's own sentence boundaries rather than from cuts in the audio.

What you can do about the gap, and what you cannot#

Knowing that most of the 19% comes from the channel rather than the model reorders your options, and several of them are cheaper than switching vendors. If any portion of your traffic arrives over WebRTC — the browser and mobile standard for carrying real-time audio over the internet rather than over the telephone network — that leg is encoded with Opus at 16kHz or better, and recording it before it is bridged down to the PSTN preserves the band that carries the fricatives; at the collection call centre, app-originated calls scored 12.4% against 19.3% for the PSTN legs on the same model. Where recording happens matters for the same reason — a stack that captures audio after mixing, transcoding and automatic gain control is scoring a signal that has been through three lossy stages, and moving the tap earlier is free accuracy.

Two interventions that feel obviously helpful are worth testing rather than assuming. Speech enhancement and noise suppression are tuned to make audio sound better to human ears, and the artefacts they introduce — spectral holes, musical noise, transient smearing — can raise WER even as the recording becomes more pleasant to listen to; measure before and after on your own set rather than enabling it by default. Upsampling narrowband audio to 16kHz before submitting it does not recover anything and sometimes costs a little, because it hands the model a spectral profile unlike anything in its training distribution; if a vendor offers a narrowband-specific endpoint or model variant, that variant has usually seen 8kHz telephony in training and is the better choice.

What none of this fixes is overlap. When two people speak at once, a single mixed channel contains a sum, and the recogniser must either pick one speaker or produce something that is neither. Roughly 8% to 15% of speech time in two-party phone conversations is overlapped, which puts a hard floor under any single-channel measurement of a conversation. There is an escape from that floor when the recording infrastructure cooperates, and it is one of the subjects of chapter five.

What the test set does not buy you#

It is not evidence that vendors mislead, and treating it that way costs you information. Public benchmarks rank systems, and rankings transfer partially: a model that is clearly better on test-other is usually at least not worse on your telephony, so benchmarks remain a reasonable way to decide which three systems to bother evaluating. What they cannot do is predict your absolute number, and absolute numbers are what capacity plans, SLAs and go/no-go decisions are built from.

A representative test set is also not a training set. The moment you start choosing vocabulary hints, prompt text or model checkpoints by watching this number, it becomes a development set and stops being an unbiased estimate. Hold out a second sample, drawn the same way, that you look at only when you are deciding whether to ship. And do not build the set from complaints or failed calls, which is the tempting shortcut, because a set drawn from known failures measures your failure population and cannot tell you the fleet-wide rate; keep those calls as a separate diagnostic corpus, clearly labelled.

Finally, the set decays. A telephony migration that changes codec, a new IVR prompt that changes what callers say first, an expansion into a new region, or a change to the recording stack's automatic gain control all shift the distribution the set was drawn to represent. Re-sampling a fresh few hundred utterances each quarter, and comparing the old and new sets under the same model, tells you whether your audio has moved even when your model has not.

There is one thing the collection call centre discovered after all of this work that no amount of test-set hygiene addressed. They had a defensible, well-measured 19% on their own audio, and their downstream agent still got payment amounts wrong far more often than 19% of the time. The next chapter is about why a correctly measured aggregate can still be the wrong thing to look at.