Benchmarking two synthesisers on scripts that can actually fail

The bake-off memo was one page. Provider A: time to first audio 220 ms, $0.06 per 1,000 characters. Provider B: 250 ms, $0.15 per 1,000 characters. Recommendation: B, because it sounds noticeably better, and the 30 ms difference is imperceptible.

The test behind it was forty sentences, each between six and fourteen words, all ending in a full stop, all read at the default speed. The tech lead had listened to all eighty clips through studio monitors in a quiet room, alternating A then B, knowing which was which. Every one of those choices makes the result less informative than it appears, and the script set is the one that does the most damage, because it determines what the test is even capable of detecting.

Short, clean, well-punctuated sentences are the region where every modern synthesiser is good. They contain no long-range prosody to drift, no numerals to misparse, no acronyms to spell out or fail to spell out, no names outside the training distribution, and no punctuation the text frontend has to guess about. A test built from them measures timbre preference and nothing else, which is why it produced a recommendation resting entirely on "sounds better".

Build the script set out of what production actually sends#

The scripts should come from your own logs. Sample 200 real language-model outputs from the product, keep their formatting artefacts, and then top up the categories that are rare in the sample but expensive when they fail.

That last row is the one people are most surprised by and the easiest to hit in production, because a language model that has been prompted for plain text will still emit markdown occasionally, and some synthesisers read ** as nothing while others read it as a pause or, memorably, as "asterisk asterisk".

Sixty to a hundred scripts across those categories is enough, provided you tag each clip with its category, because the aggregate score is far less useful than the per-category one. A provider that is 0.7 points worse overall and 8 points worse on numerals is a different proposition depending on whether your product reads invoice numbers.

Latency measured where your users are, under load you actually have#

Time to first audio is the metric from the first chapter, and the way a bake-off gets it wrong is by measuring one request at a time from a laptop on office wifi, with a warm connection, in the vendor's home region. Run it from the production region, over a pooled connection, at the concurrency you expect at peak, for long enough to catch a deploy on the vendor's side — an hour is a reasonable minimum.

Report three numbers per provider rather than one: p50 and p95 of time to first audio, and the steady-state real-time factor at that concurrency. The measured result on the dispatch agent's bake-off, at 40 concurrent streams from eu-west-1, was less flattering to the memo than the single-request numbers:

A 30 ms difference in median first-audio latency is not something a listener can identify inside a conversational turn where the total wait is 1,580 ms. An 860 ms difference at the 95th percentile is, and B's bursty chunk cadence with 700 ms gaps is exactly the jitter profile that forces a larger playout buffer, which costs another few hundred milliseconds on every turn. The single-number comparison in the memo hid both.

Cost per hour of audio, at your volume#

Character pricing is hard to reason about because nobody thinks in characters. Convert to cost per hour of speech, which is the unit your product generates.

WPM = 150                # conversational synthesis, measured on your own output
CHARS_PER_WORD = 5.8     # including the trailing space

def cost_per_audio_hour(price_per_1k_chars, wpm=WPM):
    chars = wpm * 60 * CHARS_PER_WORD      # 52,200 chars per hour
    return chars / 1000 * price_per_1k_chars

for name, price in [("A", 0.06), ("B", 0.15)]:
    hourly = cost_per_audio_hour(price)
    print(f"{name}: ${hourly:.2f}/audio-hour  "
          f"${hourly * 18_000:,.0f}/month at 18,000 hours")

A: $3.13/audio-hour  $56,376/month at 18,000 hours
B: $7.83/audio-hour  $140,940/month at 18,000 hours

The gap is $84,564 a month, or roughly $1.01M a year, which is not the "small at our volume" the memo assumed. It is also the number most likely to be computed wrong, because billing units differ in ways that do not appear in the price list. Some providers bill the characters you send including SSML tags, so a markup-heavy integration pays for <prosody rate="85%"> at the same rate as speech. Some bill per second of generated audio, which makes a slow-speed setting more expensive. Some apply a per-request minimum, which punishes the sentence-level chunking from chapter two — at a 100-character minimum, forty short chunks cost more than one long request for the same text. Take the character counts from your own production logs rather than from the script set, since the script set is deliberately unrepresentative in length.

Intelligibility has an instrument#

Whether the audio can be understood is measurable without asking anyone, by running the synthesised speech back through the recogniser from the previous topic and scoring word error rate against the source text. This is the second use of that trick, after the prosody grid in chapter two, and here it is doing comparison rather than regression detection.

import jiwer
from normalise import expand_numbers  # your own text normaliser

transform = jiwer.Compose([
    jiwer.ToLowerCase(),
    jiwer.RemovePunctuation(),
    jiwer.RemoveMultipleSpaces(),
    jiwer.Strip(),
])

def score(provider, scripts, asr, telephony=True):
    rows = []
    for s in scripts:
        audio = provider.synthesise(s.text)
        if telephony:
            audio = to_g711_ulaw_8k(audio)   # match the channel users hear
        hyp = asr.transcribe(audio)
        wer = jiwer.wer(transform(expand_numbers(s.text)),
                        transform(expand_numbers(hyp)))
        rows.append({"category": s.category, "wer": wer, "text": s.text})
    return rows

The telephony line matters more than it looks. If your users hear the agent over a phone call, the audio is resampled to 8 kHz and encoded with G.711 μ-law — the eight-bit narrowband codec the telephone network has carried voice on for decades — before it reaches their ear, which flattens exactly the high-frequency detail that makes one voice sound crisper than another in a studio. Benchmarking on 24 kHz PCM when you ship over μ-law compares two things neither of your users will ever hear.

Per-category results are where this stops being a formality:

Word error rate scores the pair of systems, not the synthesiser alone, so a recogniser that has trouble with one voice's timbre will penalise it unfairly. The check is cheap: pull the forty clips where the two providers disagree most and have a person transcribe them. If the human transcripts agree with the recogniser's ranking, the proxy is holding.

Naturalness needs raters, blinding and enough of both#

Naturalness is a preference and has to be measured as one. A blind, randomised, multi-rater side-by-side test is the workable format: play clip A and clip B of the same script in random order, ask which sounds more natural, and collect enough judgments that the result is not noise.

The sample size is calculable. To detect a genuine 60/40 preference against the 50/50 null, at a 5% two-sided significance level with 80% power — an 80% chance of the test finding the preference if it is genuinely there — you need roughly 194 paired judgments. Twenty raters times ten clips each gets you there. Anything materially smaller produces a confidence interval that spans 50%, which means you have not measured a preference and should say so rather than reporting the point estimate.

Two confounds will decide the result for you if you let them. Loudness is the larger one, because louder audio is reliably rated as better quality, and providers ship at different default levels — normalise every clip to −16 LUFS before rating. Order is the other, so randomise which provider plays first and counterbalance across raters. Neither of these is optional, and both are ten minutes of work.

The dispatch bake-off run this way returned 54% preferring B, with a 95% confidence interval of 47% to 61%. That interval includes 50%, so the honest reading is that the naturalness difference the memo was built on did not replicate once the clips were loudness-matched and the raters could not see which was which. The original impression was a single person, unblinded, invested in a recommendation, listening to unnormalised audio on equipment that no user has — a data point of one, collected by someone who wanted an answer.

Deciding, and the part that changed the decision#

Read straight, that table says A on everything except the one axis where A is badly beaten, and the numerals gap is the one that would make a support agent unusable. What resolves it is that the gap has a cause: A's text frontend was reading $1,204.50 as "one thousand two hundred four point five zero dollars" about a third of the time and mangling 03/11/2026 into a day-first reading. Both are fixed with the <say-as> markup from chapter two, emitted by the language model as structured fields rather than left to the synthesiser to guess. Re-running the numerals category with markup brought A to 4.1% against B's 3.9%, a difference well inside the noise of the measurement.

So the choice is A, and the reason to trust it is that every axis has a number with a method behind it, including the axis where the answer is "we could not tell". The parts of the test that would make it worthless if they were missing are all things you can list: unblinded raters, unnormalised loudness, scripts from the wrong distribution, a single-stream latency measurement, cost computed on the price list instead of your own character counts, and studio-quality audio for a telephony product.

Keep the script set in the repository next to the code, tagged by category, with the expected word error rates checked in as a baseline. Vendors ship new model versions without telling you, and a bake-off is a snapshot with a short half-life — re-running it quarterly and on every announced model change turns it into a regression suite. What you ship on top of it is the rest of this topic: an adaptive playout buffer sized by observed underruns, clause-level chunking with lookahead context so the pipeline can start early without flattening the intonation, per-term turn instrumentation reported at the tail rather than the mean, and a voice that has a verification statement behind it. The first turn a real user hears will still surprise you, which is why the instrumentation ships with the feature rather than after it.