Playback starts on the first chunk and stalls when the buffer drains

The dispatch assistant demoed well. Drivers could ask it where their next load was, and it answered out loud in about a second. The synthesis dashboard, meanwhile, reported a p50 of 1.9 seconds and a p95 of 4.4 seconds per response — the median, and the value that only one response in twenty exceeds — which nobody could square with the demo feeling instant. Two engineers spent a week trying to work out which number was lying.

The stage both numbers were timing is text to speech, almost always shortened to TTS: a model that takes a string of text and returns an audio waveform of a voice saying it. It exists because every other part of a voice product deals in words rather than in sound. The recogniser from the previous topic turned the driver's question into text, a language model answered in text, and something has to make that answer audible to a person whose hands are on a steering wheel. A library of pre-recorded clips cannot do the job, because the answer — a load number, a street address, a delivery window — is assembled at runtime and has never been spoken before. The same stage sits at the end of phone support agents and IVR (interactive voice response) menus, screen readers, in-car and smart-speaker assistants, audiobook and dubbing pipelines, and any accessibility feature that reads a page aloud. What matters for the dispatch team's argument is that a modern synthesiser produces the waveform incrementally rather than handing back one finished file at the end, so the audio arrives as a sequence of chunks and a player can begin on the first of them while the rest is still being generated.

Neither number was lying. The dashboard measured the wall clock from the synthesis request to the last byte of the WAV, and the demo measured how long a human waited before hearing a voice. Those are different quantities because the client did not wait for the last byte; it opened an audio sink — the queue the operating system drains at a fixed rate to feed the speaker — and pushed each chunk into it as it arrived. The first chunk landed 240 ms after the request, so the driver heard a voice at 240 ms while the dashboard was still counting to 1,900.

The same product then shipped a "read me the full manifest" feature, and support tickets started arriving about the voice stuttering. Short answers were fine. Long answers — thirty seconds of continuous speech, forty-two line items — broke apart into speech, silence, speech, silence, starting around fourteen seconds in and getting worse toward the end. The dashboard was unchanged. The p50 was still 1.9 seconds, because the p50 was computed over short answers.

Time to first audio is the metric the ear reports#

For anything conversational, the quantity that determines whether the product feels alive is the interval between the request and the first sample the user hears. Call it time to first audio, TTFA. It is the analogue of time to first token in a text stream, and it matters for the same reason: a stream that starts early gives the listener something to attend to while the rest is produced, so the perceived wait ends at the first syllable rather than at the last.

This is why "total synthesis time" is close to useless as a product metric, even though it is the number every synthesis SDK hands you for free. A provider that returns a complete 30-second file in 3.0 seconds and a provider that returns the first 200 ms chunk in 250 ms and finishes in 21 seconds will feel nothing alike, and the first one — faster by the dashboard — is the one that feels broken. Total synthesis time is still worth tracking, because it bounds your cost and your concurrency, but it is a capacity metric rather than a responsiveness metric.

TTFA has to be measured at the sink, not at the socket. There are four places the clock can stop, and they routinely differ by a hundred milliseconds or more: when the HTTP response headers arrive, when the first body byte arrives, when the first decoded PCM frame exists — PCM being the raw, uncompressed samples a speaker can play, which is what a compressed format like MP3 or Opus has to be decoded into first — and when the audio device actually renders a sample. On a WebRTC path with an Opus decoder and a 20 ms frame size you can lose 40–60 ms in the decode and jitter path alone; on a browser client using an AudioContext that has been suspended by autoplay policy you can lose an unbounded amount, because nothing renders until the user gestures. Instrument the last of the four.

import time, asyncio

async def speak(client, text, sink, prebuffer_s=0.2):
    t0 = time.perf_counter()
    first_chunk_at = None
    buffered_s = 0.0
    started = False

    async for chunk in client.stream_tts(text, output_format="pcm_24000"):
        now = time.perf_counter()
        if first_chunk_at is None:
            first_chunk_at = now - t0          # network + model latency
        buffered_s += len(chunk) / 2 / 24000   # 16-bit mono @ 24 kHz
        sink.write(chunk)
        if not started and buffered_s >= prebuffer_s:
            ttfa = sink.start()  # returns the clock at first rendered sample
            started = True
            print(f"first_chunk={first_chunk_at*1000:.0f}ms ttfa={ttfa*1000:.0f}ms")

A trace from that instrumentation on the manifest feature, with the sink logging every underrun, is where the stutter stopped being mysterious:

14:02:11.204  request sent           chars=1487
14:02:11.451  first_chunk=247ms
14:02:11.663  playback started       ttfa=459ms   buffered=0.21s
14:02:13.900  buffer=0.61s
14:02:18.900  buffer=0.36s
14:02:24.100  buffer=0.09s
14:02:25.640  UNDERRUN               silence=380ms
14:02:27.900  UNDERRUN               silence=520ms
14:02:31.220  UNDERRUN               silence=640ms
14:02:41.006  stream complete        audio=30.4s  wall=29.8s

The buffer is not oscillating around a healthy level. It descends, monotonically, from 0.61 seconds to nothing, and once it hits nothing it never recovers — the gaps get longer as the run goes on because each underrun is only patched by however little audio has accumulated during the silence.

The regime test: is steady-state generation faster than realtime?#

The quantity that explains the trace is the ratio of wall-clock time spent generating to the duration of audio generated. Speech people call it the real-time factor. An RTF of 1.2 means the synthesiser produces one second of audio every 1.2 seconds of wall clock; an RTF of 0.71 means it produces one second of audio every 0.71 seconds, which is the same thing as 1.4x realtime. Playback always consumes audio at exactly 1.0 seconds per second, so the buffer's rate of change is 1/RTF − 1 audio-seconds per wall-second, and the sign of that expression decides everything about which fix will work.

At RTF 1.2 the buffer loses 0.167 audio-seconds for every wall-second of playback. Nothing about that changes as the utterance goes on, so the deficit over an utterance of length L is L/6 seconds, and the prebuffer required to survive it is exactly that deficit. Since the prebuffer itself accumulates at the generation rate, the wall clock before the first syllable is 1.2 × L/6, or L/5.

There is no setting in that table. Every row wants a different prebuffer, the required value scales with the length of the utterance, and the last column is the honest description of what you have built: a system that waits a fifth of the utterance before it says anything. At 30 seconds that is a 6-second pause after the user stops talking, which is worse than the stutter it fixes. This is the sense in which no prebuffer rescues a sub-realtime synthesiser — not that large buffers fail to work, but that the buffer which works is a function of a length you often do not know in advance, and its value is large enough to destroy the responsiveness that streaming existed to provide.

What does work below realtime is pipelining: split the text at sentence boundaries, run more than one synthesis request concurrently, and concatenate the results in order. Two concurrent streams at RTF 1.2 each give an aggregate production rate of 1.67 audio-seconds per wall-second, an effective RTF of 0.6, which puts you in the other regime entirely. This is the fix the trace above called for, and the manifest feature got it: forty-two line items chunked into groups of three sentences, three requests in flight, buffer occupancy that climbed instead of fell.

Pipelining has a precondition that is easy to miss when you are using a hosted API and impossible to miss when you are not. It buys you throughput only if the concurrent requests actually run on separate compute. Against a vendor endpoint they usually do, and your only limit is a concurrency quota. Against a model you are self-hosting on one GPU, two concurrent streams contend for the same kernels, each individual stream slows to something near RTF 2.4, and the aggregate rate is roughly what it was — you have added complexity and gained nothing. Measure aggregate RTF under the concurrency you intend to run at, never at concurrency one.

Above realtime, the argument is about who pays for the hiccup#

The second regime looks different in every respect. Suppose steady-state generation runs at 1.4x realtime, RTF 0.71, but the stream occasionally goes quiet for 600 ms — a garbage collection pause on the vendor's side, a TCP retransmit, a scheduler hiccup. The buffer now gains 0.29 audio-seconds per wall-second whenever the stream is flowing, so a 600 ms interruption drains 600 ms of buffer and then takes about 2.1 seconds of normal streaming to recover. Buffer occupancy is a sawtooth around a healthy level rather than a slide toward zero, and a prebuffer is exactly the right instrument, because the thing you are absorbing is bounded and transient.

With a 200 ms prebuffer, a 600 ms hiccup produces roughly 400 ms of audible silence in the middle of a word. With an 800 ms prebuffer it produces none, and there is 200 ms of margin left over. The proposal to raise the prebuffer is arithmetically correct about the thing it claims.

The cost is the part that gets skipped. A fixed 800 ms prebuffer adds 600 ms to the start of every turn, including the turns that were never going to stall. If hiccups occur on 6% of turns, you have made 94% of the product measurably slower in order to protect 6% of it, and 600 ms is not a subtle amount of extra latency at the front of a conversational turn — it is roughly the difference between a reply that feels like an answer and one that feels like a pause. Whether that trade is right depends on which failure the user notices more, and on a real product both failures get noticed, which is why the fixed-value argument has no clean winner.

An adaptive buffer settles it by refusing to pick one number. Start the target at 200 ms, count underruns per session or per route, and on the second underrun raise the target to 800 ms for the remainder of that session, decaying it back toward 200 ms after a few clean turns. The 94% of turns that never stall keep their fast start, the sessions on a bad network or a degraded region pay the 600 ms only after they have demonstrated they need it, and the policy is a dozen lines of state rather than an argument.

class AdaptiveBuffer:
    def __init__(self, floor=0.20, ceiling=0.80):
        self.floor, self.ceiling = floor, ceiling
        self.target = floor
        self.clean_turns = 0

    def on_underrun(self):
        self.target = self.ceiling
        self.clean_turns = 0

    def on_clean_turn(self):
        self.clean_turns += 1
        if self.clean_turns >= 5:
            self.target = max(self.floor, self.target - 0.20)
            self.clean_turns = 0

Where this stops being a buffering problem#

The two regimes want different instruments, and the diagnostic that separates them is one number you can measure in an afternoon. Sub-realtime generation is a throughput problem and wants pipelining; above-realtime jitter is a variance problem and wants a buffer sized by observation. Prebuffering is not a general-purpose fix for slow synthesis, and pipelining is not a fix for a flaky network — using either in the wrong regime produces a system that is worse on the dimension you were not looking at.

Neither of them is free in the audio itself, which the buffer arithmetic cannot show you. A prebuffer only delays sound. Chunking the text at sentence boundaries so it can be pipelined changes what the synthesiser is asked to say: it hands the model one sentence at a time, with no sight of what comes next, and the model's decisions about pitch, pace and stress are made on that partial view. The manifest feature came out of the pipelining change with a healthy buffer and a new complaint, which was that the voice sounded like forty-two separate announcements read by the same person rather than one list. That is a prosody problem, and prosody is the next thing to take control of.