Prosody is what you control and what chunking takes away

A pharmacy chain shipped a callback that reads discharge instructions to patients, with an accessibility control labelled "speak slowly" that mapped to the synthesiser's speed parameter at 0.7. It passed review, because the reviewers listened to "Take one tablet twice daily with food" and heard a slow, clear voice. Three weeks later a pharmacist called in a complaint: a patient had heard "take one tablet twice daily" as "take one tablet twice a day, eight" and had asked whether they were meant to take eight.

The audio was intact. What had happened was that the sentence in question was "Take one tablet twice daily for twenty-eight days", and at speed 0.7 the model was reliably eliding the /t/ in "twenty-eight" and inserting a pause after "daily" that was long enough to read as a sentence boundary. At speed 1.0 the same sentence was clean. Nobody had listened at 0.7 to anything other than the two example sentences in the review doc.

That is the shape of the whole problem with prosody controls. Prosody is everything about a spoken sentence other than which words it contains: the pace, the rise and fall of pitch, the loudness, and the placement of stress and pauses. A synthesiser produces it by shaping the phonemes themselves — the individual speech sounds a word is built from — which is how a /t/ comes to go missing when you ask for a slower delivery. The controls work across the range where you tested them and degrade in ways that are specific to the setting, the text and the voice, so a control that is correct at its default and correct at one extreme can be wrong at the other, and the only way to know is to listen at each setting with text that can actually break.

Three layers you can steer, with different failure modes#

Control over how something is said arrives at the synthesiser through three separate channels, and confusing them is the source of a lot of wasted effort.

Markup is the oldest of the three. SSML, the Speech Synthesis Markup Language, is an XML vocabulary standardised by the W3C for saying how text should be spoken; it lets you annotate the text itself with rate, pitch, volume, pauses, emphasis, and explicit pronunciations, and it is the only layer that gives you per-word control rather than per-utterance control.

<speak>
  Take <emphasis level="moderate">one</emphasis> tablet
  <prosody rate="85%">twice daily</prosody>
  <break time="250ms"/>
  for <say-as interpret-as="cardinal">28</say-as> days.
  Your next refill is due
  <say-as interpret-as="date" format="mdy">11/14/2026</say-as>.
  Call <phoneme alphabet="ipa" ph="ˈklɛərmɒnt">Claremont</phoneme> Pharmacy
  with questions.
</speak>

Engine parameters are the second layer, and on most current neural APIs they are where the real behaviour lives. These are per-request scalars: a speed multiplier, a stability or temperature-like control that trades consistency against expressiveness, a style exaggeration weight, sometimes a free-text instruction field.

{
  "text": "Take one tablet twice daily for 28 days.",
  "model_id": "eleven_flash_v2_5",
  "voice_settings": {
    "stability": 0.55,
    "similarity_boost": 0.75,
    "style": 0.15,
    "speed": 0.85
  },
  "previous_text": "Here are your discharge instructions.",
  "next_text": "Do not take this medication on an empty stomach."
}

The third layer is context conditioning, which is not a control in the usual sense — it is what the model is allowed to see. Fields like previous_text and next_text do not change the audio you get back for those strings; they change how the target sentence is spoken, because the model conditions its pitch contour and pacing on surrounding text it will not synthesise. This layer is the one that chunking destroys, and the rest of the chapter turns on it.

The failure modes differ by layer. Markup fails by being ignored: a lot of neural endpoints accept an SSML document, honour <break> and <phoneme>, and silently drop <prosody rate> because the acoustic model has no conditioning input for it. Test that your markup does anything at all before you build a feature on it — synthesise the same sentence with rate="50%" and rate="200%" and compare durations, and if they match within a few milliseconds the tag is decorative. Engine parameters fail by degrading the acoustics rather than by being ignored, which is harder to notice, because the request succeeds and the audio plays.

The evaluation that would have caught the pharmacy bug is a grid rather than a sample. Take fifty sentences drawn from real discharge instructions — the ones with dosages, dates, drug names and abbreviations, not the two clean ones from the review doc — and synthesise each at slow, fast, quiet and emphatic settings as well as the default. That is 250 clips, which is an hour of listening or twenty minutes of scripted checking. Since the previous topic left you with a recogniser, the cheap first pass is to transcribe every clip and diff it against the source text: the pharmacy's own numbers, once they ran this, were 2.1% word error at speed 1.0 and 9.4% at speed 0.7, concentrated almost entirely in numerals and drug names. A recogniser is not an ear and will happily accept audio that sounds unpleasant, so the transcription pass finds intelligibility failures and tells you nothing about naturalness. It is a filter that turns 250 clips into the eleven you need to actually listen to.

What the synthesiser cannot condition on, it gets wrong#

The reason to chunk text at all is the arithmetic from the last chapter: sentence-sized requests start producing audio before the language model has finished generating its answer, and they can be pipelined across concurrent streams. A model that streams tokens at 40 per second takes about 4.5 seconds to produce a 180-word answer, so waiting for the full text before calling the synthesiser puts a 4.5-second floor under time to first audio, and that floor disappears if you send the first sentence the moment its terminal punctuation appears.

What the listener hears from that pipeline is three declarations rather than one list. Intonation in English carries structure across sentence-sized spans: pitch declines gradually through a clause and resets at the start of the next one, a colon before a list holds the contour open, non-final list items end on a slight rise, and the final item falls. A synthesiser given "a pallet of glass," in isolation has no way to know it is a medial list item, so it produces the contour for a complete utterance, and the same happens for the next two. The audio is individually correct and collectively wrong.

The same mechanism produces a cluster of related artefacts. A chunk that begins with "and" or "which" gets a sentence-initial pitch reset that sounds like a false start. A question split before its final clause loses the terminal rise, because the rise is realised on the last few syllables and the model did not know a question mark was coming. Loudness and speaking rate drift between chunks, since each request re-derives them, so a long answer wanders in tempo. Speaker timbre can shift too, especially at low stability values, which produces the impression of a voice that keeps clearing its throat.

Crossfading the chunk boundaries is the fix people reach for first, and it addresses a different problem than the one they have. Crossfading removes clicks and discontinuities at the seam, which is worth doing, and it cannot supply a rising contour that was never generated. If the intonation is wrong, it is wrong across the whole chunk, not at its edges.

Chunking strategies, and what each one costs#

Three changes recover most of the lost prosody without giving up the latency win.

Send lookahead and lookbehind context. When you emit chunk n, pass the previous chunk's text as previous_text and, if the language model has already produced it, the next chunk as next_text. You will often have the next sentence in hand by the time the current one finishes synthesising, because token generation outruns audio playback. This costs nothing in latency and recovers list contours and question rises in most cases.

Chunk at clause boundaries with a minimum length. Splitting on every . produces chunks of three words, which is where the artefacts are worst. A rule of "split at sentence-final punctuation, but accumulate until the buffer is at least 120 characters, and never split inside a quoted string or a numeral" keeps chunks long enough for the model to establish a contour. The first chunk is the exception worth special-casing: make it short deliberately, since it is the one that sets time to first audio, and accept its prosody cost.

Hold one generation context open. Providers with a websocket streaming interface let you push text into a single ongoing synthesis session, which keeps the model's internal state across the boundary rather than starting cold. This is strictly better than independent HTTP requests for continuity, and it constrains you to one stream, which means you cannot pipeline across concurrent connections at the same time.

That table also settles the argument from the last chapter about the synthesiser running at 1.4x realtime with 600 ms hiccups and a 200 ms prebuffer. Chunking and pipelining is the correct instrument when generation is the bottleneck, and at 1.4x realtime generation is not the bottleneck — the buffer refills between hiccups, so throughput is adequate and only variance is hurting. Paying for that variance with degraded intonation across every boundary of every answer is a bad trade, because you spend a permanent quality cost on a transient problem. The adaptive buffer handles it for the cost of a few lines of state. Chunking earns its prosody cost when time to first audio is the binding constraint or when steady-state generation is below realtime, and in those cases you take the boundary artefacts and mitigate them with context fields rather than pretending they are not there.

The controls do not reach emotion#

Prosody controls change how words are pronounced, timed and stressed. They do not give you a model of what the speaker means, which is why a sentence marked up to be spoken slowly and gently still sounds wrong when it delivers bad news, and why style exaggeration applied to neutral text reads as parody rather than warmth. Appropriateness is a property of the pairing between the text and the delivery, and none of these controls know what the text is about. Systems that get this right choose different voices and different parameter presets per conversational intent, decided by the language model that produced the text, which is a routing problem rather than a synthesis one.

There is also a limit that no amount of tuning crosses: a synthesiser reading a chunk cannot recover information the text never carried. "Read the number 4174" is ambiguous between four digits and a cardinal, and "Dr. Sale" is ambiguous between a title and an abbreviation. Markup resolves those, so the durable fix is upstream — have the language model emit <say-as> annotations or a structured field, rather than hoping the synthesiser guesses.

Everything so far has treated the synthesiser as the system. In a real voice agent it is one stage among six, and its 250 ms of first-audio latency sits inside a turn that also has to detect that the user stopped talking, finalise a transcript, generate a first token, cross a network and fill a playout buffer. Optimising the stage you happen to own is how teams end up shaving 40 ms off a 250 ms term while a 500 ms term sits untouched, so the next thing to put a number on is the whole turn.