Realtime voice agents

Read these first

• Text to speech
• The agent loop implemented from scratch

What you will know

• Build a voice agent with an end-to-end latency budget you can name

  Measure separate targets for capture, turn detection, inference, synthesis, network transit, and playback, then verify p50 and tail behavior against the budget.

• Handle barge-in so a user can interrupt

  Stopping audible output is insufficient: cancel or invalidate in-flight generation, tool work, queued audio, and stale events without losing the new utterance.

• Detect turn ends without cutting people off

  VAD alone confuses pauses, hesitations, background speech, and slow speakers, so evaluate turn decisions against annotated audio across noise and speaking styles.

• Instrument and trace a realtime call across capture, transport, inference, synthesis, and playback

  Correlated timestamps expose whether perceived delay comes from buffering, network jitter, model time-to-first-token, synthesis, or playback rather than from a single aggregate metric.

• Compare a speech-to-speech pipeline with an STT-to-LLM-to-TTS pipeline under matched latency, quality, and cost tests, then defend a production choice and fallback

  Direct audio models may reduce handoff latency but can weaken inspectability, tool control, or recovery, making the simpler staged design preferable for some products.

Chapters

1. 1

   A voice turn is a chain of deadlines, not a request

   Capture, turn detection, inference, synthesis, network transit and playback each consume part of a budget the user experiences as one silence. Writing a number against every stage, with separate p50 and tail targets, is what turns 'it feels laggy' into a claim you can test. A chat pipeline that meets an average target can still be unusable, because conversation is judged on the turns that went wrong.

   Read chapter
2. 2

   Silence detection cannot tell you a sentence finished

   Voice activity detection reports acoustic energy; deciding a turn is over is a semantic judgement about whether the thought completed. A 700ms threshold cuts off elderly callers mid-sentence and still feels sluggish to staff who speak in short bursts. Condition the threshold on what was just said, since trailing digits, conjunctions and hesitation markers all predict more speech, and evaluate turn decisions against annotated audio across noise levels and speaking styles.

   Read chapter
3. 3

   Your microphone hears your own agent, and 'mm-hm' is not an interruption

   Barge-in assumes an open microphone while the agent is speaking, and that is where the audio plumbing decides whether the feature is possible at all. Without acoustic echo cancellation on a speakerphone or a laptop the agent hears its own output, its own detector fires, and it interrupts itself on every turn; muting the microphone during playback removes that problem and removes barge-in along with it. The second decision is double-talk: a caller saying 'mm-hm' or 'right' while the agent talks is a backchannel, not an interruption, and an agent that stops dead whenever the user agrees with it is the most common shipped defect in this product category.

   Read chapter
4. 4

   Barge-in is a cancellation problem, and history is part of it

   Stopping playback is the easy half. You also have to cancel in-flight generation, abandon or compensate tool work already dispatched, drop queued audio and ignore events that arrive from the turn you just killed, all without losing the utterance that interrupted. The piece teams forget is conversation history: the assistant turn must be truncated to the audio the user actually heard. Once every turn is cancellable, starting one speculatively costs only discarded inference.

   Read chapter
5. 5

   The client audio leg wants UDP, and here is why

   A plain WebSocket runs over TCP, so a single lost packet stalls every packet behind it while it is retransmitted, and in a live call that delay never gets paid back. WebRTC runs over UDP with a jitter buffer and packet loss concealment, trading a barely audible artefact for continuity. The distinction matters most on exactly the mobile and hotel-wifi connections your users are calling from.

   Read chapter
6. 6

   On the PSTN you inherit the transport and much of the budget

   The previous chapter's argument applies where you get to choose the transport, which means a browser or an app. An outbound call to a dental patient is a SIP leg into the PSTN, where you get 8kHz narrowband through a codec you did not pick, a carrier-side jitter buffer you cannot tune, sometimes provider-side voice activity detection making endpointing decisions on your behalf, and a per-leg latency floor that spends a large share of the budget before any model runs. Write the budget against that floor rather than against browser numbers, and know which stages are actually left for you to move.

   Read chapter
7. 7

   A median cannot show you the eight percent complaining

   1.05 seconds against a 1.2 second budget is a statement about the middle of the distribution and says nothing about the calls that generated complaints. Emit correlated timestamps at capture, transport, model first token, synthesis first chunk and playback start, keyed by call and turn, then pull the traces for the complaining calls specifically. The tail usually resolves to one stage, and it is rarely the one the aggregate dashboard implicated.

   Read chapter
8. 8

   Covering a 1.4-second tool call without painting yourself in

   The obvious answer is a filler phrase, and it is half right: the filler has to be pre-synthesised, because generating one costs the very latency it exists to hide. The half that gets skipped is what happens when the lookup takes six seconds or fails outright, leaving an agent that has already promised an answer. Write the overrun and failure paths in the same change as the filler.

   Read chapter
9. 9

   Speech-to-speech trades away the text checkpoint

   Direct audio models remove two handoffs and can respond in a third of the time, and they also remove the place where evals, output guardrails, redaction and logging attach. For an appointment agent that must never state a price, that checkpoint is the enforcement point, not a nicety. Decide by what the product must be able to prove, and design the staged fallback before the realtime session becomes load-bearing.

   Read chapter