On the PSTN you inherit the transport and much of the budget

The agent that hit 940 milliseconds in the browser was pointed at a phone number, and the first measured call came back at 1,480 milliseconds median with nothing in the application changed. The stages that the team owned had not moved: endpointing was the same policy, the model's time to first token was 361 milliseconds against 358 in the browser, synthesis was unchanged. The extra 540 milliseconds was distributed across two rows of the budget table that had been written for a browser and were now describing something else entirely.

Reasoning about that number requires knowing what the audio actually crossed. The call is on the PSTN, the public switched telephone network — the worldwide arrangement of carriers, exchanges and handsets that predates the internet and still carries every ordinary phone call — which your software reaches through a trunk provider, a wholesaler that sells connectivity into that network and speaks SIP, the session initiation protocol, to set calls up and tear them down. An outbound call to a patient's mobile therefore leaves your media server as RTP to that trunk provider, enters the carrier network, is transcoded at least once, traverses a mobile core, and is finally rendered by a handset whose audio pipeline predates everything in this material. The reverse path does the same in the other direction. None of it is negotiable, and most of it is not observable.

Everything to the left of your media server is inherited, including the first hop, which is whichever mobile voice path the handset happens to be on — the older GSM circuit or the packet-switched VoLTE that carries voice over the LTE data network. The two hops there account for the bulk of the difference, and they are not one number but several small ones that individually look ignorable.

What the network hands you before you do anything#

The codec is the first inheritance and the most consequential for quality rather than latency. The telephone network carries G.711 — μ-law in North America and Japan, A-law elsewhere — which samples at 8 kHz and passes roughly 300 Hz to 3,400 Hz. That band was chosen in the 1960s to make speech intelligible to a human listener at the lowest cost, and it succeeds at that while removing exactly the high-frequency energy that distinguishes fricatives from each other. Speech recognition on the same utterances degrades measurably: on internal test sets it is normal to see word error rate, the share of words the recognizer gets wrong once insertions and deletions are counted, rise from around 6% on 16 kHz audio to around 10% on the 8 kHz version of the same recordings, with the errors concentrated in the places that hurt a scheduling agent — S against F, M against N, "fifteen" against "fifty".

The consequence for design is upstream of the model. The trailing-content rules from chapter 2 matter more on a phone, because digit sequences are both more common in telephone interactions and more error-prone, and because a caller reading a reference code out loud is the exact case where the recognizer is weakest and the endpointing threshold needs to be longest. Confirmation strategy changes too: reading a date back is worth the extra turn on a phone call in a way it is not on a laptop.

The jitter buffer is the second inheritance and the one that shows up in the latency numbers. Your media server has one you control, and the handset and the carrier each have one you do not, and their depths are chosen conservatively because carriers optimize for never producing an audible glitch. That is a reasonable priority for a phone network and the wrong one for a conversational agent, and there is no interface through which you can express the disagreement.

The third is that some providers make endpointing decisions for you. Telephony platforms have long offered a "gather speech" primitive with a speech timeout, and if you build on those higher-level primitives you get a turn boundary handed to you as an event, computed by a component that has never seen your prompt, does not know whether you asked a closed question, and cannot be conditioned on trailing content. Everything in chapter 2 becomes unavailable at once. The way to keep it is to take the raw media stream — the bidirectional audio socket that these platforms also expose — and do turn detection yourself.

{
  "type": "session.update",
  "session": {
    "audio": {
      "input":  { "format": { "type": "audio/pcmu" } },
      "output": { "format": { "type": "audio/pcmu" } }
    },
    "turn_detection": { "type": "server_vad", "silence_duration_ms": 700 }
  }
}

Setting the session to μ-law rather than 24 kHz PCM matters for a reason beyond correctness: it removes a resampling stage in each direction. Feeding 8 kHz telephone audio to a pipeline expecting 24 kHz means upsampling that invents nothing and costs a few milliseconds, and synthesizing at 24 kHz to be downsampled for the trunk means paying for bandwidth that the codec discards. Telephony frames arrive as base64 μ-law payloads at 20-millisecond intervals, 160 bytes each, and the tightest pipeline is the one that keeps them in that representation end to end.

Rewriting the budget against the floor#

The useful exercise is to reprint chapter 1's table with the telephone numbers and a column that says who can move each line. The floor is what remains after every optimization available to you has been made.

Roughly 490 milliseconds of that call is spent before your first line of code runs and after your last one, which reframes the optimization problem. A team targeting a 1.2-second conversational feel on the telephone has 710 milliseconds to spend across turn detection, transcription, inference and synthesis, and those four currently sum to 910. The way that budget gets met is by attacking the largest line — endpointing — with the conditioning from chapter 2, and by accepting that the browser's numbers were never the target.

The one inherited line that is partly yours is geography. If your trunk provider's point of presence is in Virginia, your media server is in Oregon, and your model endpoint is in Ireland, you have added two intercontinental legs to a path that already had a floor, and the fix is placement rather than code — media server in the same region as the trunk POP, model endpoint in the same region as the media server. It is common to find 150 milliseconds in this on a first deployment, which is more than any prompt optimization will produce.

Measuring the floor instead of assuming it#

The numbers in that table are defensible starting points and they are not your numbers, because the floor depends on your trunk provider, your regions and the networks your callers are on. Measuring it takes an afternoon and is worth doing before any optimization work is scoped, because it tells you whether the target you have been given is reachable at all.

The instrument is a loopback call. Configure a test route where your media server echoes received audio straight back with no processing, dial it from a real handset on a real mobile network, and play a short click or a 1 kHz burst into the handset while recording the handset's own output. The delay between the click and its echo in that recording is the full inherited round trip, including both handset audio pipelines, both access networks, the carrier core, transcoding in both directions and your trunk leg. Halving it gives a usable one-way floor. Doing it from four or five different networks — two mobile carriers, a landline, an office VoIP handset, a softphone — produces a spread rather than a point, and the spread is more informative than the middle of it.

# Received frames are echoed with a monotonic mark, so the server's own
# contribution to the loop is measured rather than assumed.
async def loopback(stream):
    async for frame in stream.inbound():        # 160-byte mu-law, 20 ms
        t_in = time.monotonic()
        await stream.outbound(frame)
        metrics.observe("loopback.server_ms", (time.monotonic() - t_in) * 1000)

If your server's own contribution to that loop is above about 15 milliseconds you have a buffering problem of your own making, and it is usually a media server configured with a 100-millisecond jitter buffer inherited from a video conferencing default. That is the one buffer in the chain you can shorten, and shortening it from 100 to 40 milliseconds is 60 milliseconds recovered in each direction.

What the phone gives back#

The inheritance is not uniformly bad, and two of the things you lose control over are things you were previously responsible for. Echo is handled: the handset is held against a head, which is the physical isolation from chapter 3 for free, and carriers run their own echo cancellation on hybrid junctions. Barge-in works on a phone call with no client-side work at all, because the network is full duplex by design and has been since it carried nothing but humans. You also get an out-of-band signalling channel in DTMF — dual-tone multi-frequency, the paired tones a keypad emits, which the network carries as discrete key events rather than as speech to be recognized — and it is worth remembering: a caller reading a 12-digit policy number into a narrowband codec is the worst case for recognition, and "press it on your keypad" turns a 10% error rate into zero.

The load characteristics differ too. A telephone call is a bounded resource with a known duration and a per-minute cost, typically somewhere around a cent a minute for the leg itself, and that gives you a denominator that browser deployments lack. When the model spend per call is 4 cents and the trunk is 1.2 cents, a 300-millisecond latency improvement that requires a more expensive model is a decision you can price.

What none of these numbers tell you#

Everything in this chapter and the last is a median with a plausible spread, and the trace from a single well-behaved call. The clinic's actual complaint was never about the median. It was about a specific recording, on a specific Tuesday, where a caller waited 1.9 seconds — and the budget above, met in full on the average call, is entirely compatible with that recording existing. Finding out which of the eleven lines produced it requires measurement that survives contact with a distribution.