Silence detection cannot tell you a sentence finished

Two complaints arrived in the same week about the same setting. The clinic's patients, who skew over seventy, said the agent kept interrupting them: they would give a phone number, pause for breath after the area code, and the agent would start talking over the rest of it. The clinic's own staff, using the same agent through an internal line to check the day's schedule, said it was slow — they would say "cancellations today" and then sit through a delay that felt like the system had not registered they had finished. One configuration value was producing both reports, and it was this one:

{
  "type": "session.update",
  "session": {
    "turn_detection": {
      "type": "server_vad",
      "threshold": 0.5,
      "prefix_padding_ms": 300,
      "silence_duration_ms": 700
    }
  }
}

That block is a session.update message sent to a hosted realtime session, which is the arrangement where the provider holds the conversation open over one connection and you adjust its behaviour by sending it settings mid-call. Moving silence_duration_ms down to 400 fixes the staff complaint and makes the patient complaint worse. Moving it to 1,000 fixes the patients and makes the whole product feel like a fax machine. There is no third value, because the parameter is answering the wrong question.

What the detector actually reports#

Voice activity detection classifies short frames of audio — typically 10, 20 or 30 milliseconds — as speech or not speech. Classical implementations such as the one shipped inside WebRTC, the real-time audio and video stack built into every browser, do it with energy and spectral features and an aggressiveness setting from 0 to 3; modern ones such as Silero VAD run a small neural network that costs around a millisecond of CPU per frame and is far more robust to background noise. Either way the output is a per-frame probability that a human voice is present, and the threshold field above is where that probability has to land to count as speech.

That is a genuinely useful signal, and it is not the signal the endpointing decision needs. "No speech energy for 700 milliseconds" and "this person has finished what they were saying" are different propositions, and the gap between them is filled by pauses that have nothing to do with being done. People pause to breathe, to retrieve a word, to check a number on a piece of paper, and to think — and the pause a speaker takes mid-utterance is frequently longer than the gap they leave at the end of one. Measurements on spontaneous telephone speech put within-turn pauses commonly in the 200–1,000 millisecond range, with older speakers and speakers reading out data sitting at the upper end of that. Any silence threshold in that range will fire inside sentences, and any threshold above it will add its full duration to every well-formed turn.

Telephony solved a narrower version of this a long time ago, and the way it solved it is instructive. Speech IVR systems — interactive voice response, the press-one-for-billing phone menus that have answered business numbers since the 1990s — built on VoiceXML exposed two separate timeouts: a completion timeout used when the recognizer's grammar had matched something valid, and an incomplete timeout used when the audio so far matched only a prefix of a valid utterance. The system waited longer when it could tell you were partway through, and that worked because the grammar defined exactly what a complete utterance was. Open-ended conversational speech has no grammar, so the mechanism was dropped and the industry standardized on one silence value — which is how a parameter that used to be two ended up being asked to do the work of both.

The distinction has a name worth keeping straight. Voice activity detection is the acoustic question, and it is essentially solved. Endpointing, or turn detection, is the conversational question — has this speaker yielded the floor — and it is not solved, because the evidence that settles it is mostly linguistic. A human listener knows the caller is not finished after "my number is five five five" without waiting at all, because the utterance is syntactically and semantically incomplete. That knowledge is in the words, and a detector looking only at energy does not have the words.

Conditioning the wait on what was just said#

The usable version of this is to keep the silence timer and make its duration a function of the trailing content of the transcript so far. You already have partial transcripts streaming in a chained pipeline, and the last few tokens carry most of the predictive value. A caller who has just said "and" is going to keep talking; a caller who has just completed a noun phrase that answers the question you asked is probably done.

Wired into the loop, this is a small amount of code sitting between the frame-level detector and the decision to commit the turn:

DIGIT_WORDS = {"zero","oh","one","two","three","four","five","six","seven","eight","nine"}
CONTINUERS = {"and","but","or","so","because","for","to","my","the","a","of","with"}
FILLERS    = {"um","uh","er","hmm","like","well"}

def silence_threshold_ms(tail: str, expecting: str | None) -> int:
    words = tail.lower().replace(",", " ").split()
    if not words:
        return 700
    last = words[-1].strip(".?!")
    if last in DIGIT_WORDS or last.isdigit():
        return 1200
    if last in CONTINUERS:
        return 1100
    if last in FILLERS:
        return 1000
    if expecting == "closed" and len(words) <= 3:
        return 300
    if tail.rstrip().endswith((".", "?")):
        return 500
    return 700

def should_commit(silence_ms: int, tail: str, expecting: str | None) -> bool:
    return silence_ms >= silence_threshold_ms(tail, expecting)

The expecting argument is the cheapest source of signal in the whole system and the one most often left on the table. Your agent asked the question, so it knows whether it asked "which day works for you?" or "tell me what happened". A closed question licenses a 300-millisecond threshold, because a one-word answer followed by silence is unambiguous, and that single branch is what fixes the staff complaint without touching the patient case at all.

Word lists are the floor, and the ceiling is a model. Providers now expose a semantic variant of turn detection — OpenAI's Realtime API, the hosted session interface those session.update messages are addressed to, offers "type": "semantic_vad" with an eagerness setting instead of a millisecond value, and LiveKit ships a small transformer that scores end-of-utterance probability on the running transcript in roughly 25 milliseconds of CPU. Both are doing what the table above sketches, with a learned function instead of a hand-written one, and both are worth adopting once you have the evaluation described below, because without it you cannot tell whether the model is better than your rules on your callers.

{
  "type": "session.update",
  "session": {
    "turn_detection": {
      "type": "semantic_vad",
      "eagerness": "low"
    }
  }
}

The fix that turns cut-offs into collisions#

The tempting shortcut is to drop the threshold to 400 milliseconds for responsiveness and add a recovery rule: if the caller starts speaking again within a second of the agent starting, stop the agent and listen. It is tempting because it is one small change on each side and it does produce a system that never leaves anyone waiting.

It converts a cut-off into a collision, which is worse. When the agent begins speaking over a caller who was mid-sentence, three things have already happened that the recovery rule does not undo. The caller has heard a voice start on top of theirs, which is socially jarring in a way that a slightly long pause is not, and their reliable human response is to stop and repeat themselves from the beginning. The caller's continuing speech — the second half of the phone number — arrived while the agent's own audio was playing, so on a speakerphone it may be partly lost to echo and on any device it is now competing with output for the detector's attention. And the agent has committed a turn whose transcript is "my number is five five five", which is what the model saw and what your conversation history now contains, so the recovery has to reconcile a partial user utterance with a partial assistant reply, on every affected turn.

The rate matters more than the mechanics. On a form-filling call where callers read out phone numbers, dates of birth and reference codes, a flat 400-millisecond threshold will cut in on a substantial fraction of turns — in the clinic's audio it was just over a fifth of the turns containing digits. A recovery path that runs on one turn in five is not a safety net, it is the main flow, and it will define what the product sounds like.

Deciding it works means annotated audio, not a feel#

Turn detection is a classifier, so it gets evaluated like one, and the dataset is recorded calls with human labels marking the true end of each user turn. A hundred calls is enough to start; the labelling is fast because the annotator only has to place a marker where they, as a listener, would have known the caller was done.

Two metrics come out of it. The cut-off rate is the fraction of turns where the system committed before the human marker, and it is the one users complain about. Added latency is the median gap between the human marker and the system's commit on turns it did not cut off, and it is the line item this decision contributes to the budget from the previous chapter. They trade against each other by construction, so a single aggregate hides the whole problem — which is why the evaluation has to be sliced by the populations that behave differently.

Two things in that table are worth reading carefully. The staff row got dramatically faster and no less accurate, which is the closed-question branch doing its work. The speakerphone row barely moved, because its cut-offs are not caused by hesitation — they are caused by a second voice in the room being detected as the caller stopping and someone else starting, which no amount of trailing-word analysis addresses. That slice needs different machinery, and it is the same machinery that decides whether the agent's own voice counts as speech.

What conditioning does not buy you#

None of this makes turn detection correct, and it is worth being plain about the residue. There is genuine ambiguity in speech that no model resolves, because the speaker themselves has not decided whether they are finished; conversation between humans handles this with overlap and repair rather than with a correct decision, and a voice agent has neither. Prosody carries a good deal of the signal — a rising pitch at the end of "my number is five five five" marks continuation as clearly as the syntax does — and a system reading only the transcript throws that away, which is one of the arguments for models that consume audio directly.

The remaining category is the one the speakerphone row exposed. Everything above assumes the audio arriving at your detector is the caller and only the caller. On a laptop with an open microphone the loudest thing in the room is frequently your own agent, and the detector will faithfully report it as speech.