Your microphone hears your own agent, and 'mm-hm' is not an interruption

The bug report said the agent "stutters". What it did, on a conference-room speakerphone and nowhere else, was start a sentence, cut itself off after about a third of a second, wait, and start the same sentence again — sometimes twice in a row, sometimes settling after the second attempt. The event log made the cause plain once someone lined the timestamps up:

14:22:06.115  response.output_audio.delta      first chunk sent to client
14:22:06.180  playback_start (client)
14:22:06.521  input_audio_buffer.speech_started
14:22:06.522  barge_in -> response.cancel
14:22:07.244  input_audio_buffer.speech_stopped
14:22:07.390  transcript(user) = "I can help you with that appo-"

The user utterance that interrupted the agent was the agent. The microphone in the middle of the table picked up the speaker in the middle of the table, the frame detector did exactly what the previous chapter described and reported speech energy, and the barge-in logic — the rule that a caller who starts talking while the agent is speaking should be allowed to take the floor, which is what barge-in names, and which was correct in isolation — cancelled the response. Then the same thing happened again, because the retry played the same audio into the same room.

The echo path is a physical fact, and one component removes it#

Sound leaving a loudspeaker reaches the microphone by air and by the surfaces of the room, delayed by anywhere from a couple of milliseconds on a laptop to 60 or more in a large room with hard walls, and attenuated by an amount that depends entirely on the geometry. On a headset the attenuation is enough that nothing more is needed. On a laptop at arm's length the agent's voice arrives at the microphone at a level comparable to a person speaking quietly, and on a speakerphone it is usually the loudest thing the microphone hears.

Acoustic echo cancellation, abbreviated AEC in configuration flags and vendor documentation, removes it by subtraction. The canceller is given a reference signal — the audio being played out — and adaptively estimates the impulse response of the room, meaning the pattern of delays and reflections by which a single sound becomes what the microphone eventually hears, so it can predict what the microphone should be receiving from the loudspeaker and subtract that prediction from the actual microphone signal. What remains is whatever the microphone heard that the loudspeaker did not produce, which is the human. A good implementation achieves 30 to 40 dB of echo return loss enhancement, which is the measure of how much of its own output the canceller manages to remove — 30 dB is a thousandfold reduction in power — and the residual it leaves is quiet enough that a voice detector will not fire on it.

Two conditions have to hold for that to work, and both are where deployments break. The canceller needs the reference signal, aligned in time with the microphone capture to within a few milliseconds, which means it has to live on the device that is playing the audio; a server that knows what it sent has neither the room's response nor the device's clock. And the canceller must see the signal that actually reached the speaker, so if your playback path applies volume changes or mixing after the point where the reference is tapped, the estimate is wrong and cancellation degrades. In a browser both conditions are satisfied for free as long as you ask, because the constraint is honoured by the same audio engine that renders your output:

const stream = await navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: true,   // AEC3 in Chromium; this is the whole feature
    noiseSuppression: true,
    autoGainControl: true,
    channelCount: 1,
    sampleRate: 24000,
  },
});

The trap is playing agent audio through a path the browser's canceller does not treat as the reference — a second <audio> element on a different device, a Web Audio graph routed to a non-default output, or a native app that plays through the system mixer while capturing through a library that never registers a reference. In every one of those cases echoCancellation: true is set, the developer believes echo is handled, and the room proves otherwise.

It is worth separating echo cancellation from the two settings that usually sit beside it, since they are frequently discussed as one bundle of audio hygiene and they solve unrelated problems. Noise suppression attenuates stationary background sound — fans, traffic, the hum of an open-plan office — and it does nothing about the agent's voice, because the agent's voice is speech and a noise suppressor is built to preserve speech. Automatic gain control normalizes the caller's level so a quiet speaker at arm's length is transcribed as well as a loud one, and it can actively hurt echo cancellation by changing the microphone's gain faster than the canceller can re-adapt, which is why a small number of deployments see echo return specifically when the room goes quiet and the gain climbs. If you are hunting a leak, turning gain control off is a cheap experiment.

Verifying that the canceller is working is a one-line metric rather than a listening test, and it belongs in your dashboard from the first week: the fraction of turns where the detector reported speech starting within 500 milliseconds of playback beginning. On a healthy deployment that number sits under one percent, since a caller genuinely interrupting that early is rare. When a client ships with the reference wired to the wrong output, it jumps to somewhere north of 60% overnight and identifies itself without anyone having to reproduce the room.

Here is the loop that produces the stutter, which is worth having in your head because its signature — a self-interruption a few hundred milliseconds after playback begins, on every turn — identifies it without any debugging:

The options, and what each one costs#

Echo cancellation is not the only way to stop the loop; it is the only way that keeps barge-in. The alternatives are worth laying out together, because teams reach for the cheap ones under deadline and then discover what was traded.

The half-duplex option — audio in one direction at a time, which here means muting the microphone whenever the agent is speaking, as against the full duplex of a telephone call where both parties can be heard at once — deserves a moment because it is genuinely correct for some products and catastrophic for others. It makes the echo problem disappear at zero engineering cost and produces a system that literally cannot hear a user who tries to interrupt, which is fine for a 3-second confirmation prompt and unacceptable for an agent that reads back a five-item order. If you choose it, choose it explicitly and keep the agent's utterances short, because the length of your longest utterance is now the length of time the user is not allowed to speak.

The server-side gate is the compromise you end up at when the client is a phone system you do not control. You know the timeline of what you sent, so you suppress barge-in detection during the window when your own audio should be arriving back, with a margin for round-trip delay. It half works. The margin has to cover jitter, so it is generous, and every millisecond of it is a window where a real interruption is ignored — and if the far end has its own echo canceller with a slow adaptation period, the first half second after your audio starts is exactly where leakage occurs and exactly where your gate has to be widest.

Agreement is not interruption#

Once the microphone stays open, a second decision appears that is invisible in a half-duplex design. People talk while they are listening. "Mm-hm", "right", "okay", "yeah", "got it" — these are backchannels, and their function is to tell the speaker to keep going. Conversation analysis has studied them for decades; they are produced constantly in cooperative telephone speech, and in a sample of recorded calls from a scheduling agent, 61% of caller vocalizations that overlapped agent speech were backchannels of one or two syllables, with a further 12% being partial repetitions of what the agent had just said.

An agent that treats all of them as interruptions stops dead every time the caller agrees with it. The resulting behaviour is unmistakable: the agent gets three words into a sentence, halts, says "sorry, go ahead", the caller says nothing because they were not trying to say anything, and the agent restarts. This is the most common shipped defect in voice agents, more common than echo, because echo is caught in the office and this one is only caught when real people who say "mm-hm" start calling.

The discriminator is not volume, and it is not confidence from the voice detector, since a clear "mm-hm" scores as speech with high confidence. What separates the two is duration and content. A backchannel is short, it does not continue, and its words come from a small closed set. A real interruption either keeps going past the length of any backchannel, or its first word is not in that set.

BACKCHANNELS = {
    "mm", "mhm", "mm-hm", "uh-huh", "yeah", "yep", "yes", "right",
    "ok", "okay", "sure", "got it", "i see", "of course",
}
MIN_INTERRUPT_MS = 480      # sustained speech required to count
GRACE_AFTER_END_MS = 250    # a backchannel may keep going

def is_interruption(speech_ms: int, interim_text: str) -> bool:
    if speech_ms >= MIN_INTERRUPT_MS:
        return True
    words = interim_text.lower().strip(" .,!?").split()
    if not words:
        return False                      # energy with no words yet: wait
    if " ".join(words) in BACKCHANNELS:
        return False                      # agreement; keep talking
    return len(words) >= 2                # two real words is a bid for the floor

The MIN_INTERRUPT_MS value buys accuracy with latency, and the trade is direct: it is 480 milliseconds during which a caller who genuinely wants to interrupt is still being talked over. Setting it much below 400 starts admitting long backchannels like "yeah, yeah, okay", and much above 600 the caller repeats themselves because they think the agent did not hear. Where you land depends on how tolerable a false stop is in your product, which is a different question for a hands-free driving assistant than for an agent reading back a delivery address.

There is a middle response between stopping and ignoring that is worth knowing about: ducking, where the agent drops its output level by 6 to 10 dB when overlapping speech is detected and restores it if the speech turns out to have been a backchannel. It makes the ambiguous half second less unpleasant for the caller and it buys the classifier time. It also does not work on a telephone leg where you control neither the mixing nor the level, so treat it as an enhancement for clients you own.

What this chapter has not done#

Everything above is about the decision. The microphone stays open, the canceller keeps the agent's own voice out of the detector, and the gate distinguishes a caller taking the floor from a caller agreeing. At the end of it you have one boolean, on the order of half a second after the caller started speaking, that says an interruption is real.

Acting on that boolean is a separate problem, and it is larger than it looks. Stopping playback is a single call. The response that produced the playback is still being generated, a tool the model dispatched may already be writing to a database, several seconds of synthesized audio are sitting in buffers on both sides of the network, and events from the turn you just killed will keep arriving for a while — and the conversation history now claims the agent said four seconds of things the caller heard one second of.