Barge-in is a cancellation problem, and history is part of it

The caller interrupts 1.2 seconds into a reply that was going to run 4 seconds. The gate from the previous chapter fires, the client stops playback, and the caller says what they wanted to say. Two turns later the agent says "as I mentioned, I've moved you to Thursday at 3:15" — and the caller has never heard the words "Thursday" or "3:15", because those were at second 2.8 of a reply that was cut off at second 1.2. The caller says "you didn't mention anything", the agent insists politely that it did, and the call goes downhill from a disagreement about a fact only one party has access to.

The transcript in the model's context said the assistant produced the full four seconds, because that is what generation produced. The audio the caller received was one and a fifth seconds long. Every subsequent turn was reasoned over a conversation that did not happen.

Everything that is in flight when the boolean flips#

At the moment barge-in is detected, a voice turn has state spread across at least six places, most of them not on the machine that noticed. Enumerating them is the whole job, because each one has to be dealt with individually and none of them is handled by stopping the speaker.

The last row is the one teams forget, and it is forgotten for a structural reason: it is the only item on the list that causes no immediate symptom. Everything else fails loudly within a second — audio keeps playing, a duplicate booking appears, the agent talks over itself. History corruption fails three turns later as a confused conversation, or a week later as an agent that references a price it never quoted, and by then nobody is looking at the interruption.

The Realtime API exposes the fix directly, and the parameter that matters is a millisecond offset into the assistant's audio:

{
  "type": "conversation.item.truncate",
  "item_id": "item_D7f2",
  "content_index": 0,
  "audio_end_ms": 1200
}

That instructs the session to discard everything after 1,200 milliseconds of that assistant item, including the corresponding text, so the model's view of what it said matches what the caller heard. The number must come from the client's playback position, not from your server's send position, because those differ by the whole downlink and jitter buffer — typically 100 to 250 milliseconds, which is comfortably a clause. If you are building the pipeline yourself rather than using a hosted session, the same operation is a slice of your own message list, and the input to it is the same measurement.

def truncate_assistant_turn(history, item_id, played_ms, ms_per_char=55):
    """Trim the assistant message to the audio the caller actually heard."""
    item = history.find(item_id)
    if item.audio_duration_ms <= played_ms:
        return                                    # heard all of it
    cut_chars = max(0, int(played_ms / ms_per_char))
    heard = item.text[:cut_chars].rstrip()
    item.text = heard + " —"                      # marks a cut-off utterance
    item.audio_duration_ms = played_ms
    item.was_interrupted = True

The character estimate in that function is crude, and a better pipeline keeps word-level timings from the synthesizer so the cut lands on a word boundary. Crude beats absent by a wide margin: the model reading "I've moved you to —" will not later claim it announced a time.

Fencing the events from a turn that is dead#

Cancellation in a streaming system is not instantaneous, and the events already in flight do not know they have been orphaned. A response.cancel sent at time T does not prevent deltas — the incremental chunks of text or audio in which a streamed response arrives — that were emitted at T minus 30 milliseconds from arriving at T plus 40, and if your handler routes any delta with audio into the playback path, the agent will emit a fragment of the cancelled reply after the caller has started their new sentence.

The standard remedy is the same one distributed systems use for a leader that has been superseded: give each turn a monotonically increasing identifier, tag everything with it, and drop anything whose tag is not current.

class TurnFence:
    def __init__(self):
        self.current = 0

    def begin(self) -> int:
        self.current += 1
        return self.current

    def accept(self, turn_id: int) -> bool:
        return turn_id == self.current

async def on_event(evt, fence: TurnFence, player):
    if not fence.accept(evt.turn_id):
        metrics.incr("realtime.stale_event", tags={"type": evt.type})
        return                                   # orphaned; silently discard
    if evt.type == "response.output_audio.delta":
        await player.enqueue(evt.audio)

Counting the discards rather than dropping them silently matters more than it looks. A steady trickle of stale events after each interruption is normal and tells you the fence is doing its job; a sudden increase means either your cancel is not reaching the provider or a component is generating for a turn nobody is listening to, and both are billable.

The whole lifecycle, with cancellation as a first-class state rather than an exception path, is small enough to hold in one picture:

The transition into Cancelling from both Generating and Speaking is deliberate. An interruption that arrives before any audio has played still has a model request to abort and possibly a tool in flight, and it still needs the history handled — with audio_end_ms of zero, which means the assistant item is removed entirely rather than truncated.

Tools do not roll back because you stopped talking#

The row in that table with the widest blast radius is dispatched tool work, because a tool call is the one part of a voice turn that changes state outside your process. The agent loop covered in the earlier material treats tool results as things that come back and get appended; here the loop can be terminated between the dispatch and the result, and what happens to the effect is entirely your problem.

Sort tools into three categories before you need them. Read-only lookups — availability, order status, account balance — can be abandoned freely; you cancel the request to save money and forget it happened. Idempotent writes keyed by a request identifier can be allowed to complete, because a repeat of the same call collapses into the same effect, and it is usually cheaper to let a 300-millisecond write finish than to build a cancellation path for it. Non-idempotent effects — sending an SMS, charging a card, booking a slot that consumes inventory — need either a two-phase shape where the turn holds a reservation that expires unless confirmed, or a compensating action — a second operation that undoes the first after the fact, such as voiding a charge or releasing a slot — that you run on cancellation.

async def cancel_turn(turn_id: int, played_ms: int, session):
    await session.send({"type": "response.cancel"})
    await session.player.flush()
    truncate_assistant_turn(session.history, session.active_item, played_ms)
    for job in session.tools.in_flight(turn_id):
        if job.effect == "read_only":
            job.abort()
        elif job.effect == "idempotent":
            job.detach()                     # let it finish; result is discardable
        else:
            await job.compensate()           # release hold, void, or cancel
    session.fence.begin()

The failure this prevents is specific and expensive. A caller says "book me for Thursday", the model dispatches create_appointment, and 400 milliseconds later the caller says "actually, no, Friday". Without compensation the Thursday slot is now taken, no record of it exists in a conversation the model can reason about, and the clinic finds out when two patients arrive at once. With a hold-and-confirm shape the Thursday reservation expires in 60 seconds and nobody is inconvenienced.

The utterance that caused all this must survive it#

There is a requirement that runs through every step above and is easy to violate while satisfying the rest: the speech that triggered the interruption is the next user turn, and it started roughly half a second before you decided anything. If your cancellation path clears the input audio buffer along with everything else — which is a natural thing to write, since clearing buffers is what the rest of the function does — you lose the beginning of the caller's sentence, and the transcript arrives as "day at three instead" with the first word missing. The model then asks the caller to repeat themselves immediately after cutting them off, which is the worst possible pairing.

So the input path and the output path get flushed on different schedules. Output buffers are cleared; the input buffer is left alone, and the frames captured during the barge-in detection window — including the prefix_padding_ms of audio from before the detector fired — flow into the new turn. In a hosted session this is the default behaviour and the way to break it is to send input_audio_buffer.clear as part of your cancel handler. In a pipeline you assembled yourself, it means the microphone stream never stops being consumed, even while a cancel is in progress; the detector, the transcriber and the buffer all keep running through the transition, and only the speaking side is torn down.

A related detail is that the cancelled turn's user message is already in history and should stay there unchanged. The caller did say the thing that prompted the four-second reply; what did not happen is the reply. Deleting the user turn to "clean up" the exchange produces a conversation where the agent answers a question that was never asked.

Cancellable turns make speculation cheap#

Once cancellation is reliable, a latency option opens that is otherwise reckless. Preemptive generation starts the model turn on the current partial transcript before the endpointing decision from chapter 2 has committed, on the bet that the caller has finished. When the bet is right, you have already spent the model's time-to-first-token during the silence window, and the reply begins 250 to 400 milliseconds earlier than it otherwise would. When the bet is wrong, the caller keeps talking and you cancel the speculative turn before any audio is played.

The cost is exactly two things. You pay for inference you throw away, which on a chained pipeline with a 1,500-token prompt and 40 tokens generated before the cancel is a small fraction of a cent per false start, and at a 30% false-start rate over a six-turn call amounts to roughly two discarded generations per call. And you take on a hard requirement that every turn be cancellable in the full sense described above, because speculation makes cancellation the common path rather than the rare one — a speculative turn is cancelled on nearly a third of turns, so any leak in the fence, any tool dispatched during speculation without a compensation path, any history item left untruncated, now happens constantly instead of occasionally.

That is the honest accounting, and it is why the feature is a late optimization. It is not a way to make an unreliable cancellation path acceptable, and it does not reduce the model's actual latency; it moves the spend earlier and pays for the guesses that were wrong. Providers expose it as a knob — LiveKit's turn-taking documentation describes preemptive generation alongside the endpointing settings it interacts with — and the knob is safe to turn on only after the state machine above is real.

The part that has been assumed all along#

Every mechanism in this chapter assumed the flush actually flushes. Playback stops when told, queued audio can be dropped, and the cancel message reaches the far side quickly enough to matter. Those assumptions are properties of the transport carrying the audio, and they are not equally true of all of them. A protocol that guarantees ordered delivery will happily keep delivering the audio you asked it to abandon, and a connection that stalls for 300 milliseconds under packet loss will deliver your cancel late and your caller's interrupting speech later still.