Partials are hypotheses, finals are commitments

The scheduling voice agent's most common complaint, once the booking errors were understood, was that it felt slow. Callers talked over it, repeated themselves, and hung up at roughly twice the rate of the human queue. Instrumenting one turn end to end — from the moment the caller stopped speaking to the moment the first syllable of the reply reached their ear — gave 1,930 milliseconds, against a conversational norm where a gap over about 700ms reads as hesitation.

Look at that table and one line jumps out as free money. The recogniser has a hypothesis of the caller's words long before it declares them final — a partial that already reads i need to move my appointment to the fourteenth arrives while the caller is still speaking, and 700ms of the budget is spent waiting for the recogniser to become confident about text you already have. An engineer on the team proposed firing the LLM call on the last partial and recovering all of it. The proposal is sound, the naive implementation is dangerous, and the difference between the two is worth being precise about.

What a streaming recogniser is actually doing between partials#

A streaming decoder consumes audio in chunks, typically 100ms to 320ms, and after each chunk re-runs its search with the audio it has so far. That search carries a set of competing candidate transcripts forward together — the paths through the hypothesis space, each with a running score — and prunes the weak ones as evidence accumulates. What it emits after each chunk is the current best path, which is a guess conditioned on incomplete evidence. Later audio changes the acoustic likelihood of earlier frames — a following consonant disambiguates a preceding vowel, a following word makes one language-model path far more probable than another — so the decoder is free to rewrite words it emitted 400ms ago. That rewriting is the whole reason partials are cheap.

A final is a different kind of statement. It is emitted when the endpointer decides the utterance is over, usually after a fixed duration of non-speech, and it means the decoder has committed: no further revision, inverse text normalisation applied so that nine forty becomes 9:40, punctuation and capitalisation inserted, word timings attached. Most APIs expose this as a boolean on each event, and several expose more than one flavour of it — Deepgram distinguishes is_final on a segment from speech_final at the endpoint and offers a separate UtteranceEnd event derived from word timings, while Google's streaming API attaches an is_final flag and, on some models, a stability score between 0 and 1 for interim results.

The gap the speculation is buying is mostly the endpointer's silence threshold, and that threshold is a deliberate accuracy trade rather than laziness. Set it to 500ms and a caller who pauses mid-sentence to think — I need to move it to… the fourteenth — gets their turn cut in half, and the agent answers a fragment. Set it to 1,000ms and every turn feels sluggish. This is the parameter teams tune first and it has a hard floor, because natural mid-utterance pauses in spontaneous speech routinely run 400ms to 800ms.

Measuring how often the last partial survives#

Whether speculation is safe is an empirical question about your audio, and the measurement takes an afternoon. Log every partial with its timestamp, log the final, and compute the fraction of turns where the last partial's normalised text equals the final's normalised text. On the scheduling product's telephony, over 4,100 turns, the last partial matched the final exactly 74% of the time.

The 26% that changed are the interesting population, and their changes were not evenly distributed. About three fifths of them were pure extensions — the final contained everything the partial had plus one to three more words at the end, because the caller's last words were still being decoded when the partial was emitted. The rest were rewrites, where a word already present in the partial changed. Rewrites concentrated on exactly the tokens chapter three identified as expensive: nine forty becoming nine forty five, the fourteenth becoming the fourth of the month, I can pay becoming I can't pay. This is not a coincidence, because a number or a negation at the end of a turn is precisely where the decoder has the least right context to work with — the audio that follows a word, which is what usually disambiguates it — and it is also where the endpointer fires.

Acting blindly on the last partial therefore means accepting a wrong text on roughly a quarter of turns, and the wrongness is biased toward the slot values. Against a baseline of one wrong booking in five, that is not a trade anyone should make.

Speculate, then validate#

The version that works keeps the latency win and discards the risk, at the price of some wasted inference. Issue the LLM call on the last partial, keep it cancellable, and treat its result as provisional until the final arrives and matches. Two constraints make this safe. The speculative turn must be free of side effects, which means the model may plan a tool call but the orchestrator does not execute it until the text is confirmed. And the speculative call must be genuinely cancellable at the transport level, so a superseded turn stops consuming budget rather than racing the real one.

import asyncio

class SpeculativeTurn:
    def __init__(self, agent):
        self.agent = agent
        self.task: asyncio.Task | None = None
        self.speculated_on: str | None = None

    def on_partial(self, text: str, stable_repeats: int) -> None:
        # Only speculate on a partial that has stopped moving and is long
        # enough to carry an intent; otherwise we churn on every chunk.
        if stable_repeats < 2 or len(text.split()) < 6:
            return
        if normalise(text) == self.speculated_on:
            return
        self.cancel()
        self.speculated_on = normalise(text)
        # plan_only=True: the agent may choose a tool, but must not call it.
        self.task = asyncio.create_task(self.agent.plan(text, plan_only=True))

    async def on_final(self, text: str):
        if self.task and normalise(text) == self.speculated_on:
            plan = await self.task              # already in flight, often done
            hit.inc()
        else:
            self.cancel()
            plan = await self.agent.plan(text, plan_only=True)
            miss.inc()
        return await self.agent.execute(plan)   # side effects happen here only

    def cancel(self) -> None:
        if self.task and not self.task.done():
            self.task.cancel()
        self.task = None

With a 74% hit rate, the median turn loses 700ms of the 1,930ms budget and the tail keeps it, so the distribution becomes bimodal rather than uniformly faster — worth remembering when you set an alert threshold on p95 turn latency. The cost is one extra LLM call on 26% of turns plus the occasional wasted call from a partial that stabilised and then moved again; at roughly $0.004 per turn that is under a tenth of a cent per conversation, which is cheap next to 700ms.

There are other places to find the same time, and they trade differently.

The last row is the conservative version of the same idea and is underused. Everything in the prompt that precedes the caller's latest words — system instructions, conversation history, retrieved context — can be sent and cached while the caller is still speaking, so that when the final arrives only the last few dozen tokens need to be processed before generation begins.

Two places the same revision shows up outside the LLM call

Barge-in makes the speculation machinery load-bearing rather than optional. When the caller starts speaking over the agent's reply, the orchestrator has to stop playback, drop the partially delivered response, and start a new turn — and if a speculative call from the interrupted turn is still in flight, it will complete a few hundred milliseconds later and try to hand back a plan for a conversation state that no longer exists. Keying every speculative task to the turn identity it was created under, and discarding results whose key no longer matches the current turn, is the same guard as the text comparison above applied to a different kind of staleness.

The other place is the screen. Products that display a live transcript — captions, agent-assist panels, meeting UIs — are rendering partials, and rendering them naively produces text that visibly rewrites itself several times a second, which readers find harder to follow than a slightly delayed transcript that stays put. The usual treatment is to split the display at the stable prefix: render the words that have not changed across the last two or three partials in normal weight, render the unstable tail in grey, and only commit the line when the final arrives. Some APIs help directly by marking word-level stability or by exposing a stability score you can threshold; where they do not, tracking the longest common prefix of consecutive partials costs nothing and gives the same boundary.

Measure the two numbers under conditions you will actually have#

Time to first partial and time to final are the operational metrics, and both must be measured against a known audio timeline rather than against wall clock at the point of receipt, or you will attribute network delay to the recogniser and vice versa. Feed a recorded file through the socket in real time, stamp each frame with its position in the audio, and record the offset between the audio timestamp of a word and the wall-clock arrival of the event containing it.

# Replay a file at real-time pace and record arrival offsets.
FRAME_MS = 20
t0 = time.monotonic()
for i, frame in enumerate(frames):
    target = t0 + (i * FRAME_MS) / 1000
    await asyncio.sleep(max(0, target - time.monotonic()))
    await ws.send(frame)

# On each event: offset = (arrival wall clock - t0) - audio_position_of_last_word
# localhost, no impairment:      TTFP 180ms   TTF 690ms
# 4G profile, 120ms RTT, 1% loss: TTFP 340ms   TTF 910ms

Run that over an impaired link, because the difference is large and it is the number your callers experience. The pathological case worth checking for explicitly is a vendor or configuration that buffers: some batch-oriented endpoints accept a stream and return nothing until several seconds of audio have accumulated, which produces an excellent WER and a time to first partial of 5.2 seconds. A recogniser can be accurate and unusable at the same time, and only one of those two properties appears on a benchmark page.

What streaming is not#

Partials are not the output of a smaller or worse model; they come from the same decoder given less evidence, and the quality difference is about available context rather than capacity. There is a real accuracy gap between streaming and batch transcription of the same audio, usually one to three percentage points of WER, and it comes from the streaming decoder having no right context and no opportunity for a full-utterance rescoring pass, in which the candidates the search kept alive are re-ranked once the whole utterance is available and a stronger language model can be run over each of them end to end. If a use case tolerates delay — call summarisation, compliance review, meeting notes — running batch transcription over the recording afterwards is both cheaper and more accurate, and a great many products end up doing both: streaming for the live experience, batch for the record of what was said.

Latency is also not a substitute for the metrics of the last three chapters. A system can hit 180ms to first partial and still corrupt every date, and a fast wrong answer is worse than a slow one when it results in a booking.

The revision behaviour in this chapter — a label emitted early, then rewritten once more audio arrives — is not unique to words. It is the defining characteristic of two other outputs your pipeline is probably treating as facts.