Covering a 1.4-second tool call without painting yourself in

The order lookup is not going to get faster this quarter. It calls a warehouse management system through an integration layer, it takes 1.4 seconds at the median, 5.8 seconds at the 95th percentile, and it returns an error on 0.7% of calls. Those are the real numbers and they are not a bug to be traced with the tooling from the last chapter; the query genuinely takes that long. The caller has just read out an order number and is now listening to silence that will last four times the length of any other silence in the conversation.

The standard response is a filler phrase — "let me check that for you" — and it is the right instinct executed wrongly more often than not. The wrongness has two parts, and the second one is the expensive one.

A filler you synthesize is a filler that arrives late#

Generating "let me check that for you" through your text-to-speech service costs a time to first chunk of 150 milliseconds at the median and 400 at the tail, from the budget in chapter 1, plus the model turn that decided to say it. So the sequence becomes: caller stops speaking, 300 milliseconds of endpointing, 350 milliseconds for the model to emit the filler text, 150 milliseconds to synthesize it, and only then does sound arrive — 800 milliseconds after the caller stopped, which is roughly the whole silence you were trying to cover. You have spent the budget to announce that you are about to spend the budget.

The fix is to render the fillers once, at build or deploy time, and hold them as audio. They are short, fixed, and there are not many of them, so a set of a dozen at around 1.2 seconds each is under 300 kilobytes in the codec you need. Render them in the same voice as the agent, at the same sample rate and in the same encoding as the live path — μ-law at 8 kHz for a telephone leg, per chapter 6 — so that no resampling or transcoding happens between the pre-rendered clip and the model's own audio, since that transition is audible when the two differ.

FILLERS = load_prerendered("fillers/", codec="pcmu")  # {name: bytes}

async def with_cover(tool_coro, player, fence_id: int):
    task = asyncio.create_task(tool_coro)
    elapsed = 0.0
    for delay, clip, required in COVER_PLAN:
        remaining = delay - elapsed
        done, _ = await asyncio.wait({task}, timeout=remaining)
        if done:
            return task.result()                      # returned before this tier
        elapsed = delay
        await player.play(FILLERS[clip], turn_id=fence_id)
        if required == "deadline":
            task.cancel()
            raise ToolDeadline(elapsed)
    return await task

The first entry in COVER_PLAN carries the detail that separates a good implementation from an irritating one: the filler is scheduled, not played immediately. If the tool returns in 320 milliseconds — which it does on a cached lookup — the caller hears the answer directly, with no "let me check" in front of it. Announcing a wait that does not happen adds a second of conversation for nothing, and an agent that does it on every tool call sounds like it is stalling for a living.

The tiers, and what each one is allowed to claim#

Writing the cover plan as a table forces the overrun and failure paths to exist, because their rows are empty until someone fills them.

The "must not say" column is the one that stops the agent from painting itself into a corner. Language models asked to produce a filler will happily generate "great, I've found your order" before the tool has returned anything, because that is a natural-sounding thing to say at that point in a conversation, and it is a claim the system cannot honour 0.7% of the time. Pre-rendered clips make this a review problem rather than a runtime risk: someone reads the twelve strings once and confirms none of them assert a result. If you instead let the model produce filler text live, the constraint has to be in the prompt and it will be violated occasionally, which is a strange risk to accept for a phrase that never changes.

The 8-second deadline is doing something the other tiers are not. It converts an unbounded wait into a bounded one and hands the conversation back with an option — a callback, an SMS with the details, a transfer to a human, or simply the offer to try again. Without it, a hung integration produces a call where the agent said "let me pull that up" and then nothing at all, and the caller hangs up after twenty seconds having been promised an answer. That is the specific shape of "painting yourself in": a commitment made in the first tier that no later tier can discharge.

Both paths into Recover deserve the same treatment, and the reason is that from the caller's side a timeout and an error are the same event. What differs is what you record: a deadline should be logged with the elapsed time so that a systematically slow integration is visible in the same telemetry as everything else, and the conversation history should note that a lookup was attempted and did not complete, so that a later turn does not have the model confidently referring to order details it never received.

The filler is part of the turn, with everything that implies#

A pre-rendered clip is audio the agent is playing, which means it is subject to every mechanism from chapters 3 and 4. The microphone stays open while it plays, so a caller who says "actually it's the other order" 400 milliseconds into "let me pull that up" is interrupting, and the interruption has to cancel the tool call along with the playback — which is exactly the effect classification from the cancellation path, since a read-only lookup can be abandoned freely. The clip is enqueued with the current turn id so that a filler from a cancelled turn cannot play over the next one, and the history entry for the filler is truncated to what was heard like any other assistant audio.

There is also an ordering subtlety worth getting right. When the tool result arrives while a filler is still playing, letting the clip finish costs up to a second of pointless delay, and cutting it dead mid-word sounds like a glitch. The workable middle is to keep clips short enough that finishing them is cheap, and to cut only at a word boundary if you have the timings. A 1.2-second maximum on filler length is a good constraint for exactly this reason.

Twelve clips, not one#

The reason to render a dozen rather than one is that a fixed phrase repeated on every lookup in a call becomes the most memorable thing about the agent, and callers report it as a tic. Rotating through a small set without repeating within a call is enough; the set does not need to be large, because a caller rarely triggers more than three or four lookups. Keep them varied in length as well as wording, since identical duration is itself a pattern a listener picks up.

The second-tier clips are worth writing differently from the first, and not merely as a longer wait. A tier-1 filler covers a delay the caller has not yet noticed, so it should be short and neutral. By 3 seconds the caller knows something is slow, and a clip that acknowledges it — naming the system, or saying the lookup is taking longer than usual — reads as competence, whereas a second cheerful "just a moment" reads as a loop. This is the same asymmetry that makes a progress bar better than a spinner, and it costs nothing beyond writing four extra strings.

Whether any of it works is measurable with the telemetry from the previous chapter, and the metric is not latency. Count the calls where the caller spoke during a tool wait — "hello?", a repetition of the order number, or any speech at all in the window between dispatch and result — and track it as a rate per tool-covered turn. That number is the caller telling you they thought the call had dropped, and it should fall sharply when covering is introduced and stay near zero afterwards. Watch the abandonment rate for the same window too: callers who hang up during a lookup are the failure this whole mechanism exists to prevent, and if that rate is unchanged after adding fillers, the timing of the tiers is probably wrong rather than the idea.

What a filler does not do#

It does not make the call faster, and treating it as a latency fix is how a team ends up covering the wrong thing. The investigation in the previous chapter found long calls with a 1.8-second model response, and no filler helps there: a phrase that occupies 900 milliseconds in front of an 1,800-millisecond wait leaves 900 milliseconds of silence and adds a second to a turn that was already too long. Fillers are for delays that are known, bounded and caused by something outside the conversation, which is a much narrower category than "delays".

It is also not the first thing to try. Two other moves usually beat it. Dispatching the lookup speculatively, from the partial transcript, the moment the caller has said a plausible order number means the 1.4 seconds overlaps with the endpointing wait and the model turn — often reducing the visible delay to under half a second, at the cost of occasionally querying for a number the caller then corrected. Caching helps in the cases where a caller asks about the same order twice in a call, which is common. Both of these change the number; the filler only changes how the number feels, which is worth doing after the number has been attacked rather than instead.

Finally, the whole apparatus assumes there is a place in your architecture where the agent's textual intent exists before audio is produced — a tool call you can see dispatched, a decision you can gate on, a fixed string you can render in advance and check against a list of things it must not claim. That assumption has been quietly load-bearing since the first chapter, and there is a class of model where it does not hold.