First token, first useful content, and last token are three different waits

The streaming change went live on a Tuesday and the engineer who shipped it had the numbers ready before anyone asked. Measured in the browser, across roughly 4,000 real sessions, time to first token fell from 2.1 seconds to 300 milliseconds. The model was the same model, the prompt was the same prompt, and the server did the same work; the only difference was that the response now arrived as a token stream rendered into the answer pane rather than as one JSON body at the end. The expectation was that the steady trickle of support tickets saying some version of "the assistant is slow" would fall. A week later the rate was 34 tickets against a trailing average of 36, which is inside the weekly noise, and the team's first theory was that users were reporting a stale impression from before the change.

Reading twenty of those tickets kills that theory. The assistant answers coverage questions for call-centre agents, and the questions all have the same shape: is a member on the Bronze plan covered for an out-of-network MRI with prior authorisation. The answer the model produces restates the plan, summarises the relevant clause of the policy document, lists the conditions that apply, and says whether the member is covered in the final sentence. A typical answer runs 380 tokens and the deployment sustains about 42 tokens per second, so the whole thing takes 9.0 seconds to generate. The verdict sits at roughly token 360, which is 8.6 seconds in. Before streaming, the agent on the call waited 9.1 seconds and then saw everything at once. After streaming, they saw the first word at 0.3 seconds and the answer they needed at 8.6. The number that improved by 85% was not the number anyone was waiting on, and the number they were waiting on improved by 5%.

The distance between the number that moved and the number anyone was waiting on is what this topic is about. Latency is the delay between asking a system for something and receiving it, and latency optimization is the engineering work of shrinking that delay for whoever is on the other end — by changing how much prompt the model has to read before it can start, what it is asked to produce and in what order, how many calls run one after another, when the work is begun relative to when it is wanted, and what the interface shows while it runs. The work exists as its own discipline here because a language model is unusually slow by the standards of everything else in a backend. A database query returns in single-digit milliseconds; a model writing a few hundred tokens — the small pieces of text it emits one at a time, roughly a short word each — takes seconds, because every token needs another pass through the model and those passes are strictly ordered. You meet the work in three recognisable places: a support ticket saying the assistant is slow, a product requirement that arrives as "something on screen within 400 milliseconds", and a comparison slide ranking two deployments by a single latency figure. In all three the first move is to ask which delay is meant, because one request contains several and they do not move together.

The tickets themselves were more specific than the summary metric suggested. Several said the assistant "starts talking and then takes forever to get to the point", which is a description of exactly what changed rather than a complaint about something unrelated. Streaming had made the wait more legible without making it shorter, and a wait you can watch is not obviously better than a wait you cannot; a call-centre agent with a customer on the line reading nine seconds of preamble is aware of every one of those seconds in a way they were not when the screen was simply blank.

Three clocks running on the same request#

A single request produces at least three separate waits, and they can move independently of each other in either direction.

Time to first token is the interval between the request leaving the client and the first content token being painted. It covers admission, queueing, prompt processing, and the network on the way back. It is the milestone that streaming exists to reduce, and it is the one that shows up in vendor comparisons because it is easy to measure and does not depend on knowing anything about the task.

Time to first useful content is the interval until enough of the response is on screen that the user can act on it. There is no protocol-level event for this; it is a property of the task and of how the answer happens to be ordered. On the coverage assistant it is the moment the word "covered" or "not covered" appears. On a code-completion feature it is the first line of the suggested body. On a summarisation panel it might be the first bullet. Nothing in the transport knows about it, which is why it is the milestone teams most often fail to instrument.

Completion time is the interval until the last token arrives. It is what the client waits for whenever the output is consumed as a whole rather than read progressively, and it is what a downstream service waits for when the model's output feeds another step in a pipeline.

Here is the same request seen as a stream, with wall-clock offsets from the moment the request was issued. The data: lines are what actually crosses the wire on a server-sent-events connection, which is an ordinary HTTP response the server holds open and writes to a line at a time as content becomes available; the offsets are from the client's own clock, which is the only clock that sees all three milestones.

t=0.000s   POST /v1/messages  (stream=true)
t=0.291s   data: {"type":"content_block_delta","delta":{"text":"For a"}}      <- first token
t=0.315s   data: {"type":"content_block_delta","delta":{"text":" member"}}
t=0.338s   data: {"type":"content_block_delta","delta":{"text":" on the"}}
   ...     (restates plan, quotes clause 4.2, lists three conditions)
t=8.613s   data: {"type":"content_block_delta","delta":{"text":" is covered"}} <- first useful content
t=8.994s   data: {"type":"message_stop"}                                       <- completion

The gap between the first marked line and the second is 8.3 seconds of text that the call-centre agent scrolls past. It is not filler — a compliance reviewer would want the clause citation, and the assistant is right to produce it — but the person holding the phone is not reading it.

Of the three, only the first and third correspond to events the transport can see. The stream emits a first delta and a stop message, and any HTTP client can timestamp both without knowing what the request was for. The middle milestone has no event, no header, and no library that will find it, because "useful" is a fact about the task rather than about the bytes. That asymmetry is why first-token time became the number everyone reports: it is the one that a generic benchmark harness can compute for any model on any prompt, and it inherited the authority that comes with being comparable across vendors. Nothing about being easy to measure makes it the number your users are waiting on.

Streaming itself arrived in these products by way of the chat interfaces that popularised them, where the token-by-token reveal was as much an aesthetic decision as a latency one. It reads as a system thinking aloud, and it works because the reader consumes the answer in the order it is produced. Carrying that same transport into an interface where the user is hunting for one fact inside the output keeps the aesthetic and drops the mechanism that made it useful.

Which clock the task is actually on#

The mistake that produces a dashboard like the one above is treating "perceived latency" as a single quantity that streaming improves. Perceived latency is whichever of the three milestones the task depends on, and which one that is follows from how the output is consumed.

Three of those five rows are cases where a first-token improvement is invisible. That is not an argument against streaming, which costs almost nothing to enable and is strictly better than not streaming in every row of the table. It is an argument against reading a first-token improvement as evidence that anything a user cares about got faster.

The coverage assistant was fixed without touching the model, the infrastructure, or the streaming code. The prompt was changed to require the answer to open with a verdict line, then give the reasoning:

COVERED: yes, with prior authorisation on file.

For a member on the Bronze plan, an out-of-network MRI is reimbursed at 60%
under clause 4.2 provided that ...

Time to first useful content went from 8.6 seconds to about 0.4, because the verdict is now the first eight tokens rather than the last eight. Completion time did not move — the answer is the same length and the deployment generates at the same rate — and neither did cost. The support-ticket rate for slowness fell to single digits over the following three weeks. The entire win came from asking which milestone the task was on and then reordering the output so that milestone arrived early, which is a prompt change rather than a latency change in any mechanical sense.

That reordering is not free in general, and it is worth being honest about the trade. Asking a model to commit to a verdict before it has written out the reasoning removes the reasoning's ability to influence the verdict, and on genuinely borderline cases that costs accuracy. On this workload it was measured on a 200-case eligibility set and the disagreement rate against the reason-first ordering was 1.5%, all on cases a human reviewer also called ambiguous, which the team accepted. On a workload where the reasoning does real work — multi-step arithmetic, or anything where the model needs to look something up mid-answer — verdict-first is a meaningful quality regression.

The way out of that trade is to keep the reasoning first for the model and put the verdict first for the reader, which requires the two to be separate fields rather than separate sentences. Streaming a structured response makes that possible, because the client can render blocks as they close rather than as they arrive:

{"type":"content_block_start","index":0,"content_block":{"type":"text","name":"analysis"}}
... 340 tokens of clause reading, rendered into a collapsed panel ...
{"type":"content_block_stop","index":0}
{"type":"content_block_start","index":1,"content_block":{"type":"text","name":"verdict"}}
{"type":"content_block_delta","index":1,"delta":{"text":"COVERED: yes"}}

The verdict still arrives at 8.6 seconds under this arrangement, so it does not help the coverage assistant on its own. It helps when the analysis can be shortened or moved off the critical path — generated once per member rather than once per question, or produced by a separate call whose result is cached — because the structure gives you somewhere to put that split. The ordering decision and the caching decision are the same decision seen from two sides, and neither is available until you have said which milestone the task stops on.

What this does not fix#

Separating the three milestones is a measurement discipline, and measurement disciplines do not make anything faster on their own. Every second in that trace is still being spent. If the pipeline is slow because the prompt carries 40,000 tokens of retrieved context, knowing that the task is governed by completion time tells you where to look and nothing more.

It is also not the same idea as a loading skeleton or a progress indicator, though the two get filed together under "perceived performance". A skeleton changes what the user looks at during a wait that has not changed. Reordering the output changes when the content they need actually exists. Both are legitimate, and they fail differently: a skeleton that promises content which then takes nine seconds to arrive erodes trust in the indicator, whereas an early verdict is either right or wrong on its own terms.

The third milestone deserves the same caution in the other direction. Completion time is the right target for a pipeline stage, and it is a poor target for a conversational surface, because optimising it tends to mean shortening the output and a shorter answer is a different product rather than a faster one. An assistant that went from 380 tokens to 120 would post a 68% improvement in completion time and would be answering a narrower question. Latency numbers that were achieved by producing less have to be reported next to whatever measures whether the smaller answer still does the job, and the discipline of doing that consistently is the subject of the last chapter here.

Finally, none of the three milestones is a stand-in for the others when you are choosing between systems. A deployment can win decisively on first token and lose decisively on completion, and unless you know which milestone your task is on, the comparison you run will rank them in an order that has nothing to do with what your users wait for. That ranking problem — two models, two clocks, and an answer that flips depending on which one you read — is concrete enough to work through with real numbers, which is where this goes next.