Time to first useful content, not time to first token

The dashboard for Atlas showed a p50 time to first token of 300 milliseconds and a p50 completion of 6.0 seconds, with 22% of answer requests abandoned — the handler navigated away, re-asked, or closed the pane before the answer finished. The proposal on the table was to move to a faster model, at roughly twice the per-token cost, on the argument that six seconds is too long for an interactive product.

The first token in question was the word Let in the sentence Let me look through the claims handling manual for the appeals timescale. Nothing in that sentence is usable. It restates the question, it announces an intention, and the handler who reads it knows nothing they did not know when they pressed enter. Time to first token was measuring the moment the system started talking, and the number the abandonment was tracking was the moment it started saying something.

Instrumenting that separately meant defining useful operationally, and the structured stream from chapter 2 made it mechanical: useful content begins at the first token of the headline field. On the same traffic, p50 time to first useful content was 4.1 seconds.

Where the four seconds actually went#

Two of those bars are pure overhead. The preamble costs 530ms and says nothing; the restatement at the start of the second call costs another 1,010ms and says the same nothing in different words. Together they are 1.54 seconds of the 4.1-second time to first useful content, and removing them is a prompt change plus a schema constraint, which is to say free. The rerank bar is 1.4 seconds spent on a cross-encoder scoring 40 candidates, and dropping the candidate set to 15 with a smaller model took it to 310ms at a cost of 1.2 points of retrieval recall on the evaluation set.

A faster model changes the three model bars. Of the 6.0 seconds, roughly 3.9 is generation, and a model with half the latency profile removes about 1.9 seconds — real, and less than the 2.6 seconds available from the two free changes plus the reranker, at zero additional inference cost. The honest answer to the proposal is that the model is the fourth thing to try, and that the case for it should be made after the pipeline stops spending 1.5 seconds on filler, because a purchase justified by a number nobody has decomposed is a guess with an invoice attached.

Abandonment tracks the useful number#

The reason to care which metric is named is that they behave differently against the outcome.

Correlation between abandonment and time to first useful content across sessions was 0.63. Against time to first token it was 0.08, which is what you would expect from a quantity that is nearly constant at 300ms whatever else happens. Against total completion time it was 0.21, and the weakness there is informative: handlers who got a usable first sentence at 1.8 seconds routinely stayed through an eight-second completion, because they were reading. Completion time matters most where the user cannot act on a partial result — a full letter draft they intend to copy, an export, a batch — and for those flows streaming buys nothing at all and the number to optimise is the last token.

There is an arithmetic reason the first useful token matters so much more than the last one. Generation ran at about 45 tokens per second, which is roughly 34 words per second, while people read prose at four to five words per second. Once the headline has started, the text arrives more than seven times faster than it can be consumed, so from the reader's point of view the remaining stream is free — they are never waiting for the next word, they are behind it. That is the whole mechanism by which streaming improves the experience, and it explains why the improvement collapses entirely if the first useful word is late: everything before the headline is dead time that no amount of throughput after it can repay. It also explains why streaming stops helping on surfaces where the user does not read the output as it arrives, such as a generated letter destined for a composer, since a person watching text they intend to copy is waiting on the last token at the same rate as anyone else.

Instrumenting time to first useful content is a few lines once the stream is structured, and the discipline is to emit the mark from the render path rather than the network path, since a token that has arrived but not painted has not helped anyone.

let requestStart = performance.now();
let ttfuc: number | null = null;

for await (const partial of partialObjectStream) {
  if (ttfuc === null && partial.headline && partial.headline.length > 0) {
    ttfuc = performance.now() - requestStart;
    telemetry.mark("ttfuc_ms", ttfuc, { answer_id, surface });
  }
  ui.update(partial);
}

telemetry.mark("ttlt_ms", performance.now() - requestStart, { answer_id });

Abandonment needs a definition that matches what the user did rather than what the socket did. Atlas counted a request abandoned if the pane was closed, the query was re-issued, or focus left the console for more than 15 seconds before the done event, and deliberately did not count a network drop, because those are an availability problem with a different owner.

What to put in the wait#

The 4-to-8-second bucket is where design changes the outcome most, and the treatments are not equivalent. Atlas ran four of them.

An indeterminate spinner left abandonment in that bucket at 26%. A determinate progress bar driven by an estimate — the animation that fills at a rate chosen to look plausible — took it to 22% in the first fortnight and back to 27% by week six, while the survey item the system tells me what it is doing fell and the system feels unreliable rose. The bar was lying in a way users could detect: it reached 90% and sat there, or it completed while the answer was still streaming, and once a progress indicator is caught being wrong it stops being read and starts being evidence about the product.

Streaming the model's reasoning into the main column, which chapter 2 rejected on composition grounds, took abandonment to 19% and was rejected anyway because it did so by burying the answer, and the metric that improved was not the one anyone cared about.

Real stage labels — Searching the claims manual, then Reading 3 passages, then Drafting the answer, each appearing when the corresponding pipeline stage actually started — took the bucket to 17% and held there. They also changed the shape of the complaints: handlers stopped reporting that Atlas was slow and started reporting that retrieval was slow, which was true and actionable. Stage labels are honest in the specific sense that they are emitted by the code that does the work, so they cannot drift from reality the way an estimated bar does.

emit({ type: "stage", id: "retrieval", label: "Searching the claims manual" });
const chunks = await retrieve(query, { acl: user.scope });
emit({ type: "stage", id: "reading", label: `Reading ${chunks.length} passages` });

The published research on this is consistent with those results and worth reading directly; The UX of AI: Research-Backed Best Practices for Designing AI Interfaces collects the evidence on how waiting treatments and trust signals interact, including the finding that a wait users can attribute to a named activity is tolerated far better than an equally long unexplained one.

Movement that costs trust#

A fast answer can still feel unreliable, and the mechanism is usually the page moving. When the answer block is not given reserved height, the arrival of the headline pushes the source list down, the marks settling from chapter 4 shift the paragraph again, and a handler who has started reading loses their place twice. Atlas measured cumulative layout shift on the answer pane at 0.31 in the first implementation, against a 0.1 threshold generally treated as acceptable. Cumulative layout shift, usually written CLS, is a browser-reported score that accumulates every time visible content moves position without the user having asked it to, weighted by how much of the viewport moved and how far. The qualitative feedback matched the number: interviewees used the word jumpy seven times in a set of twelve interviews.

Fixing it was a matter of committing to the final layout before the content exists. The answer block reserves three lines. The source list renders as a fixed-height skeleton, meaning a grey placeholder shaped like the content that is coming, and it carries the count from retrieval as soon as retrieval completes, which is before the model has written anything about them. Marks apply as underline and background only, never as inserted elements. Post-fix CLS was 0.02, no timings changed, and the survey item about reliability moved by four points.

The related error is the skeleton that does not match its content. A three-line skeleton followed by an eleven-line answer is a shift by another route, so the skeleton should be sized to the p50 answer for that surface and the block should grow downward only.

Where perceived latency stops helping#

None of this makes the system faster, and there is a point past which perception work stops paying. Above roughly ten seconds, handlers stopped waiting regardless of how well the wait was explained, and the productive design changes at that point are different in kind: run the query in the background and notify, break the task into steps the user can start acting on, or decline to answer synchronously at all. Perception buys tolerance in the two-to-eight-second band, not indefinitely.

It also does nothing for the tail. A p50 of 4.1 seconds with a p99 of 31 seconds produces a product that most people experience as fine and a small number of people experience as broken, and those people are disproportionately the ones with the largest document sets, which is to say the largest customers. Stage labels make a 31-second wait legible; they do not make it acceptable.

The last limit is that speed and honesty are in tension at exactly one point, the gating described in chapter 4, where a copy control stays disabled until attribution resolves. That gate costs 300ms of perceived speed on every answer to prevent an unchecked sentence leaving the system. Chapter 9 returns to that class of trade with the rest of them.

Abandonment tells you that a handler left. It says nothing about whether the handlers who stayed got an answer worth having, and the instinct at this point is to ask them, on every answer, with a widget under the text.