Streaming and time to first token

A request has a startup clock and a per-token clock. Streaming collapses the first and leaves the second exactly where it was, because the same prefill and the same number of decode steps still have to run. Measure them separately with a two-armed sweep: hold the output fixed and push the prompt to watch startup move, hold the prompt fixed and push the output to watch the tail move. Batching, cache hits, chunked prefill and buffering proxies all blur the split, which is why one aggregate latency number hides what users feel.

Autoregressive generation — read the current version at https://daviddominguez.dev/learn/en/autoregressive-generation/tail-latency

Back to top