Output length and latency

A 20,000-token prompt with a 50-token answer finishes about ten times sooner than a 200-token prompt with a 2,000-token answer, because the prompt is one largely parallel pass while every generated token costs its own sequential pass. Ranking two requests by total tokens gets this backwards, and on the same hardware one output token is worth roughly 140 prompt tokens. Compare the two shapes at fixed model, hardware and concurrency, and report the split — how much of the wall clock fell before the first token and how much after it.

Autoregressive generation — read the current version at https://daviddominguez.dev/learn/en/autoregressive-generation/output-length-and-latency

Back to top