A faster first token can still finish three times later

Two candidate deployments went into a bake-off for a support assistant, and the benchmark that came out of it fit on one slide. Model A returned its first token in 200 milliseconds and then generated at 30 tokens per second. Model B took 900 milliseconds to the first token and then generated at 110. The slide ranked them by time to first token, A above B, with a note that A was "4.5x more responsive". The recommendation was to ship A.

The workload on that slide was support answers with a median length of about 600 tokens. Running the arithmetic on the same two rows gives A a completion time of 0.2 + 600/30 = 20.2 seconds and B a completion time of 0.9 + 600/110 = 6.35 seconds. The model the slide called 4.5x more responsive takes more than three times as long to produce a finished answer. Both numbers on the slide were measured correctly. They were measurements of two different milestones, and the slide ranked on one of them without saying which task the ranking was for.

Where the two curves cross#

The interesting quantity is not either completion time on its own but the output length at which the ordering flips. A finishes first while its first-token advantage is larger than its throughput disadvantage, and the crossover is where the two completion times are equal:

ttft_a, rate_a = 0.200, 30    # seconds, tokens/sec
ttft_b, rate_b = 0.900, 110

# 0.2 + n/30 == 0.9 + n/110
crossover = (ttft_b - ttft_a) / (1 / rate_a - 1 / rate_b)
print(f"{crossover:.1f} tokens")     # 28.9 tokens

for n in (20, 30, 100, 600, 2000):
    a = ttft_a + n / rate_a
    b = ttft_b + n / rate_b
    print(f"{n:5d} tokens   A {a:6.2f}s   B {b:6.2f}s   {'A' if a < b else 'B'} first")

   20 tokens   A   0.87s   B   1.08s   A first
   30 tokens   A   1.20s   B   1.17s   B first
  100 tokens   A   3.53s   B   1.81s   B first
  600 tokens   A  20.20s   B   6.35s   B first
 2000 tokens   A  66.87s   B  19.08s   B first

Twenty-nine tokens is about twenty words. Any answer longer than a single short sentence finishes sooner on B, and the margin widens linearly with length. A's advantage is a fixed 700 milliseconds; B's advantage grows at 0.024 seconds per token, which is why it takes so little output to consume the head start. Publishing a first-token ranking for a workload whose median answer is twenty times the crossover length is not a close call misjudged, it is a ranking of the wrong axis.

Where each number comes from#

The two figures are not two views of one quantity, because they are produced by two different phases of inference with different bottlenecks, and knowing which is which tells you what can be changed.

Time to first token is dominated by prefill: the model processes the entire prompt in one pass before it emits anything. Prefill is compute-bound and roughly linear in prompt length, so a deployment that prefills at 12,000 tokens per second spends 0.67 seconds on an 8,000-token prompt before the first output token exists. That is most of Model B's 900 milliseconds. The remaining 230 milliseconds is queueing, network, and the fixed cost of getting scheduled.

Throughput is dominated by decode, which produces one token per forward pass — one complete run of the current input through every layer of the model — for every sequence in the batch and is limited by how fast the weights can be read from memory rather than by how much arithmetic is done. Decode speed barely depends on prompt length, which is why a 30,000-token prompt and a 500-token prompt generate at nearly the same tokens per second on the same deployment while their first-token times differ by seconds.

That split has an immediate consequence for the bake-off. Model B's first-token disadvantage is mostly prompt processing, and the support assistant's prompt carries about 8,000 tokens of retrieved policy text, meaning passages that a search step pulls from the policy corpus and pastes into the prompt so the model can answer from them. Trimming retrieval from twelve passages to four takes the prompt to roughly 2,800 tokens and B's first token to about 0.46 seconds, which halves the only dimension on which A was ahead. No such lever exists on the other side: nothing you do to the prompt makes Model A generate faster than 30 tokens per second, because decode speed is a property of the model and the hardware. When one of the two numbers is a property of your prompt and the other is a property of the deployment, they should not be weighed as though both were fixed.

What the user is doing while the tokens arrive#

The reason both numbers exist is that different tasks stop on different milestones, which the first chapter set out. Applying that to these two models:

The two rows at the bottom are not arguable. When the consumer of the output is a JSON parser, a schema validator, or another model call, nothing exists until the last token lands. First-token time in those cases is a property of the transport that no one observes; the screen is blank at 0.2 seconds and it is still blank at 0.9, and the only number that has ever mattered is 20.2 against 6.35. Shipping A there costs 14 seconds per request in exchange for an event nobody sees.

The chat row is the one worth arguing about, and it is where the shallow version of this analysis lives. The argument for A goes: chat is a read-along interface, time to first token is the metric that governs perceived latency, therefore A. The first two clauses are true and the conclusion still does not follow, because "governs perceived latency" is not the same as "is the only thing perceived". A user in a WhatsApp-style chat perceives two things — how long the bubble sits empty, and how long until they can stop waiting and reply. A buys 0.7 seconds on the first and loses 13.9 seconds on the second.

There is a further constraint that decides it on this workload. Silent reading runs at roughly 240 words per minute for ordinary prose, which is 4 words per second, or about 5.3 tokens per second at the usual three-tokens-to-four-words ratio. Model A generates at 30 tokens per second and Model B at 110. Both outrun the reader by a wide margin, so in both cases the user is reading at their own pace with text queued ahead of them, and the stream stops being the constraint a second or two in. The 700-millisecond first-token gap is the only part of A's advantage the user experiences at all, and it is competing against the moment the answer is complete and they can act.

What would overturn it#

Two things would flip the chat recommendation back to A, and both are properties of the answers rather than of the models.

The first is length. If the median support answer were 25 tokens rather than 600 — a one-line acknowledgement, a yes-or-no, a status lookup — then A is below the crossover and finishes first as well as starting first, and there is nothing left to weigh. This is why the length distribution and not just its median matters: a workload with a median of 40 tokens and a p90 of 900 is two workloads, and routing the short ones to A and the long ones to B is a legitimate design that neither row of the original slide suggests.

Routing on this axis is unusually tractable because the deciding quantity is a single number and a cheap classifier can estimate it. The intent already tells you most of what you need — a balance lookup is short, a policy explanation is long — and getting it wrong costs latency rather than correctness:

CROSSOVER = 29     # tokens, from the calculation above

def choose_model(prompt: str) -> str:
    expected_tokens = predict_answer_length(prompt)   # cheap classifier on intent
    return "A" if expected_tokens < CROSSOVER else "B"

The margin around the crossover is thin enough that a misprediction near it barely matters: at 40 tokens the two models are within 400 milliseconds of each other. Mispredictions far from it are where the cost lives, and they are asymmetric — sending a 900-token answer to A costs 24 seconds, while sending a 15-token answer to B costs 0.6. A router in doubt should pick B, which is the opposite of what the original slide would have led anyone to build.

Abandonment complicates the reading further, because a user who gives up at 10 seconds never experiences a completion time at all. If 8% of chat sessions on Model A are abandoned mid-stream against 1% on B, the completion-time distributions are not comparable — A's is truncated by the users it lost. Any comparison run long enough to matter has to record abandonment beside the timings, or the slower model will look better on the requests that survived.

The second is where the deployment's numbers came from. A 900-millisecond first token measured on an idle endpoint is not a 900-millisecond first token under production concurrency, and throughput in tokens per second is the number most sensitive to load, because it falls as the serving batch fills. If B's 110 tokens per second is a quiet-hours figure that degrades to 45 at peak while A's 30 holds steady because it is a smaller model on dedicated capacity, then the 600-token completion times are 20.2 and 14.2 rather than 20.2 and 6.35, and the decision is closer than the slide in either direction suggests. Benchmarks that report a single number per model, with no percentile and no concurrency level, are reporting a best case that the production path will not see.

The thing this comparison is not#

Neither of these numbers is a measure of how fast the model is at the task, only of how fast it emits tokens. A model that answers the question in 180 tokens at 30 tokens per second finishes at 6.2 seconds and beats a model that takes 600 tokens at 110, and output length is a property of the model and the prompt together, not a constant of the workload. Comparing two deployments on a fixed 600-token assumption is comparing them on an assumption; measuring each one's own median output length on the same 200 real prompts, and computing completion time from that, is comparing them on the workload.

The comparison also says nothing about whether either answer is correct, which matters because throughput is the dimension along which quality is most reliably traded. A deployment reaching 110 tokens per second at a quarter of the price is frequently a smaller model, and the right way to read that row is as a candidate for the same accuracy evaluation the incumbent went through rather than as a free win. Latency numbers with no accuracy numbers beside them are the same category error as first-token numbers with no completion numbers beside them, one level up.

Everything in this chapter assumed the numbers on the slide were the numbers the user experiences. They were measured against a benchmark client sitting next to the endpoint, and the first chapter's trace was measured in a browser. Those two clocks do not agree, and the gap between them is where a month of flat dashboards can coexist with a steady stream of complaints.