Prefill and decode

The prompt is known in full before generation starts, so all of its positions are computed in a single largely parallel pass under a causal mask. The per-position key and value state from that pass is kept, so each generated token costs one incremental step rather than a fresh run over the whole prefix. Prefill is compute-bound and amortises one weight read across every prompt token; decode is bandwidth-bound and can amortise it across nothing, which is why an output token costs roughly a hundred prompt tokens on the same hardware.

Autoregressive generation — read the current version at https://daviddominguez.dev/learn/en/autoregressive-generation/token-costs

Back to top