Comparing latency across deployments

Output length ranks two requests reliably on one model, one machine, one concurrency level and comparable prompts, and inside that fence it explains almost all the variance in request latency. Across deployments it compares two exchange rates as though they were one: a 70B model on four A100s decodes at 20 tokens a second where an 8B on an H100 manages 90, so 500 tokens can lose to 2,000. Measure decode rate per deployment, compare tokens divided by rate, and carry the conditions with the prediction.

Autoregressive generation — read the current version at https://daviddominguez.dev/learn/en/autoregressive-generation/comparing-latency-across-deployments

Back to top