CORE GUIDE
Latency and throughput basics
AI serving performance is a queueing and workload problem: response time, concurrency and token generation compete for finite compute capacity.
Mental model
Latency describes how long one request waits and runs; throughput describes how much work the system completes over time. Optimizing one can hurt the other when batching, concurrency and model size consume shared capacity.
Why it matters
A model that is acceptable in an isolated demo can become unusable under concurrent traffic. Long prompts increase prefill work, long generations occupy serving capacity, and aggressive batching can improve throughput while delaying individual requests. Product architecture needs explicit targets for time-to-first-token, generation speed, tail latency and concurrency rather than one vague 'fast enough' requirement.
01
Break response time into workload components
Separate queue wait, input processing or prefill, first-token delay and subsequent token generation. Measure these under representative prompt lengths and concurrent load, not only one request at a time. Capacity planning then chooses trade-offs such as smaller models, shorter contexts, caching, batching, routing or admission control according to the product's latency and throughput objectives.
02
Example: support chat under a traffic spike
At low load, a large model starts responding in under a second. During a spike, requests queue and tail latency rises above ten seconds even though per-token generation speed is unchanged. The fix may be concurrency limits, routing routine cases to a smaller model or reducing context cost—not prompt wording.
Common failure modes
- Benchmarking latency with one request and extrapolating directly to production concurrency.
- Optimizing average latency while ignoring tail behavior users actually experience.
- Increasing context or output length without accounting for shared serving capacity.
Engineering heuristics
- Measure time-to-first-token and end-to-end latency separately for streaming products.
- Test representative prompt lengths and concurrency levels.
- Track cost and successful-task throughput alongside raw tokens per second.
Takeaways
- 01Serving speed depends on workload and queueing, not only model architecture.
- 02Throughput improvements can increase individual latency.
- 03Capacity targets should be expressed in product-level service objectives.
Used in
This Concept is reused across these canonical learning paths.