Inference Latency, Batching, and KV Cache
Serving tradeoffs for throughput, tail latency, autoregressive decoding, batching, and key-value cache memory.
Inference is a serving system wrapped around linear algebra. The model matters, but so do queueing, batching, cache layout, memory bandwidth, and tail latency. Optimize the path users wait on, not only peak throughput.
Latency components
A request usually spends time in several places:
\[T_{request} = T_{queue} + T_{preprocess} + T_{model} + T_{postprocess} + T_{network}\]Batching improves device utilization by sharing kernel launches and increasing arithmetic intensity. It also adds queueing delay. Dynamic batching is a latency-throughput tradeoff, not a free speedup.
For an interactive service, track p50, p95, p99, and timeout rate. A high average tokens/sec can coexist with unacceptable tail latency.
Autoregressive decoding
Decoder-only language models generate one token at a time. Scaled dot-product attention is:
\[\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_{head}}} + M\right)V\]The prompt phase, often called prefill, processes the input sequence in parallel. The decode phase produces new tokens sequentially. Decode is often memory-bandwidth bound because each new token reads model weights and attends over cached context.
KV cache
For causal attention, previous key and value tensors do not need to be recomputed at every decode step. Store them per layer:
\[K_{cache} \leftarrow \text{concat}(K_{past}, k_t), \qquad V_{cache} \leftarrow \text{concat}(V_{past}, v_t)\]A rough full-model KV-cache size is:
\[\text{bytes} \approx 2 \times L \times B \times T \times H_{kv} \times d_{head} \times \text{bytes\_per\_element}\]where $L$ is layer count, $B$ is batch size, $T$ is sequence length, $H_{kv}$ is the number of key-value heads, and the factor 2 accounts for keys and values.
This memory grows linearly with sequence length and batch size. Long-context serving is often limited by KV cache memory before raw compute.
Batching failure modes
Continuous batching can mix requests with different prompt lengths and generation lengths. Without careful scheduling, short requests wait behind long ones. Padding waste can dominate if batch shapes are poorly grouped.
Large batches improve throughput but can increase time-to-first-token. For chat, time-to-first-token and inter-token latency are separate user-visible metrics.
Cache eviction and cancellation matter. If a client disconnects, its cache allocation should be released promptly. Otherwise abandoned requests can consume memory and degrade unrelated traffic.
Practical checks
- Measure time-to-first-token, inter-token latency, total latency, and tokens/sec.
- Track queue depth and batch size distribution.
- Track KV cache bytes, allocation failures, and eviction count.
- Separate prefill and decode metrics.
- Test mixed prompt lengths, early cancellation, and max-context requests.
- Compare p95 and p99 latency under load, not only single-request latency.
Design guidance
Pick a serving objective first: lowest latency, highest throughput, cheapest batch jobs, or interactive tail behavior. The scheduler, batch size, cache policy, and admission control should match that objective.