Ask an LLM serving framework what its batch size is and the honest answer is: it depends on the millisecond. Requests join and leave the batch between individual forward passes, not between requests. That one scheduling decision — made at token granularity instead of request granularity — is where most of the throughput gains in modern inference servers come from. It is also the source of a latency pathology that every team running vLLM or a similar engine eventually hits: one user pastes a huge document, and every other user's stream visibly freezes.
This post walks through how continuous batching actually works, why it multiplied throughput by an order of magnitude, and why the prefill/decode split means the naive version trades one latency metric for another.
Static batching wastes the GPU on whoever finishes early
The classic way to serve a neural network is request-level batching: collect N requests, run the model once, return N results. That works fine for models that do one forward pass per request — a classifier, an embedding model. LLM generation is different: it's autoregressive, so a single request means running the model once per output token, and different requests want wildly different numbers of tokens.
Under static batching, the batch is formed once and held until every sequence in it finishes. A request that completes after 10 tokens sits as dead weight in the batch — its slot doing nothing — while a neighbor generates 800 more. The GPU is "busy," but a growing fraction of each forward pass is wasted on finished sequences, and new requests queue outside until the entire batch drains.
Orca's move: schedule iterations, not requests
The Orca paper (Yu et al., OSDI '22, Seoul National University and FriendliAI) was the first system to break this. Its core idea, iteration-level scheduling, is simple to state: the scheduler runs one iteration (one forward pass, one token per active sequence) at a time, and re-decides the batch composition before the next one. A sequence that just emitted its end-of-sequence token leaves the batch immediately; a queued request takes its slot on the very next iteration. No slot ever idles waiting for the slowest sequence.
The complication is that batching arbitrary requests mid-flight isn't trivial: sequences in different phases have different shapes (one is processing a 900-token prompt, another is generating its 51st token). Orca's second technique, selective batching, handles this by applying batching only to the operations where it's valid, letting the rest run per-sequence. Together, the two techniques delivered a reported 36.9x throughput improvement at the same level of latency versus NVIDIA's FasterTransformer on a GPT-3 175B model.
Every serious serving engine since — vLLM, TGI, DeepSpeed-FastGen, TensorRT-LLM — is built on this idea, usually under the name continuous batching or in-flight batching.
The numbers: 23x, measured
Anyscale benchmarked this with OPT-13B on a single 40GB A100, sending 1,000 requests with 512 input tokens and output lengths drawn from an exponential distribution (mean 128). At the most generation-heavy configuration (max sequence length 2048), naive static batching managed 81 tokens/s. Continuous-batching systems (Hugging Face's text-generation-inference, Ray Serve) reached roughly 8x that. vLLM — continuous batching plus PagedAttention's memory efficiency, which lets far more sequences fit in the batch — hit about 1,900 tokens/s, a 23x improvement over the static baseline.
Two things are worth reading out of that result. First, the gain grows with output-length variance: the more unevenly sequences finish, the more static batching wastes. Second, continuous batching and memory management compound — freeing a finished sequence's slot only helps if you can also free and reuse its KV-cache memory fast enough to admit someone new.
The catch: a batch is now two different workloads
Continuous batching means newly admitted requests and mid-generation requests share iterations. But those two kinds of work are not symmetric:
- Prefill processes the entire prompt in one pass — thousands of tokens of parallel work that saturates the GPU's compute.
- Decode advances each sequence by exactly one token — very little compute per sequence, with low compute utilization (the SARATHI paper measured decode-only batches costing up to an order of magnitude more per token than the same tokens piggybacked on prefill compute).
So what happens when a request with a very long prompt arrives while 40 other users are mid-stream? The scheduler has to run that prompt's prefill, and the iteration it runs in takes as long as the prompt is large. Every decoding sequence sharing that iteration waits. From the outside: 40 chat streams stall simultaneously, then resume. Time-to-first-token (TTFT) for the new request looks great; inter-token latency (ITL) for everyone else spikes.
vLLM's earlier (V0) scheduler made exactly this trade, and its docs said so: the default policy "prioritizes prefills and doesn't batch prefill and decode to the same batch," which "optimizes the TTFT (time to the first token), but incurs slower ITL (inter token latency)."
Chunked prefill: slice the elephant
SARATHI's fix is to stop treating a prefill as an indivisible unit. Two pieces:
- Chunked prefills — split a long prompt's prefill into equal-sized chunks processed across multiple iterations, instead of one monolithic pass.
- Decode-maximal batching — build each batch as one prefill chunk plus as many decode requests as fit. The decodes ride along on iterations that were already compute-saturated by the chunk, at marginal cost far below running them alone.
The reported results: up to 10x decode throughput on LLaMA-13B (A6000) with a 1.33x end-to-end gain, 4.25x decode throughput on LLaMA-33B (A100) with 1.25x end-to-end, and for pipeline-parallel GPT-3, a 6.29x reduction in pipeline bubbles for a 1.91x end-to-end improvement — because uniform, chunk-sized iterations also stop the micro-batch imbalance that stalls pipeline stages.
The deeper point: chunking doesn't make prefill cheaper. It makes iteration time bounded and predictable, which is what a latency guarantee actually requires.
What your engine does today, and the knob that matters
In current vLLM (V1), chunked prefill is enabled by default whenever possible, and the scheduling policy inverts the old one: it prioritizes decode, batching all pending decode requests before scheduling any prefill. In-flight streams advance every iteration; prompt processing soaks up whatever token budget remains.
That budget is max_num_batched_tokens, and it is the TTFT/ITL dial:
| Setting | Effect |
|---|---|
| Smaller (~2,048) | Better ITL — less prefill work rides in each iteration, so decodes are delayed less |
| Larger | Better TTFT and throughput — more prefill tokens per batch; vLLM's docs suggest going above 8,192 for smaller models on large GPUs |
Two closing cautions from the same docs. First, this dial only redistributes latency between the two metrics — if the GPU can't keep up with aggregate load, no schedule saves you. Second, watch for preemption: when KV-cache space runs out, vLLM evicts running requests and recomputes them later, which quietly wrecks end-to-end latency; if you see preemption warnings, that's a memory-sizing problem (raise gpu_memory_utilization, lower concurrency, or add parallelism), not a scheduling one.
Continuous batching turned the batch from a data structure into a per-token scheduling decision. Chunked prefill finished the job by making the iterations themselves uniform. If your p99 inter-token latency has mysterious multi-second spikes, don't start with the model or the GPU — start with what your scheduler does when a long prompt shows up.
References
- Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI '22)
- Anyscale: Achieve 23x LLM Inference Throughput & Reduce p50 Latency
- SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
- vLLM Documentation: Optimization and Tuning
- vLLM v0.8.2 Documentation: Optimization and Tuning (V0 scheduler behavior)