Every LLM request your server handles is really two workloads wearing one trench coat. Microsoft's Splitwise team power-capped an H100 from 700 W down to 350 W and found that token-generation latency showed almost no impact — while prompt processing on the same GPU was highly sensitive to the same cap. One phase of the request is starving for FLOPs; the other barely uses them. That asymmetry is why modern serving stacks — DistServe, Splitwise, vLLM's disaggregated prefill, and the schedulers built on them — increasingly refuse to run both phases on the same GPU at the same time.
Two phases, opposite bottlenecks
Autoregressive inference splits cleanly in two:
- Prefill processes the entire prompt in one pass to produce the first token and populate the KV cache. Every prompt token's attention and MLP work is available at once, so the GPU runs big, dense matrix multiplies. This phase is compute-bound, and its latency is what users feel as TTFT (time to first token).
- Decode generates one token per forward pass per request. Each step must re-read the KV cache for the request's entire context to produce a single token's worth of math. This phase is bound by memory bandwidth and capacity, and its pace is TPOT (time per output token), also called time between tokens.
Splitwise characterized this on DGX-A100 and DGX-H100 machines running Llama2-70B and BLOOM-176B under vLLM, and the two phases look like different applications. Prefill throughput saturates early — past roughly 2,048 prompt tokens in a batch, adding more stops helping. Decode is the mirror image: throughput keeps climbing as you batch more requests together, all the way until batch size 64, where the machine runs out of KV-cache memory — and even at 64, time between tokens rises by less than 2x. Power tells the same story: prompt-phase power draw climbs with batch size, while token-phase draw stays flat no matter how many tokens are in flight.
The punchline is how lopsided real traffic makes this. In Splitwise's production traces, a coding workload had a median of about 1,500 prompt tokens but only 13 generated tokens; conversation had 1,020 and 129. Yet because decode steps are so much smaller than a prefill pass — for BLOOM-176B, a 1,500-token prompt takes about as long as six output tokens — most of a request's wall-clock life is spent in the memory-bound decode phase, trickling tokens out of a GPU whose compute units are mostly idle.
The interference problem
Continuous batching, the standard scheduling technique, mixes both phases in one batch: each iteration, the scheduler admits new prompts alongside in-flight decodes. That maximizes utilization on paper, but the phases damage each other. When a 2,000-token prefill lands in the batch, every co-scheduled decode request's next token waits behind it — a latency spike that shows up directly in tail inter-token latency. The DistServe authors (OSDI '24) frame the deeper cost: colocating the phases doesn't just cause interference, it couples their resource allocation and parallelism. You are forced to pick one batch size, one tensor-parallel degree, one GPU type for two workloads that want different values of all three.
Two escape routes have emerged, and they make opposite trades.
Escape route 1: chunk the prefill
Sarathi-Serve's answer is to keep the phases on the same GPUs but break prefill into near-equal-sized chunks, then build every iteration's batch from ongoing decodes plus at most a chunk of pending prefill. No decode ever stalls behind a full prompt pass — the authors call it stall-free scheduling — and the uniform batch sizes also reduce pipeline bubbles for pipeline-parallel deployments. Measured against vLLM under tail-latency constraints, they report 2.6x higher serving capacity for Mistral-7B on one A100, up to 3.7x for Yi-34B on two, and up to 5.6x end to end for Falcon-180B with pipeline parallelism.
The catch is the knob. vLLM's own documentation, which ships chunked prefill, notes that it can control tail latency but that picking a good chunk size is hard — too large and decodes still stall behind chunks, too small and you sacrifice the dense-matmul efficiency that made prefill fast.
Escape route 2: give each phase its own GPUs
Disaggregation goes further: run prefill on one pool of GPU instances and decode on another, and ship the KV cache between them. Each pool now gets its own batch policy, its own parallelism strategy, and — this is the part that affects your cloud bill — potentially its own hardware generation.
The obvious objection is the transfer. After prefill, the KV cache for a long prompt on a 70B-class model is gigabytes of tensors that must land on the decode machine before the second token can be computed. Splitwise's naive implementation — serialize the whole cache after prefill completes — added up to 3% to end-to-end latency on large prompts, and a brutal 64% to the second token's latency. The fix is to overlap: transfer each layer's KV cache asynchronously while the next layer's prefill is still computing. With layer-wise transfer over InfiniBand (200 Gbps on their A100 machines, 400 Gbps on H100, implemented with MSCCL++), the visible transfer cost dropped below 7% of prompt computation time, 0.8% of end-to-end latency, and 16.5% on the second token. For short prompts there isn't enough compute to hide behind, so Splitwise uses a hybrid: below 512 tokens (on H100) it just serializes.
Once the phases are separated, the hardware heterogeneity follows. In Splitwise's no-batching measurements on Llama2-70B, the H100 — with 3.43x the TFLOPs of the A100 — cut median TTFT roughly in half (185 ms to 95 ms on the coding trace), but cut time between tokens only to 0.70x (52 ms to 31 ms), tracking its 1.64x memory-bandwidth edge rather than its compute edge. Per coding request, the A100 was actually cheaper ($0.42 vs. $0.52) at equal or better energy. Decode simply does not need the latest GPU; prefill does. Splitwise's cluster designs exploit exactly that, reporting 1.4x higher throughput at 20% lower cost than homogeneous designs, or 2.35x more throughput under the same cost and power budget. DistServe, co-optimizing parallelism per phase and placing phases according to cluster bandwidth, reports up to 7.4x more requests served — or 12.6x tighter latency SLOs — than colocated baselines, while keeping over 90% of requests within their latency targets.
So why do vLLM's docs say it "does not improve throughput"?
Read side by side, the claims look contradictory. DistServe: up to 7.4x more requests per GPU. vLLM's disaggregated-prefill documentation, in capital letters: "Disaggregated prefill DOES NOT improve throughput."
Both are correct, because they measure different things. Raw token throughput — tokens per second per GPU with no latency constraint — does not improve when you split phases; you've added a KV-cache transfer and dedicated some GPUs to a phase that finishes in bursts. A well-tuned colocated system with huge batches can match or beat it on that metric. What disaggregation improves is goodput: DistServe defines it as the maximum request rate each GPU can sustain while meeting both the TTFT and TPOT targets. Under SLOs, colocated systems must over-provision to absorb interference spikes; disaggregated ones don't. vLLM's docs motivate the feature in exactly those terms — you can tune time-to-first-token without touching inter-token latency, and you get control over tail ITL that chunked prefill only approximates.
That framing is the practical decision rule:
| Your constraint | Reasonable default |
|---|---|
| Throughput-only (offline batch, evals) | Colocated continuous batching, large batches |
| Latency SLOs, single-node scale | Chunked prefill, tuned chunk size |
| Strict TTFT and TPOT SLOs, cluster scale, fast interconnect | Disaggregated prefill/decode |
One honest caveat from the trenches: vLLM still marks its disaggregated prefill experimental, and delegates production-grade KV transfer to third-party connectors (LMCache, Mooncake, NIXL, and others) precisely because the transfer layer is infrastructure-dependent — the layer-wise overlap trick only hides the copy if there's InfiniBand-class bandwidth to hide it in. Disaggregation isn't a config flag; it's a cluster design. But the underlying fact it exploits isn't going anywhere: half of your serving workload doesn't need the GPU you bought for the other half.
References
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (OSDI '24)
- Splitwise: Efficient Generative LLM Inference Using Phase Splitting (arXiv:2311.18677)
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (arXiv:2403.02310)
- vLLM Documentation: Disaggregated Prefilling