Once you move an LLM off a hosted API and onto your own GPUs, the economics change completely. The cost is no longer a per-token line item that scales with usage — it is a fixed fleet of accelerators that you pay for whether they are busy or idle. And in most self-hosted deployments, they are idle far more than their owners realize. The single biggest source of wasted spend in LLM infrastructure is not the price of the hardware; it is how poorly that hardware is utilized between tokens.

The good news is that closing this gap rarely requires buying anything. The largest wins come from changing how requests are scheduled and how memory is managed — software problems with well-understood solutions. This post walks through the three that matter most.

Why naive serving wastes GPUs

LLM decoding is memory-bandwidth-bound, not compute-bound. Generating each token requires streaming the entire model's weights from GPU memory through the compute units. For a single request, the arithmetic units sit mostly idle while waiting on memory — you are paying for a Ferrari to sit in traffic.

The way to fix this is batching: process many requests together so that one expensive pass over the weights produces tokens for dozens of sequences at once. But traditional static batching, borrowed from classic ML serving, batches requests at admission and holds the batch until every sequence finishes. Because generations have wildly different lengths, the whole batch waits on its slowest member, and finished slots go unused. You get some amortization, but you leave most of it on the table.

Continuous batching: the biggest single win

Continuous batching (also called in-flight batching) changes the unit of scheduling from a request to a single decoding step. After every step, completed sequences are evicted and waiting requests are admitted in their place, so the batch stays full continuously rather than draining and refilling.

The effect is dramatic. In Anyscale's widely cited benchmark, continuous batching with memory optimization via vLLM delivered up to 23x the throughput of static batching. Even without vLLM's memory tricks, continuous batching on systems like Hugging Face's text-generation-inference reached 8x, compared with roughly 4x for a highly optimized static-batching implementation. The same GPUs, the same model — an order of magnitude more work per dollar, purely from smarter scheduling.

PagedAttention: stop fragmenting the KV cache

Continuous batching exposes a second bottleneck: the KV cache. Every active sequence stores the attention keys and values for its context, and that memory grows with both batch size and sequence length. At scale the KV cache, not the model weights, is what caps how many requests you can hold concurrently — and concurrency is exactly what batching depends on.

Naive serving reserves a contiguous block of memory for each sequence's maximum possible length, which strands enormous amounts of memory in fragmentation and over-reservation. PagedAttention, introduced by vLLM, borrows the idea of virtual memory paging: it splits the KV cache into fixed-size blocks allocated on demand and referenced through a lookup table. Memory is handed out just-in-time as sequences grow, near-eliminating the waste. Because more sequences now fit in the same VRAM, effective batch sizes climb, and it is this that lets vLLM more than double the throughput of a naive continuous-batching implementation.

Quantization: fit a bigger model on a smaller GPU

The third lever attacks the model itself. Weights are typically served in 16-bit floating point, but they do not have to be. Quantization stores them in lower precision — INT8 or INT4 — shrinking both the memory footprint and the bandwidth cost of every forward pass.

The arithmetic is straightforward and worth internalizing. A 70-billion-parameter model at FP16 needs about 140 GB just for weights (70B × 2 bytes), forcing a multi-GPU deployment. At INT4 (0.5 bytes per parameter) those same weights need roughly 35 GB — small enough to fit on a single 80 GB accelerator, with room to spare for the KV cache. Collapsing a two- or four-GPU setup down to one is often the largest line-item reduction available, and because inference is memory-bound, the smaller weights usually speed things up too.

The honest caveat: aggressive quantization can degrade quality, and how much depends on the model and the task. Modern schemes like GPTQ and AWQ are calibrated to minimize this, but INT4 is not free — you should always evaluate on your own workload before shipping it, not trust a benchmark leaderboard.

Putting it together

These techniques compose. A production-grade open-source stack like vLLM gives you continuous batching and PagedAttention out of the box, and accepts quantized weights directly:

bash
# Serve a quantized 70B model with an OpenAI-compatible API.
# Continuous batching and PagedAttention are on by default.
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --quantization awq \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256

Stacked, the effect is multiplicative rather than additive: quantization roughly halves the hardware needed to hold the model, continuous batching multiplies the requests each GPU serves, and PagedAttention raises the ceiling on how many of those requests fit at once. It is entirely realistic to take a naive FP16, single-request deployment and reduce its cost per token by well over half — without changing the model's user-facing behavior and without provisioning a single new GPU.

Key Takeaways

  • Utilization, not hardware price, is the real cost driver. Naive serving leaves GPUs idle between tokens; the biggest savings come from software.
  • Continuous batching is the highest-leverage change. Benchmarks show up to 23x throughput over static batching on the same hardware.
  • PagedAttention removes the KV-cache ceiling that otherwise caps how much you can batch, more than doubling naive continuous-batching throughput.
  • Quantization changes your hardware bill of materials — an INT4 70B model can drop from ~140 GB to ~35 GB of weights — but must be quality-tested on your own workload.
  • Adopt a serving stack that gives you all three by default, then measure cost per token as your north-star metric rather than raw latency alone.

References

Running your own models and not sure how much you're overpaying? Optimizing LLM inference — GPU deployment, throughput tuning, quantization, and cost reduction — is one of the things we do at Teqners. If your inference bill has outgrown your intuition for it, we can help.

LLMInferenceGPUvLLMCost Optimization