Spec sheets sell TFLOPS. An A100 advertises 312 TFLOPS of dense FP16 compute; buyers compare that number the way they once compared CPU clock speeds. But when that GPU is generating tokens for a single chat session, almost none of that compute is in use — and buying a card with more of it will barely change how fast the words appear. The number that actually governs decode speed is printed further down the spec sheet: memory bandwidth.
This isn't a quirk of one framework or one model. It falls straight out of arithmetic you can do on a napkin, and once you've done it, a whole cluster of LLM serving techniques — quantization, batching, speculative decoding, grouped-query attention — stop looking like a bag of tricks and start looking like the same trick.
The ops:byte ratio, or what your GPU wants from you
NVIDIA's GPU performance documentation frames every kernel as a race between two pipelines: the math units and the memory system. Each operation has an arithmetic intensity — how many floating-point operations it performs per byte it moves to or from memory. Each processor has an ops:byte ratio — its math bandwidth divided by its memory bandwidth. The rule is simple:
"An algorithm is math limited on a given processor if the algorithm's arithmetic intensity is higher than the processor's ops:byte ratio. Conversely, an algorithm is memory limited if its arithmetic intensity is lower."
For a V100 — 125 FP16 Tensor TFLOPS against roughly 900 GB/s of HBM bandwidth — NVIDIA puts the ops:byte ratio at 40–139 depending on which level of the memory hierarchy the data comes from. Modern inference GPUs land higher. An A10 delivers 125 TFLOPS against 600 GB/s: 208 ops per byte. An A100 delivers 312 TFLOPS against ~1.5 TB/s: also about 208. To keep the math units of these chips busy, a kernel has to perform on the order of two hundred operations for every byte it loads.
Now look at what NVIDIA's own examples say about the shapes that show up in inference. A fully-connected layer with 4096 outputs and 1024 inputs at batch size 1 has an arithmetic intensity of 1 FLOP per byte. The same layer at batch size 512 hits 315 FLOPS/byte and becomes math-limited. ReLU sits at 0.25. Layer normalization is under 10. Batch size 1 is exactly what single-stream LLM decoding is.
Decode: 2 FLOPs per weight, then wait
Autoregressive generation produces one token per forward pass, and each pass multiplies activation vectors against every weight matrix in the model. A matrix-vector product does 2 FLOPs (a multiply and an add) per matrix element — and each FP16 element is 2 bytes that must come in from HBM. That's an intensity of about 1 op/byte, stranded two orders of magnitude below the ~208 the hardware wants. The math units finish their share almost instantly and stall waiting for the next stretch of weights to arrive.
This gives decode latency a formula that contains no FLOPS term at all. As kipply's classic walkthrough derives it: a forward pass costs about 2P FLOPs for a model with P parameters, but in the memory-bound regime the time per token is just
time per token ≈ (bytes of weights) / (memory bandwidth)Baseten's inference guide runs this for Llama 2 7B on an A10: 14 GB of FP16 weights over 600 GB/s is 23 ms per token — about 43 tokens/second — before you've profiled a single kernel. They also compute the model's actual decode arithmetic intensity at roughly 62 ops/byte against the A10's 208.3: memory-bound, exactly as the napkin predicted. Estimates like this are why practitioners can predict a deployment's ballpark tokens/sec from the spec sheet alone.
It also explains the utilization numbers that shock people the first time they see them. If the hardware can do ~208 ops per byte and your workload only offers ~1–2, the math pipeline runs at well under 1% of peak. The GPU isn't misconfigured. The workload just doesn't have enough arithmetic per byte to feed it.
Batching fixes it — until the KV cache un-fixes it
The obvious escape is batching. Serve many sequences at once and each weight byte you load gets reused across every sequence in the batch: intensity scales with batch size. The crossover shows up at strikingly similar points across hardware. From kipply's A100 numbers, the flops-to-bandwidth ratio of 208 means matmuls become compute-bound around batch 208. Google's scaling book derives the same threshold for TPU v5e in bfloat16: matmuls are compute-bound iff the per-replica token batch exceeds 240.
Two problems. First, as the scaling book puts it, "generation happens serially, token-by-token, this requires us to batch multiple requests together, which is hard" — you need hundreds of concurrent live requests, not a big dataset you can chop up like in training or prefill.
Second, attention doesn't play along. Batching amortizes weight loads because every sequence shares the weights — but every sequence has its own KV cache, so those bytes are loaded once per sequence no matter what. During generation, attention reads the entire cache to produce a handful of FLOPs; the scaling book pegs its arithmetic intensity at roughly 1 and is blunt about it: "we cannot do anything to improve the arithmetic intensity of attention during generation." Worse, the cache is enormous: for LLaMA 2-13B at 8192 context, just four sequences' KV caches exceed the memory used by the weights themselves. On the A10 setup above, Baseten's arithmetic leaves room for about 19,230 cached tokens — around 4 concurrent sequences at a 4096 context — far below any compute-bound crossover.
The scaling book compresses the whole tradeoff into one line:
max tokens/s ≈ batch × bandwidth / (batch × kv_cache_bytes + param_bytes)Small batches waste bandwidth on re-reading weights; large batches spend it on KV caches instead. Either way, the numerator is bandwidth.
One lens, many "tricks"
Read back through the serving playbook with bytes moved as the cost model and everything clicks into place:
- Weight quantization speeds up decode even with no faster math. INT8 weights halve the bytes per forward pass, so the memory-bound formula halves the time per token. The speedup comes from the numerator, not the ALUs.
- Speculative decoding works because verification is nearly free. Checking several drafted tokens in one forward pass loads the weights once — roughly the same bytes as generating a single token — while the idle math units absorb the extra FLOPs.
- Prefill and decode are different workloads. Prefill processes the whole prompt at once, so its matmuls run at high intensity and are compute-bound; decode is bandwidth-bound. That's the asymmetry behind chunked prefill and prefill/decode disaggregation.
- Grouped-query and multi-query attention are bandwidth features. Shrinking the KV cache cuts the per-sequence bytes that batching can't amortize, raising the batch size you can afford.
- The spec that predicts your latency is GB/s. An H100's FLOPS advantage over an A100 matters enormously for training and prefill; for single-stream decode, the bandwidth ratio is the better first-order predictor.
None of this means FLOPS never matter — at high batch, in prefill, and in training they're the whole game. But if the question is "why is my chatbot generating 40 tokens a second on a 300-TFLOPS card," the answer isn't in the math units. Divide the model's bytes by your memory bandwidth. That's your speed limit, and everything else is a technique for moving fewer bytes.