A 400-billion-parameter model in fp16 needs roughly 800 GB just to hold its weights — 400B parameters times two bytes each. The largest single GPU you can buy has 80 GB. The math simply does not close: no one GPU can run a frontier model. So how do these models run at all? The answer is that the model gets sliced across many GPUs, and the most fundamental way to slice it is pipeline parallelism.
🎬 The 60-second version is on Instagram — watch the reel. This post goes deeper on the pipeline bubble, micro-batching, and when to reach for tensor parallelism instead.
TL;DR
- A ~400B model needs ~800 GB in fp16 — far beyond a single 80 GB GPU.
- Pipeline parallelism splits the model by layers across GPUs, like an assembly line; activations hop from one GPU to the next.
- Naive scheduling leaves GPUs idle waiting their turn — the pipeline bubble, wasted time on your most expensive hardware.
- Micro-batching feeds many small batches through the pipeline so every stage stays busy and the bubble shrinks.
- Tensor parallelism splits inside each layer and needs very fast links (NVLink/InfiniBand); large clusters combine both.
Slicing a model by layers
A neural network is a stack of layers. Pipeline parallelism assigns different contiguous groups of layers to different GPUs. GPU 0 holds the first chunk of layers, GPU 1 the next, and so on — each GPU is one stage of an assembly line. A batch enters stage 0, its output (the activations) is sent over the interconnect to stage 1, then to stage 2, until the final stage produces the result. During the backward pass, gradients flow back down the line in the opposite direction.
The appealing property is that pipeline parallelism is relatively frugal with bandwidth: the only thing crossing between GPUs is the activation tensor at each stage boundary, not the weights and not every intermediate computation. That makes it workable even across GPUs connected by ordinary datacenter networking rather than exotic high-speed links.
The pipeline bubble
There is a catch, and it is the whole reason pipeline parallelism is subtle. Run it naively — push one batch through the whole line before starting the next — and most of your GPUs sit idle most of the time. While stage 0 works on the batch, stages 1, 2, and 3 have nothing to do yet. When the batch reaches stage 3, stages 0, 1, and 2 are now idle. This idle time is called the pipeline bubble, and on a four-stage pipeline it can waste a large fraction of your compute. You bought four GPUs and are effectively using one at a time — a terrible return on the most expensive hardware in the building.
Micro-batching shrinks the bubble
The fix is to keep every stage fed. Instead of sending one big batch down the line, split it into many micro-batches and stagger them onto the pipeline like items on a conveyor. As soon as stage 0 finishes the first micro-batch and passes it to stage 1, it starts the second micro-batch — so stage 0 and stage 1 are now working simultaneously on different micro-batches. Once the pipeline is full, all stages are busy at once. The bubble is confined to the fill-up at the start and the drain at the end, and it shrinks relative to the useful work as you add more micro-batches. Modern schedules (such as one-forward-one-backward, or 1F1B) go further by interleaving forward and backward passes to keep utilization high while bounding memory.
Pipeline vs tensor parallelism
Pipeline parallelism is one axis of splitting; tensor parallelism is the other. Rather than giving whole layers to different GPUs, tensor parallelism splits the math within a single layer — a large matrix multiply is partitioned so each GPU computes part of it, and the partial results are combined. This lets a single layer that is too big for one GPU span several, but it comes at a price: the GPUs must exchange data in the middle of every layer, which demands very high-bandwidth, low-latency links like NVLink or InfiniBand.
So the two are complementary. Pipeline parallelism sips bandwidth and splits across stages; tensor parallelism is communication-heavy and splits within layers. Real large-scale training rarely picks one — it combines pipeline, tensor, and data parallelism together (often called 3D parallelism) to fit the model and keep thousands of GPUs busy at once. If you want the memory-side companion to this — cutting the cost of serving these models once trained — see Serving LLMs for Less.
Key Takeaways
- Frontier models exceed single-GPU memory by more than 10× — a 400B model is ~800 GB in fp16, so it must be split across GPUs.
- Pipeline parallelism splits by layers into an assembly line and is bandwidth-frugal (only activations cross stage boundaries).
- The pipeline bubble — idle GPUs waiting their turn — is the central inefficiency; micro-batching shrinks it by keeping every stage busy.
- Tensor parallelism splits inside layers and needs NVLink/InfiniBand; big clusters combine pipeline, tensor, and data parallelism.
This is the deep-dive companion to a @teqnersinsights reel. Follow @teqnersinsights for the 60-second versions, and browse the full reel library for more LLM infrastructure deep-dives.