Every training step under PyTorch's automatic mixed precision does something that sounds like a bug: it multiplies your loss by a factor that starts at 65,536, backpropagates the wrong (scaled) gradients, then quietly divides them back down before the optimizer sees them. If an inf shows up, it throws the whole step away — gradients computed, optimizer untouched — and tries again with a smaller factor.
None of this is accidental. It's the machinery that keeps fp16 training from silently zeroing out most of the gradient signal. And the reason bf16 training needs none of it comes down to where each format spends its 16 bits.
fp16's window is 40 powers of 2 wide — and gradients live outside it
IEEE half precision splits its 16 bits into 1 sign bit, 5 exponent bits, and 10 mantissa bits. Per NVIDIA's mixed precision training guide, that buys you normalized values from 2⁻¹⁴ (≈6.10e-5) up to a maximum of 65,504. Subnormals stretch the floor down to 2⁻²⁴ (≈5.96e-8), for a total dynamic range — denormals included — of 40 powers of 2. Single precision spans 264.
Anything smaller than 2⁻²⁴ becomes exactly zero. The PyTorch AMP docs state the consequence plainly:
Gradient values with small magnitudes may not be representable in float16. These values will flush to zero ("underflow"), so the update for the corresponding parameters will be lost.How much signal actually sits down there? The original mixed precision paper (Micikevicius et al., 2017) histogrammed activation gradients while training a Multibox SSD detector: 67% of values were zero outright, and 2% of all values fell in the [2⁻²⁴, 2⁻²³) range alone — the very bottom subnormal bin, where fp16 has almost no precision left and one halving away from flushing to zero. Trained naively in fp16, SSD diverged. The same paper found that a loss-scaling factor of just 8 "recovers the relevant gradient values and mixed-precision training matches FP32 mAP."
Notice what that implies: the gradients weren't unrepresentably small in some fundamental sense. They were three binary orders of magnitude too small for the window. fp16's range above 1.0 — all the way to 65,504 — was sitting there unused, because gradients in a converging network are almost all fractions. The distribution is lopsided; the format isn't.
Shift the distribution, don't widen the window
Loss scaling exploits that lopsidedness with one multiply. Scale the loss by S before calling backward(), and by linearity of the chain rule every gradient in the network comes out multiplied by S — shifted up into fp16's representable range for the entire backward pass. Before the optimizer applies the update, divide by S to get the true gradients back.
The catch is choosing S. Too small and gradients still underflow to zero — which is silent: training just converges worse, and nothing crashes. Too large and gradients overflow to inf — which is loud and immediately detectable. That asymmetry is the whole design insight behind dynamic loss scaling: since only one failure mode announces itself, probe upward until you hit it, then back off.
GradScaler: a sawtooth wave with a skip button
PyTorch implements this in torch.amp.GradScaler. The defaults, from the source:
| Parameter | Default | Meaning |
|---|---|---|
init_scale | 2¹⁶ = 65,536 | starting scale factor |
growth_factor | 2.0 | multiplier after a clean streak |
growth_interval | 2000 | iterations without inf/NaN required to grow |
backoff_factor | 0.5 | multiplier when inf/NaN appears |
The mechanism, per the docstring: scaler.step(optimizer) first checks the unscaled gradients. If infs or NaNs are found, it skips the underlying optimizer.step() — the parameters remain uncorrupted — and update() multiplies the scale by backoff_factor. If 2000 consecutive iterations pass without an overflow, the scale doubles. Plotted over a training run, the scale factor is a sawtooth: slow doubling, instant halving.
Two consequences of those defaults are worth internalizing:
- Your first few steps probably didn't happen. Starting at 65,536 deliberately overshoots; the docstring notes that infs "often" appear in the first few iterations while the scale calibrates, and each one skips the optimizer step. That's expected, not a bug — but it means step counts and LR schedules see a few phantom iterations.
- Skipped steps are the health metric. After calibration, skipping "should occur rarely (once every few hundred or thousand iterations)." A scale that keeps collapsing means overflows aren't coming from the scale being too high — they're coming from your model, and
GradScaleris dutifully halving its way toward zero while masking a divergence.
Loss scaling is only one of three techniques the 2017 paper needed to match fp32 accuracy. The other two: keep an fp32 master copy of the weights that accumulates updates (an fp16 weight can't absorb a tiny update — the addition rounds to no-op), and do fp16 arithmetic that accumulates into fp32 outputs. autocast bakes the spirit of this in at the op level: matmuls and convolutions (matmul, mm, bmm, conv1d/2d/3d, linear) run in fp16 where tensor cores pay off, while reductions and anything exponent-sensitive — softmax, log_softmax, layer_norm, sum, and the loss functions themselves — are autocast to fp32.
bf16: same 16 bits, different bet
bfloat16 splits its 16 bits as 1 sign, 8 exponent, 7 mantissa — the same exponent width as fp32. That's not a coincidence; per Google's Cloud TPU article, the format was designed from the observation that "neural networks are far more sensitive to the size of the exponent than that of the mantissa," and the exponent was matched to fp32 specifically "to ensure identical behavior for underflows, overflows, and NaNs."
With fp32's dynamic range, gradients that underflow in fp16 are directly representable, so the entire apparatus above disappears. Google's framing: "Unlike FP16, which typically requires special handling via techniques such as loss scaling, BF16 comes close to being a drop-in replacement for FP32." The PyTorch AMP docs reflect the same split — CPU autocast with bf16 is documented as autocast-only, no GradScaler in the loop.
The bet, of course, has another side: those exponent bits came out of the mantissa. bf16 keeps 7 mantissa bits to fp16's 10 — an 8× coarser grid at any given magnitude. bf16 wagers that neural nets tolerate noise in gradient values far better than they tolerate gradients vanishing entirely. A decade of large-model training on TPUs and post-Ampere GPUs has mostly settled that wager in bf16's favor.
Why fp16 knowledge still pays rent
If bf16 wins, why understand the fp16 machinery at all? Because the constraint didn't vanish — it moved. Plenty of deployed hardware still lacks fast bf16. Mixed precision's economics remain compelling wherever you land: NVIDIA's guide cites tensor cores at 8× the throughput of the fp32 pipelines and up to 3× end-to-end training speedups, plus the halved memory traffic that comes free with 16-bit storage. And the fp8 formats now arriving have even narrower windows than fp16, resurrecting the same underflow-vs-overflow tension with fewer bits to spend. The min/max/skip sawtooth of GradScaler is the cleanest case study we have of a pattern that keeps coming back: when your values don't fit the format, don't clip them — shift them, and let the loud failure mode steer.