Every training framework tells you the same thing: if your GPU can't fit a batch of 32, run 8 micro-batches of 4 and accumulate gradients before stepping. Mathematically equivalent, just slower. In October 2024, researcher Benjamin Marie toggled that setting on a Hugging Face Trainer run and watched the loss curves diverge. Unsloth's team confirmed and publicized the problem, and Hugging Face shipped a fix within a day — because for token-level tasks like causal language modeling, the equivalence had quietly been false the whole time.
The bug is worth understanding even if you never touch the affected versions, because it isn't a typo someone made once. It's a seam between two reasonable defaults — per-batch mean loss and per-step gradient averaging — that produces wrong gradients only when your sequences have different lengths. Anyone writing a custom training loop can reproduce it today in ten lines.
Where the equivalence actually comes from
Gradient accumulation rests on the linearity of differentiation. If the full-batch loss is a sum over examples, the gradient of the sum is the sum of the per-micro-batch gradients, so you can compute them one micro-batch at a time, add them into .grad, and step once at the end. Nothing about that argument is wrong.
But nobody trains on a summed loss; everyone normalizes. For image classification with batch size B split into N micro-batches of equal size B/N, each micro-batch computes a mean over B/N examples, and the trainer divides each micro-loss by N before backward(). Multiply it out: every example ends up weighted 1/B, exactly matching the full batch. The equivalence holds because every micro-batch contains the same number of things being averaged.
Causal LM training breaks that premise. The loss is a mean over tokens, not sequences, and the number of real (non-padding) tokens per micro-batch depends on which sequences the sampler happened to draw. The moment micro-batches carry different token counts, "average of the means, divided by N" and "sum of everything, divided by the total count" are different numbers — and, more importantly, different gradients.
A worked example
Say one accumulation window contains two micro-batches:
| Micro-batch | Real tokens | Summed token loss | Mean loss |
|---|---|---|---|
| A (short sequences) | 10 | 20.0 | 2.0 |
| B (long sequences) | 90 | 90.0 | 1.0 |
The full-batch loss over all 100 tokens is (20 + 90) / 100 = 1.10. The accumulated version computes (2.0 + 1.0) / 2 = 1.50. The reported loss is inflated, which is cosmetic — but the gradient weighting is not. Under the correct loss, every token contributes with weight 1/100. Under the accumulated mean-of-means, each token in micro-batch A is weighted 1/10 × 1/2 = 1/20, while each token in B gets 1/90 × 1/2 = 1/180. Tokens that happen to land in padding-heavy micro-batches are trained on nine times harder than tokens in dense ones.
That's not noise that averages out, either. It systematically upweights short sequences, and the distortion changes with gradient_accumulation_steps: the same dataset trained at bs=32 and at bs=4, accum=8 optimizes two different objectives. Note what the example implies in reverse: if every micro-batch has the same real-token count — as in pretraining with packed, fixed-length blocks — the denominators are all equal and the bug vanishes. That's part of why it survived so long. It bites hardest in supervised fine-tuning, where padded, wildly variable-length conversations are the norm.
Where it lived in the code
Hugging Face's post-mortem is candid about the mechanism. Every transformers model with a ForCausalLM head ships a default loss: pass labels in, and the forward computes cross_entropy(logits, labels, ignore_index=-100) — with PyTorch's default reduction="mean", normalized over that micro-batch's own non-ignored tokens. The Trainer then treated the returned scalar as one vote among N, scaling by the accumulation step count. Each component is sensible in isolation. The per-model default loss was, in their words, limited by design — power users were expected to take the logits and compute loss themselves. But the Trainer leaned on the defaults for simplicity, and the composition silently mis-normalized.
The fix, merged in PR #34191 on October 17, 2024 and released in v4.46.0 a week later, moves the denominator outside the micro-batch:
loss = nn.functional.cross_entropy(
shift_logits, shift_labels, ignore_index=-100, reduction="sum"
)
loss = loss / num_items # total real tokens in the WHOLE accumulation windowreduction="sum" makes the per-micro-batch losses genuinely additive again, restoring the linearity that made accumulation valid in the first place. The subtle part is num_items: you can't know the window's total token count until you've seen all its micro-batches. The Trainer now prefetches gradient_accumulation_steps batches up front and counts label positions that aren't -100:
num_items_in_batch = sum(
(batch["labels"].ne(-100)).sum() for batch in batch_samples
)That count is threaded into the model forward as num_items_in_batch, which is why model signatures across the library grew **loss_kwargs in follow-up PRs (#34198 added user-supplied loss functions; #34283 broadcast the fix to the remaining architectures).
The same bug, one level up: DDP
Zach Mueller, who maintains Hugging Face Accelerate and implemented much of the trainer-side fix, points out that data parallelism reintroduces the identical mismatch across a different axis. Under DDP, each rank computes a loss normalized by its own token count, and the all-reduce averages gradients uniformly across ranks — mean-of-means again, now between GPUs that drew different-length sequences. The fix is the same shape: gather the token counts before normalizing.
num_items_in_batch = accelerator.gather(num_items_in_batch).sum().item()That's an extra collective per step — a real cost, but the price of the run meaning what you think it means. With both fixes in place, Mueller's experiments show bs=8, accum=1 and bs=4, accum=2 producing loss curves that match to several decimal places, which is what "mathematically equivalent" was supposed to buy all along.
What to take from it
- "Equivalent" claims have preconditions. Gradient accumulation is equivalent to large-batch training only when the loss is a sum, or a mean over a count that's constant per micro-batch. Token-level losses with padding satisfy neither.
- If you write your own loop, sum first, divide last. Use
reduction="sum"per micro-batch and divide by the window's total real-token count. Never divide bygradient_accumulation_stepswhen token counts vary. - Audit your loss the cheap way. Run a few steps with accumulation on and off at the same effective batch size. The curves should be nearly identical. Marie found a years-old bug in one of the most-used training libraries with exactly this check.
- Watch the seams between defaults. The model's loss default and the trainer's accumulation logic were each fine. Correctness bugs in ML rarely throw exceptions; they train you a slightly different model and let you ship it.