You set torch.manual_seed(0), seeded NumPy and Python's random, pinned every library version, ran your training script twice on the same GPU — and got two different models. Nothing is broken. A 2024 study from Oak Ridge National Laboratory and Groq trained the same graph neural network 1,000 times with every conventional source of randomness controlled, and got 1,000 unique sets of weights. The randomness you seeded was never the only randomness in the system.

Floating-point addition is not associative

The root cause is arithmetic, not RNGs. In real numbers, (a + b) + c == a + (b + c). In floating point, each addition rounds to the nearest representable value, so the order of operations changes the result:

python
>>> (0.1 + 1e20) - 1e20
0.0
>>> 0.1 + (1e20 - 1e20)
0.1

On a CPU running single-threaded code, summation order is fixed by the program, so this doesn't cause run-to-run variation. On a GPU, it does — because of how parallel reductions are implemented.

Many CUDA kernels accumulate into shared memory using atomicAdd. Atomics guarantee that no update is lost, but they guarantee nothing about order: thousands of threads race to commit their contributions, and the winner of each race depends on warp scheduling, clock jitter, and whatever else the SMs were doing. The Shanmugavelu et al. paper ("Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications", arXiv:2408.05148) puts it precisely: executing the additions in an unspecified order "is equivalent to applying a random permutation P to the series x_i before computing the serial sum." Same inputs, same kernel, different rounding — on every run.

How much drift, and does it matter?

Per operation, almost nothing. The paper measured upper-bound elementwise variability across runs of about 3.35e-6 for scatter_reduce and 5.03e-6 for index_add on the GPUs they tested (V100, H100, GH200, and AMD MI250X). Six decimal places of agreement sounds like reproducibility.

Training destroys that intuition. Gradients feed weight updates, weight updates feed the next forward pass, and nonlinear activations amplify tiny differences instead of averaging them away. In the paper's controlled experiment — a two-layer GraphSAGE network on the Cora citation dataset, 10 epochs, constructed so that "the only source of non-determinism in our implementation of this DNN is the index_add operation" — all 1,000 repeated training runs produced distinct weights. The models converged to similar loss values, but bit-for-bit they were 1,000 different artifacts. If your regression test asserts exact equality on weights or logits across training runs, it is testing the GPU scheduler.

The second saboteur: algorithm selection

Even if every kernel were order-stable, you can still get different results because the kernel itself changes between runs.

With torch.backends.cudnn.benchmark = True, cuDNN times several convolution algorithms on your actual shapes and picks the fastest. PyTorch's reproducibility notes warn that "due to benchmarking noise and different hardware, the benchmark may select different algorithms on subsequent runs." Different algorithm, different summation tree, different bits. Setting benchmark = False pins the selection — but the selected algorithm may itself use atomics, which is why there's a separate torch.backends.cudnn.deterministic = True flag for the convolution implementations.

cuBLAS has its own version of this. NVIDIA's documentation guarantees that routines from a given toolkit version "generate the same bit-wise results at every run when executed on GPUs with the same architecture and the same number of SMs" — note how narrow that is: same library version, same chip layout. And the guarantee "no longer holds when multiple CUDA streams are active," because the library may pick different internal implementations to optimize throughput across streams. The documented fix is the CUBLAS_WORKSPACE_CONFIG environment variable: :16:8 (may cost performance) or :4096:8 (costs roughly 24 MiB of GPU memory per handle) forces deterministic behavior even with concurrent streams.

What `torch.use_deterministic_algorithms(True)` actually does

PyTorch's one-line switch does two different things depending on the operation:

  1. Swaps in a deterministic implementation where one exists. CUDA convolutions, scatter_add_, index_add, cumsum on CUDA floats, the backward pass of torch.gather and Embedding, and a few dozen others silently switch to order-stable kernels.
  2. Throws a RuntimeError where none exists. torch.nn.functional.interpolate in bilinear/bicubic modes, NLLLoss on CUDA, CTCLoss backward, MaxUnpool2d, torch.bincount with weights on CUDA, grid_sample backward, and others simply have no deterministic CUDA implementation — PyTorch refuses to run them rather than lie to you.

The error-throwing behavior is a feature: it turns silent nondeterminism into a stack trace pointing at the exact op you need to replace or move to CPU.

The full recipe for a deterministic PyTorch run, straight from the official notes:

python
import torch, random, numpy as np

torch.manual_seed(0)          # seeds all devices, CPU and CUDA
random.seed(0)
np.random.seed(0)

torch.use_deterministic_algorithms(True)
torch.backends.cudnn.benchmark = False
# plus: export CUBLAS_WORKSPACE_CONFIG=:4096:8

DataLoader workers need their own care, because each worker is a separate process with its own RNG state. The documented pattern:

python
def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % 2**32
    numpy.random.seed(worker_seed)
    random.seed(worker_seed)

g = torch.Generator()
g.manual_seed(0)
DataLoader(dataset, num_workers=4,
           worker_init_fn=seed_worker, generator=g)

The price, and the fine print

Determinism costs real performance. Deterministic reductions replace free-running atomics with structured strategies — fixed-shape reduction trees, or sorting indices before accumulating — that add synchronization and memory traffic. In the paper's GraphSAGE inference benchmark on an H100, the deterministic variant ran at 3.92 ms versus 2.17 ms for the nondeterministic one — roughly a 1.8x slowdown for that workload. PyTorch's docs are blunter: "deterministic operations are often slower than nondeterministic operations, so single-run performance may decrease for your model." The actual penalty is workload-dependent; ops dominated by dense GEMMs barely notice, scatter-heavy graph workloads notice a lot.

And even after paying it, read the scope of the promise. PyTorch: "Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms," and "results may not be reproducible between CPU and GPU executions, even when using identical seeds." Full determinism buys you same code, same version, same hardware, same run twice. It does not make your A100 results match your H100 results, or survive a PyTorch upgrade.

When to pay for it

Treat determinism as a debugging tool and a validation tool, not a default:

  • Bisecting a regression: when you need to know whether a code change altered the math or you're just seeing run-to-run noise, deterministic mode is the only way to get a clean signal.
  • Testing: unit tests comparing tensors across runs need deterministic mode, or tolerances calibrated to real variability — the per-op numbers above are a starting point, but post-training drift is unbounded.
  • Everything else: accept that a training run is a sample from a distribution. If a metric decision hinges on a single seeded run, the honest fix is more runs, not more determinism flags — determinism makes runs repeatable, not representative.

The seed was never the whole story. The scheduler holds the rest of it.

References

reproducibilitypytorchcudafloating-pointdeterminism