A training run that cannot be reproduced cannot be debugged. When a loss curve diverges on Tuesday and not on Monday, you need to know whether the cause is your code change, a library upgrade, a flaky node, or plain randomness. This tutorial shows how to make PyTorch runs reproducible, what that costs, and where to stop, because bitwise determinism is a tool, not a default.
Three levels of reproducibility
- Statistical reproducibility. Different seeds give metrics within a known spread. This is the right goal for most production work.
- Same-hardware determinism. The same code, seed, library versions and GPU type give identical results. This is what you want for debugging.
- Cross-hardware bitwise identity. Mostly unrealistic. Different GPU architectures use different kernels and reduction orders.
Pick the level before you pay for it.
Step 1: Seed everything
import os, random
import numpy as np
import torch
def seed_everything(seed: int = 1234):
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed) # seeds CPU and all CUDA devices
torch.cuda.manual_seed_all(seed)
os.environ["PYTHONHASHSEED"] = str(seed) # only fully effective if set before interpreter start
Step 2: Ask PyTorch for deterministic algorithms
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8" # needed for deterministic cuBLAS on CUDA 10.2+; set before CUDA work starts
torch.use_deterministic_algorithms(True) # raises if an op has no deterministic implementation
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False # autotuning can pick different kernels per run
With use_deterministic_algorithms(True), ops that have a deterministic variant switch to it, and ops that do not will raise a RuntimeError. That error is useful: it names the offending op. Use warn_only=True while you survey the damage, then decide per op whether to replace it or accept it. Common offenders are scatter and index-add style operations, some upsampling backward passes, and torch.Tensor.put_. Check the "Reproducibility" notes in the PyTorch docs for your version, since the list changes between releases.
Step 3: Make data loading deterministic
Data order is the most common hidden source of variance. Give the DataLoader its own generator and seed the workers:
def seed_worker(worker_id):
worker_seed = torch.initial_seed() % 2**32
np.random.seed(worker_seed)
random.seed(worker_seed)
g = torch.Generator()
g.manual_seed(1234)
loader = torch.utils.data.DataLoader(
dataset,
batch_size=64,
shuffle=True,
num_workers=4,
worker_init_fn=seed_worker,
generator=g,
)
For distributed training, call sampler.set_epoch(epoch) on a DistributedSampler each epoch, otherwise the shuffle is the same every epoch (or differs by accident in resumed runs). If you augment on the GPU, route that randomness through an explicit torch.Generator(device="cuda") rather than the global state.
Step 4: Make resumption exact
A run restarted from a checkpoint should match one that never stopped. Save more than weights:
state = {
"model": model.state_dict(),
"optim": optimizer.state_dict(),
"sched": scheduler.state_dict(),
"scaler": scaler.state_dict() if scaler else None,
"step": step,
"rng": {
"python": random.getstate(),
"numpy": np.random.get_state(),
"torch": torch.get_rng_state(),
"cuda": torch.cuda.get_rng_state_all(),
},
}
torch.save(state, path)
You also need to restore the data position (epoch and batch index, or the sampler state) so you do not replay or skip samples.
Step 5: Verify, do not assume
Write a test that runs 20 steps twice and compares:
def run(seed):
seed_everything(seed)
model, opt, loader = build()
losses = []
for i, (x, y) in enumerate(loader):
if i == 20: break
loss = criterion(model(x.cuda()), y.cuda())
opt.zero_grad(); loss.backward(); opt.step()
losses.append(loss.item())
return losses
assert run(1) == run(1), "run is not deterministic"
Run it in CI on the GPU type you train on. When it fails, bisect: disable torch.compile, then mixed precision, then multi-worker loading, then distributed, and re-test at each stage. The first change that restores equality points at the culprit.
What determinism costs
- Speed. Deterministic kernels are sometimes slower, because they avoid atomics. Measure on your own model rather than trusting a generic figure. Keep
cudnn.benchmark=Falseonly on debug runs if the cost is material. - Compilation.
torch.compilecan change fusion and reduction order across versions. Pin your PyTorch version and compare compiled runs only against compiled runs. - Distributed reductions. All-reduce order depends on topology and world size. Changing the number of GPUs changes results even with identical seeds. Treat world size as part of the run's identity.
- Mixed precision. BF16 is deterministic given identical kernels, but small rounding differences make "equal" mean "equal on this stack" only.
A practical policy
- Production training: seed, set the worker seeds, checkpoint RNG state, and judge changes against a measured seed-to-seed spread (run three seeds on the baseline once).
- Debugging and regression hunting: switch on full determinism, fix hardware and versions, and bisect.
- Record the environment with each run: PyTorch, CUDA and cuDNN versions, GPU model, world size, seed and git commit. Most "irreproducible" results turn out to be an unrecorded dependency change.
Common pitfalls
- Seeding after building the model, so initial weights differ.
- Using
num_workers>0withoutworker_init_fnand relying on NumPy random in the dataset. - Dropout and augmentation consuming RNG at different rates after a code change, which shifts every later random draw.
- Comparing a single seed's metric to another's and calling a 0.2 point change an improvement.
Need help?
If your team is chasing a regression it cannot reproduce, or you want a seed-variance baseline and a deterministic CI check set up for your training pipeline, IntelliSensei's consultants can help. Get in touch through our contact page.