Numeric precision is the cheapest performance lever in deep learning and the one most often left at its default. Training in BF16 instead of FP32 roughly doubles throughput on any modern GPU; serving in INT8 or FP8 halves memory and raises throughput again. This guide is the decision aid we use in 2026: which precision for which job, on which hardware, with the PyTorch patterns to do it safely.
The precision cheat-sheet
| Format | Bits | Range | What it is for | Hardware with native tensor-core support |
|---|---|---|---|---|
| FP32 | 32 | huge | reference, optimizer state, reductions | everything |
| TF32 | 19 (internal) | FP32 range | FP32 matmuls at near-BF16 speed | Ampere and later (A100, RTX 30xx onward) |
| FP16 | 16 | narrow (max 65,504) | legacy mixed precision; needs loss scaling | Volta and later |
| BF16 | 16 | FP32 range, 8-bit mantissa | default training and inference precision | Ampere and later |
| FP8 (E4M3 / E5M2) | 8 | small | large-model training and inference | Hopper and later (H100, H200, B200, RTX 40xx/50xx for inference) |
| INT8 | 8 | integer | post-training quantized inference | Turing and later, most CPUs |
| INT4 / NF4 | 4 | integer / normal-float | weight-only LLM inference, QLoRA training base | via dequant kernels; any CUDA GPU |
Two things to internalize. BF16 keeps FP32's exponent range, so it does not overflow the way FP16 does; that is why it replaced FP16 as the default and why GradScaler is unnecessary with it. And FP8 is not a drop-in: it needs per-tensor scaling handled by a library, which is why it lives behind Transformer Engine and torchao rather than a dtype flag.
Step 1: turn on TF32
If you do nothing else, do this. It makes FP32 matmuls and convolutions run on tensor cores with negligible accuracy impact.
import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True
# or, in recent releases:
torch.set_float32_matmul_precision("high")
Step 2: BF16 autocast for training
Mixed precision means the forward and backward run in BF16 where safe and FP32 where not (reductions, softmax, loss), with FP32 master weights in the optimizer. torch.autocast does the routing:
model = model.cuda()
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, fused=True)
for batch in loader:
x, y = (t.cuda(non_blocking=True) for t in batch)
with torch.autocast("cuda", dtype=torch.bfloat16):
logits = model(x)
loss = torch.nn.functional.cross_entropy(logits, y)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
optimizer.zero_grad(set_to_none=True)
No GradScaler. If you are on Volta/Turing hardware without BF16 support, use dtype=torch.float16 and wrap with torch.amp.GradScaler("cuda"): scaler.scale(loss).backward(), scaler.step(optimizer), scaler.update(). The scaler exists purely to keep FP16 gradients from underflowing; it is the tell that a codebase predates BF16.
Where memory, not compute, is the limit (large models, long sequences), cast the model itself to BF16 (model.to(torch.bfloat16)) and keep FP32 master weights via the optimizer or FSDP2's MixedPrecisionPolicy. Pure-BF16 weights without an FP32 master copy lose small updates and stall training; do not do it for long runs.
Step 3: FP8 training, where it applies
FP8 training is production reality for large transformer pretraining and fine-tuning on Hopper-class and newer GPUs, and not worth the complexity below a few billion parameters. The two practical routes in PyTorch are Transformer Engine (replace nn.Linear and attention with TE modules and wrap the forward in te.fp8_autocast) and torchao's Float8Linear conversion, which swaps linear layers in an existing model:
from torchao.float8 import convert_to_float8_training
model = convert_to_float8_training(model) # swaps eligible nn.Linear layers
model = torch.compile(model) # fp8 gains depend on compile fusion
Expect 1.3-1.6x over BF16 on matmul-dominated models, with rowwise or tensorwise scaling chosen by the library. Validate loss curves against a BF16 run for the first thousand steps before trusting a long FP8 job.
Step 4: post-training INT8 quantization for serving
Once a model is trained, serving it in INT8 reduces memory bandwidth and lets you use integer tensor cores. torchao provides the current PyTorch-native API:
import torch
from torchao.quantization import quantize_, Int8DynamicActivationInt8WeightConfig
model = model.eval().cuda()
quantize_(model, Int8DynamicActivationInt8WeightConfig())
model = torch.compile(model, mode="max-autotune")
Dynamic activation quantization needs no calibration data; weights are quantized once, activations per batch. For LLM weight-only quantization, torchao's Int4WeightOnlyConfig and Int8WeightOnlyConfig are the equivalents and pair with vLLM and ExecuTorch. Whatever you quantize, measure accuracy on your evaluation set and compare with the BF16 baseline. Typical deltas: under 0.5% top-1 on image classifiers for INT8; under one point on most LLM benchmarks for INT8 and well-tuned INT4; larger regressions mean a layer needs excluding or calibration data is unrepresentative.
Decision table
| Situation | Train in | Serve in |
|---|---|---|
| CNN or small transformer, Ampere+ | BF16 autocast + TF32 | INT8 (torchao) or BF16 with torch.compile |
| Fine-tuning a 1-8B LLM on one GPU | BF16 autocast; QLoRA NF4 base if memory-bound | INT8 or INT4 weight-only via vLLM |
| 8-70B LLM, multi-GPU, Hopper+ | BF16 with FSDP2; FP8 when matmul-bound | FP8 (vLLM) or INT4 AWQ/GPTQ |
| Legacy Volta/Turing GPUs | FP16 + GradScaler | FP16 or INT8 |
| CPU inference | FP32 training elsewhere | INT8 (torchao or ONNX Runtime) |
| Edge / mobile | BF16 training elsewhere | INT8 or 4-bit via ExecuTorch delegates |
Pitfalls
Reductions and normalization layers should stay in FP32; autocast handles this, hand-casting does not. Check torch.isfinite(loss) in the training loop and log the first non-finite step rather than discovering it in a dead run. Keep evaluation precision identical to serving precision; evaluating in BF16 and serving in INT8 tells you nothing about what users get. And pin versions: torchao and Transformer Engine APIs move quickly, and a working quantization recipe is worth locking in requirements.txt.
If your training runs or serving costs would benefit from a precision review, this is core work for our performance optimization and inference optimization practices. Contact us to discuss your workload.