Does Mixed Precision Training Halve GPU Memory Usage?
The Short Answer
No. Mixed precision halves the bytes of activation tensors and of any weight tensor you convert to float16. In standard automatic mixed precision the master weights, the gradients, and the optimizer state stay float32. So the total of a training step falls by much less than half, and a model with a large Adam state saves almost nothing.
Measured Parameter Bytes
PyTorch 2.8.0, CPU build. The model is two nn.Linear(1024, 1024) layers with a ReLU between them: 2,099,200 parameters.
model.to(torch.float32) 8,396,800 bytes (4 bytes each)
model.to(torch.float16) 4,198,400 bytes (2 bytes each)The ratio is exactly 2 when the tensors themselves hold float16. An activation tensor of N elements takes 4N bytes in float32 and 2N bytes in float16, so activations also halve.
What Standard AMP Keeps in Float32
torch.autocast casts inside each operation. It does not convert the stored parameters. The module keeps float32 weights, the backward pass writes float32 gradients, and Adam keeps two float32 state tensors per parameter. For the same model:
parameters (float32) 8,396,800 bytes
gradients (float32) 8,396,800 bytes
Adam state (2 tensors, fp32) 16,793,600 bytes
total without activations 33,587,200 bytesAll three lines keep the same size with autocast on. The optimizer state alone is twice the size of the float32 parameters, so it dominates the footprint of a training step.
Where the Saving Comes From
| Block | Float32 | Standard AMP |
|---|---|---|
| Master weights | 4 bytes per value | 4 bytes per value |
| Gradients | 4 bytes per value | 4 bytes per value |
| Adam state | 8 bytes per value | 8 bytes per value |
| Activations | 4 bytes per value | 2 bytes per value |
The one row that halves is the row that dominates small models and long sequences. A model with large activations per parameter saves close to half. A model with a large optimizer state, such as a wide transformer trained with Adam, saves a small fraction.
If You Convert the Weights Instead
model.half() does store float16 parameters, and the measured size drops to 4,198,400 bytes for this model. That is a different setup from autocast. It needs loss scaling, and the optimizer state usually stays float32. Measure the whole step, not just the weights.
Measure Before You Change the Batch Size
Use the mixed precision estimator to compare the two setups for your layer sizes. The largest batch that fits is set by the total, not by the weight bytes.