How Many Parameters Does Linear(1024, 512) Have?
Linear(1024, 512) has 524,800 trainable parameters. This includes 524,288 weights and 512 bias terms.
Formula Breakdown
For a Linear layer, the parameter count is:
parameters = in_features * out_features + out_features (bias)
parameters = 1024 * 512 + 512
parameters = 524,288 + 512
parameters = 524,800The weight matrix W has shape (512, 1024) = 524,288 values. The bias vector b has 512 values. Together: 524,800 trainable parameters.
Memory Usage
In float32, this layer uses 2.00 MB of memory for weights alone. During training with Adam optimizer, multiply by 3 (weights + momentum + variance) = 6.01 MB.
Architecture Context
This layer configuration is found in general-purpose classifier heads. Understanding parameter counts helps you estimate model size, memory requirements, and the risk of overfitting. Layers with more parameters need more training data and compute to train effectively.
Linear layers are often the most parameter-heavy part of a network. For example, VGG-16 has ~124M parameters in its three fully connected layers versus only ~14M in all its convolutional layers. Modern architectures minimize linear layers by using global average pooling.
PyTorch Code to Verify
import torch.nn as nn
layer = nn.Linear(1024, 512)
# Count parameters
total = sum(p.numel() for p in layer.parameters())
print(f"Total parameters: {total}") # 524,800
# Break it down
print(f"Weight shape: {layer.weight.shape}") # (512, 1024)
print(f"Weight params: {layer.weight.numel()}") # 524,288
print(f"Bias shape: {layer.bias.shape}") # (512,)
print(f"Bias params: {layer.bias.numel()}") # 512
# Without bias
layer_no_bias = nn.Linear(1024, 512, bias=False)
print(f"Without bias: {sum(p.numel() for p in layer_no_bias.parameters())}") # 524,288Comparison: With vs. Without Bias
| Configuration | Parameters |
|---|---|
| Linear(1024, 512) (with bias) | 524,800 |
| Linear(1024, 512, bias=False) | 524,288 |
When using BatchNorm after a convolutional layer, the bias is redundant because BatchNorm has its own bias term. Setting bias=False saves 512 parameters per layer.