How Many Parameters Does Linear(256, 10) Have?

Q: How Many Parameters Does Linear(256, 10) Have?

Linear(256, 10) has 2,570 parameters. The formula is: in_features * out_features + out_features (bias) = 256 * 10 + 10 = 2,570. The weight matrix has 2,560 elements and the bias vector has 10 elements.

Linear(256, 10) has 2,570 trainable parameters. This includes 2,560 weights and 10 bias terms.

Formula Breakdown

For a Linear layer, the parameter count is:

parameters = in_features * out_features + out_features (bias)
parameters = 256 * 10 + 10
parameters = 2,560 + 10
parameters = 2,570

The weight matrix W has shape (10, 256) = 2,560 values. The bias vector b has 10 values. Together: 2,570 trainable parameters.

Memory Usage

In float32, this layer uses 0.01 MB of memory for weights alone. During training with Adam optimizer, multiply by 3 (weights + momentum + variance) = 0.03 MB.

Architecture Context

This layer configuration is found in small classifiers for MNIST/CIFAR-10 (10 classes). Understanding parameter counts helps you estimate model size, memory requirements, and the risk of overfitting. Layers with more parameters need more training data and compute to train effectively.

Linear layers are often the most parameter-heavy part of a network. For example, VGG-16 has ~124M parameters in its three fully connected layers versus only ~14M in all its convolutional layers. Modern architectures minimize linear layers by using global average pooling.

PyTorch Code to Verify

import torch.nn as nn

layer = nn.Linear(256, 10)

# Count parameters
total = sum(p.numel() for p in layer.parameters())
print(f"Total parameters: {total}")  # 2,570

# Break it down
print(f"Weight shape: {layer.weight.shape}")  # (10, 256)
print(f"Weight params: {layer.weight.numel()}")  # 2,560
print(f"Bias shape: {layer.bias.shape}")  # (10,)
print(f"Bias params: {layer.bias.numel()}")  # 10

# Without bias
layer_no_bias = nn.Linear(256, 10, bias=False)
print(f"Without bias: {sum(p.numel() for p in layer_no_bias.parameters())}")  # 2,560

Comparison: With vs. Without Bias

Configuration	Parameters
Linear(256, 10) (with bias)	2,570
Linear(256, 10, bias=False)	2,560

When using BatchNorm after a convolutional layer, the bias is redundant because BatchNorm has its own bias term. Setting bias=False saves 10 parameters per layer.