What Is the Difference Between LayerNorm and BatchNorm?

The Short Answer

nn.LayerNorm normalizes one sample at a time, over the dimensions you name in normalized_shape. nn.BatchNorm2d normalizes each channel, using statistics from the whole batch. The choice changes which tensors share a statistic, how many parameters the layer holds, and what happens in evaluation mode.

Measured Parameters

These numbers come from PyTorch 2.8.0 on a CPU build. Each layer was built with default options, so both weight and bias exist.

python3 -c "import torch, torch.nn as nn; ..."

nn.LayerNorm([32, 128])   weight (32, 128)   parameters 8192
nn.BatchNorm2d(64)        parameters 128
                          running_mean (64,)

The counts differ by a factor of 64 for the same nominal "size". LayerNorm holds one weight and one bias for every element of normalized_shape, so 2 × 32 × 128 = 8,192. BatchNorm2d holds one weight and one bias for every channel, so 2 × 64 = 128. BatchNorm also stores running_mean and running_var, one value per channel. Those buffers are not counted as parameters.

Train Mode and Eval Mode

BatchNorm behaves differently in the two modes. LayerNorm does not. The test below passes the same input to the same module twice, once in train mode and once in eval mode.

bn = nn.BatchNorm2d(64)
x  = torch.randn(4, 64, 10, 10)
bn.train(); a = bn(x)
bn.eval();  b = bn(x)
# a != b, because eval mode uses the stored running statistics

ln = nn.LayerNorm(100)
y  = torch.randn(4, 10, 100)
ln.train(); c = ln(y)
ln.eval();  d = ln(y)
# c == d, because LayerNorm has no running statistics

Measured result: torch.allclose(a, b) is False for BatchNorm2d. torch.allclose(c, d) is True for LayerNorm. A LayerNorm layer has no state to update between the two modes, so a checkpoint carries no batch statistics for it.

Dependence on Batch Size

BatchNorm needs more than one value per channel in train mode. With a batch of one, PyTorch raises an error before any output exists:

nn.BatchNorm1d(8).train()(torch.randn(1, 8))
# ValueError: Expected more than 1 value per channel when training,
#             got input size torch.Size([1, 8])

A LayerNorm layer accepts a batch of one, because its statistic comes from the sample itself. The mean of each normalized output vector is zero. Measured value for the first vector after nn.LayerNorm(100): 1.4e-08, which is zero within float32 rounding.

Which Layer to Use

• Use LayerNorm when each sample must be handled on its own: transformers, sequence models, and any model that runs with batch size 1 at inference.
• Use BatchNorm2d when the statistics of the batch are stable and the model is a convolutional image network trained with a large batch.
• Use BatchNorm1d for 2D or 3D inputs, and BatchNorm2d for 4D inputs. The rank check is strict: a 4D tensor passed to BatchNorm1d raises an error.