How many parameters does BERT-large have?

BERT-large has 335,141,888 parameters by exact count, commonly cited as 340M. It has 24 transformer layers, hidden size 1024, intermediate size 4096, 16 attention heads, vocabulary size 30,522, and max position 512. Breakdown: embeddings contribute 31,782,912 parameters. Each of the 24 transformer layers has 12,596,224 parameters (self-attention 4,198,400, feed-forward 8,393,728, plus two LayerNorms of 2,048 each). All 24 layers total 302,309,376. The pooler adds 1,049,600.

Why is BERT-large called 340M when the exact count is 335.1M?

The round number 340M comes from the original BERT paper's model-size table, which reported BERT-large as 340M while also reporting the exact figure 335M in the same table. The 335,141,888 count is what you get by summing every trainable tensor: word embeddings 31,254,528, position embeddings 524,288, segment embeddings 2,048, embedding LayerNorm 2,048, 24 layers of 12,596,224, and a pooler of 1,049,600. The ~1.5% gap between 335M and 340M is rounding to one significant figure in the headline number.

How much memory does BERT-large need?

BERT-large params alone occupy about 1,278 MB in FP32 and about 639 MB in FP16. BERT-base needs about 418 MB (FP32) and 209 MB (FP16), so BERT-large is roughly 3.06x larger. Training with Adam adds gradients and two optimizer states, about 4x the parameter bytes, or roughly 5.0 GB on top of activations, which are usually the dominant term at sequence length 512.

BERT-large Has 340M Parameters, 24 Layers, Hidden Size 1024

BERT-large has approximately 340 million parameters (335,141,888 exact). 24 transformer layers, hidden size 1024, intermediate size 4096, 16 attention heads, vocabulary 30,522.

BERT-large Configuration

Hidden size (H):   1024
Intermediate size:  4096 (4 * H)
Attention heads:    16
Head dimension:    64 (1024 / 16)
Layers:         24
Vocabulary size:   30,522
Max position:     512

Every number below follows from that block. The four BERT-large dimensions that people quote together, 24 layers, 1024 hidden size, 16 heads, 340M parameters, are not independent: hidden size and layer count drive the total, and the head count only partitions the 1024-wide hidden dimension into 16 slices of 64. Changing the head count alone would not change the parameter count at all.

Parameter Breakdown

Component          | Parameters
-----------------------------|-------------
Word Embeddings       | 31,254,528  # 30522 * 1024
Position Embeddings     |  524,288  # 512 * 1024
Segment Embeddings      |   2,048  # 2 * 1024
Embedding LayerNorm     |   2,048  # 2 * 1024
Embeddings subtotal     | 31,782,912

Per Transformer Layer:
 Self-Attention (Q,K,V,O)  | 4,198,400  # 4 * (1024*1024 + 1024)
 Attention LayerNorm    |   2,048  # 2 * 1024
 Feed-Forward (up + down)  | 8,393,728  # 1024*4096 + 4096 + 4096*1024 + 1024
 FFN LayerNorm       |   2,048  # 2 * 1024
 Per-layer total      | 12,596,224

24 Transformer Layers    | 302,309,376  # 24 * 12,596,224

Pooler (1024 -> 1024)     | 1,049,600  # 1024*1024 + 1024
-----------------------------|-------------
Total (exact)         | 335,141,888
Total (commonly cited)    | ~340M

The exact sum is 335,141,888 parameters, which is 335.1M. The 340M figure you see in most blog posts and papers is the rounded headline number from the original BERT paper's model-size table, which listed BERT-large as 340M and its exact count as 335M in the same row. Neither is wrong: 335.1M is what sum(p.numel() for p in model.parameters()) returns, and 340M is what you write when you round to one significant figure.

Notice where the mass sits. The 24 transformer layers hold 302.3M of the 335.1M parameters, 90.2% of the model. Embeddings hold 31.8M, or 9.5%. The pooler is 1.0M, a rounding error. Inside a single layer, the feed-forward block is 8,393,728 parameters and self-attention is 4,198,400, so feed-forward is almost exactly two-thirds of every layer (66.6%) while attention is one-third (33.3%).

BERT-large vs BERT-base

                 | BERT-base     | BERT-large
-----------------------------+---------------+--------------
Layers           | 12       | 24
Hidden size (H)        | 768       | 1024
Intermediate size      | 3,072      | 4,096
Attention heads        | 12       | 16
Head dimension       | 64       | 64
Vocabulary / max pos   | 30,522 / 512  | 30,522 / 512
-----------------------------+---------------+--------------
Embeddings subtotal    | 23,837,184   | 31,782,912
Per-layer total      | 7,087,872    | 12,596,224
All layers         | 85,054,464   | 302,309,376
Pooler           |  590,592    |  1,049,600
-----------------------------+---------------+--------------
Total parameters     | 109,482,240   | 335,141,888
Reported as        | ~110M      | ~340M

BERT-large is 3.06x BERT-base in parameter count, and the extra 225.7M parameters come almost entirely from two multiplications. Doubling the layer count from 12 to 24 accounts for the bulk of it, and widening the hidden size from 768 to 1024 makes every remaining matrix larger by roughly (1024/768)² = 1.78x. A single BERT-large layer already carries 12.6M parameters, which is 1.78x a whole BERT-base layer. Head count scales with hidden size so that head dimension stays at 64 in both models, which is why 768/12 and 1024/16 both land on 64.

Memory Requirements

FP32 inference: ~1,278 MB (params only, 335,141,888 * 4 bytes)
FP16 inference: ~639 MB (335,141,888 * 2 bytes)
Gradients + Adam: ~5.0 GB (params + grads + 2 optimizer states, 16 bytes/param)

For contrast, BERT-base params only:
FP32 inference: ~418 MB
FP16 inference: ~209 MB

These are weight-storage figures, not peak memory. At inference the activation memory is a function of batch size and sequence length, and at sequence length 512 with a batch of 8 it routinely exceeds the weight footprint. During training, Adam keeps a copy of the weights, the gradients, the first-moment buffer, and the second-moment buffer, which is 16 bytes per parameter, so BERT-large needs about 5.0 GB just for optimizer state before a single activation tensor is allocated. That is the reason BERT-large fine-tuning is usually done in mixed precision or with gradient checkpointing on a 16 GB card.

Counting It Yourself

from transformers import BertConfig, BertModel

cfg = BertConfig(hidden_size=1024, num_hidden_layers=24,
         num_attention_heads=16, intermediate_size=4096,
         vocab_size=30522, max_position_embeddings=512)
model = BertModel(cfg)

sum(p.numel() for p in model.parameters())
# 335141888

sum(p.numel() for p in model.parameters()
  if p.requires_grad)
# 335141888

sum(p.numel() for p in model.embeddings.parameters())
# 31782912

sum(p.numel() for p in model.encoder.layer[0].parameters())
# 12596224

If your count comes out to 333,141,888 or 336M, check your config: add_pooling_layer=True is the default and adds 1,049,600, and a tokenizer file that differs from 30,522 vocabulary entries shifts the word embedding matrix by 1024 parameters per token. Both are common sources of a count that is a million or two off. The line-by-line arithmetic for the smaller model lives on the BERT-base parameter breakdown, and you can reproduce both columns interactively with the parameter counter.

Try the Parameter Counter