What is BERT-base hidden size?

BERT-base hidden size is 768. It has 12 transformer layers, 12 attention heads, intermediate size 3072 (4 x 768), vocabulary size 30,522, and max position 512. Every token is embedded into a 768-dimensional vector.

How long does it take to load BERT-base into memory?

Loading the ~440 MB FP32 checkpoint takes roughly 1 to 3 seconds on a modern CPU with an NVMe drive, plus a similar peak of ~440 MB of RAM for the weights themselves. Loading as FP16 halves both file size and RAM to about 219 MB.

How Big Is BERT-base in Megabytes on Disk?

Q: How big is BERT-base in megabytes on disk?

BERT-base is about 438 MB on disk in FP32, 219 MB in FP16, and 110 MB in INT8. It has 109,482,240 parameters. At 4 bytes per parameter, FP32 works out to 109,482,240 x 4 / 1,000,000 = ~438 MB. The official bert-base-uncased checkpoint from Hugging Face is ~440 MB for both pytorch_model.bin and model.safetensors, matching the arithmetic.

BERT-base is ~438 MB on disk in FP32, ~219 MB in FP16, and ~110 MB in INT8. It has 109,482,240 parameters and a hidden size of 768 across 12 layers and 12 attention heads.

BERT-base Config and Hidden Size

Hidden size (H):    768   <-- the 768 answer
Intermediate size:   3072 (4 * H)
Attention heads:    12
Layers:         12
Vocabulary size:    30,522
Max position:      512

BERT-base hidden size is 768. That number appears everywhere in the architecture: the embedding dimension, the size of every attention and feed-forward tensor, and the pooler output. It is the single most-quoted config value for the model.

Size on Disk by Precision

Precision | Bytes/param | On-disk size (MB)
----------+-------------+------------------
FP32      |      4      |   ~438
FP16      |      2      |   ~219
INT8      |      1      |   ~110

Formula: 109,482,240 params x bytes / 1,000,000 = MB

Why 438 MB, Not 418 MB

The math uses decimal megabytes (1 MB = 1,000,000 bytes), which is how filesystem sizes and Hugging Face checkpoint sizes are reported. In binary mebibytes (1 MiB = 1,048,576 bytes) the FP32 checkpoint is ~418 MiB. Both are correct; the file on disk reads as ~440 MB because of safetensors headers and the optimization state that ships inside some checkpoints.

The official bert-base-uncased checkpoint from Hugging Face contains a pytorch_model.bin of about 440 MB and a model.safetensors of about 440 MB, the exact parameter weights, matching the 438 MB arithmetic above within rounding. There is no hidden "extra" copy of the model; the two files are the same weights in two serialization formats.

Loading Time and RAM

FP32 (~440 MB file):
  Disk read:  ~1-3 s  (NVMe)
  RAM for weights:  ~440 MB
FP16 (~219 MB file):
  Disk read:  ~0.5-1.5 s
  RAM for weights:  ~219 MB
Peak during torch.load: ~2x weights

Loading is bounded by disk I/O and peak memory. During deserialization torch.load briefly holds the checkpoint bytes plus the reconstructed tensors, so peak RAM spikes to roughly twice the final weight size before settling. On a modest CPU-only machine, BERT-base FP32 loads comfortably in a few seconds and runs inference with < 1 GB of RAM.

Quick Reference

Parameters:    109,482,240
FP32 on disk:  ~438 MB
FP16 on disk:  ~219 MB
INT8 on disk:  ~110 MB
Hidden size:   768
Layers:        12
Heads:         12
Vocab:         30,522

Try the Parameter Counter