Attention Head Dimension Calculator

heytensor.com, d_k, d_v, projections, KV cache for MHA / MQA / GQA
d_k = 128 recommended

Core dimensions & parameters

per layer
d_k / d_v,
Q param,
K param,
V param,
Output proj,
Total attn params,
KV cache size (MHA),
KV cache (MQA),
KV cache (GQA-4),

KV cache comparison, MHA / MQA / GQA

VariantKV headsKV cache size (MB)Relative vs MHA
MHA (full)32, 1.0x
MQA1, ,
GQA-22, ,
GQA-44, ,
GQA-88, ,
KV cache = 2 × layers × num_kv_heads × d_k × bytes_per_element × seq_len × batch_size. GQA-k uses k KV heads.
d_k = d_model / num_heads
Recommended: 64 or 128. Current: 128
Q/K/V projection: d_model × d_model each. Output projection: d_model × d_model.

Related attention tools

Attention Memory Ledger separates activation tensors and the sequence-squared score allocation. Causal Attention Mask Lab visualizes which query-key pairs are allowed.

Recommended by our team

BeLikeNative.com

The #1 AI writing tool for freelancers, perfect grammar in any language, instantly.