What is d_k and why 64 or 128?

d_k = d_model / num_heads. Common choices are 64 or 128 for stable training and efficient attention.

How does GQA reduce KV cache?

GQA uses fewer key/value heads (num_kv_heads) than query heads. KV cache scales by num_kv_heads/num_heads compared to MHA.

What is the difference between MHA, MQA, GQA?

MHA: full KV heads. MQA: single KV head. GQA: intermediate number of KV heads, balancing quality and cache efficiency.

Attention Head Dimension Calculator

heytensor.com, d_k, d_v, projections, KV cache for MHA / MQA / GQA

d_model (hidden dim)

Number of attention heads

Number of layers (for KV cache)

Sequence length

Batch size

Precision (bytes per element)

Num KV heads for GQA (user-defined)

d_k = 128 recommended

Core dimensions & parameters

per layer

d_k / d_v,

Q param,

K param,

V param,

Output proj,

Total attn params,

KV cache size (MHA),

KV cache (MQA),

KV cache (GQA-4),

KV cache comparison, MHA / MQA / GQA

Variant	KV heads	KV cache size (MB)	Relative vs MHA
MHA (full)	32	,	1.0x
MQA	1	,	,
GQA-2	2	,	,
GQA-4	4	,	,
GQA-8	8	,	,

KV cache = 2 × layers × num_kv_heads × d_k × bytes_per_element × seq_len × batch_size. GQA-k uses k KV heads.

d_k = d_model / num_heads

Recommended: 64 or 128. Current: 128

Q/K/V projection: d_model × d_model each. Output projection: d_model × d_model.

Related attention tools

Attention Memory Ledger separates activation tensors and the sequence-squared score allocation. Causal Attention Mask Lab visualizes which query-key pairs are allowed.

TENSOR PREFLIGHT · DOWNLOADABLE PYTORCH TOOLKIT

Take the next step in your own PyTorch model.

Trace a local forward pass, inspect tensor shapes in an HTML report, and turn a repair into a regression check. Includes eight broken-and-fixed workflows, source code and setup instructions.

Is this toolkit right for my problem?

Use the free calculator for a layer formula or a quick shape check.

Consider Tensor Preflight when you have a local PyTorch model and want a forward-pass report plus repeatable checks. Python 3.9+ and PyTorch 2.8+ are required. It does not run your model in this website or automatically repair it.

Open the sample report before buying to check the output format.

Python 3.9+ · PyTorch 2.8+ · Runs locally

Get Tensor Preflight — $29 One-time payment · ZIP download See a sample report