Validated utility release
QKV Activation Memory Calculator and Tensor Ledger
Account for the conceptual tensors in transformer attention, line by line. Isolate the quadratic score matrix, compare two sequence lengths, and audit every subtotal in exact integer bytes.
Built by Michael Lip · calculator runs entirely in your browser
Tensor dimensions
No inputs leave this page.Scenario A · baseline
2,048 tokens · 2-byte scalars
| Tensor / subtotal | Elements | Exact memory |
|---|
Scenario B · comparison
8,192 tokens · 2-byte scalars
| Tensor / subtotal | Elements | Exact memory |
|---|
Scope warning. “Fused” here omits only the conceptually materialized attention-score matrix. Framework kernels may save, tile, or recompute other tensors. This ledger is not a promise of peak VRAM; measure your actual workload with the chosen framework, hardware, and kernel.
What this ledger counts
This calculator deliberately exposes a small conceptual ledger instead of producing an opaque “training VRAM” number. For each layer it counts the residual-stream input, the three query/key/value projection outputs, the attention output, and, only in the materialized path, the full query-key score matrix. It then multiplies each per-layer subtotal by the layer count to show an all-layer upper-bound ledger.
Q/K/V elements = 3 × batch × sequence × hidden
score elements = batch × heads × sequence × sequence
attention output elements = batch × sequence × hidden
bytes = elements × bytes per scalar
materialized subtotal = residual + Q/K/V + score + output
fused comparison = materialized subtotal − score
The sequence × sequence factor is the important nonlinear term. With every other input fixed, doubling the sequence length doubles the residual, Q/K/V, and output tensors, but quadruples a materialized score matrix. The two-scenario view makes that difference directly inspectable.
Worked example
The live calculator is computing this example from Scenario A.
Assumptions and limitations
Included
- One residual-stream input per layer
- Separate Q, K, and V projection outputs
- Conceptual full attention-score matrix
- One attention output per layer
- An all-layer upper bound for these tensors
Not included
- Model weights, gradients, and optimizer state
- CUDA workspace, allocator fragmentation, or caches
- MLP activations, masks, dropout, logits, or loss
- Kernel-specific saved tensors and recomputation
- Runtime, throughput, or hardware availability
All displayed byte values come from JavaScript BigInt multiplication. Inputs must be positive decimal integers, hidden size must be divisible by head count, and a scenario is rejected when any displayed allocation exceeds the documented ledger ceiling of 263 − 1 bytes. That ceiling is a deliberate guardrail, not an estimate of available GPU memory.
Methodology and primary sources
The tensor dimensions follow PyTorch’s documented batch, sequence, embedding, and head semantics. The full score term follows the reference scaled dot-product attention expression, where query is multiplied by the transpose of key before softmax. The fused comparison is motivated by exact attention implementations designed to avoid materializing that full intermediate in high-bandwidth memory; it does not imitate any single kernel’s internal allocations.
- PyTorch: scaled dot product attention, reference score expression and fused implementation notes.
- PyTorch: MultiheadAttention, batch, sequence, embedding, and head dimensions.
- PyTorch: activation checkpointing, recomputation tradeoffs and implementation caveats.
- Dao et al.: FlashAttention, primary paper on IO-aware exact attention.
Executable browser fixtures
Running five independent checks…
| Fixture | Expected | Observed | Status |
|---|