Causal Attention Mask Lab

Mask rule and PyTorch semantics

For query row q and key column k, this lab allows a cell when k ≤ q + offset. If the local window is nonzero, it also requires k ≥ q + offset − window + 1. Both rules are intersected with valid sequence positions. Offset 0 and no local window produce the familiar lower-triangular causal mask.

PyTorch scaled dot-product attention treats true in a boolean attention mask as allowed. An additive mask expresses the same rule with 0 at allowed cells and negative infinity at blocked cells. See the official PyTorch scaled_dot_product_attention documentation for the API's mask behavior. Pass either generated mask as attn_mask; do not also set is_causal=True.

How to read the preview

Each row represents one query position and each column represents one key position. An allowed cell means that query can use the corresponding key when attention weights are computed. A blocked cell receives no probability mass after masking. Moving the offset changes the diagonal boundary. Adding a local window keeps the causal boundary but removes keys that fall too far behind it, producing a diagonal band instead of a full lower triangle.

The preview is deliberately bounded so a long sequence cannot create an unresponsive page. When the requested sequence exceeds the preview limit, the lab renders the leading square only. The allowed, blocked, and total counters still describe the full requested matrix because they are calculated row by row with exact integer arithmetic, independently of the rendered cells.

Boolean versus additive masks

The two exports encode the same allowed-cell predicate. Boolean mode is easier to inspect because every cell is explicitly true or false. Additive mode is useful when an attention implementation expects a bias tensor: allowed positions add zero while blocked positions add negative infinity before softmax. Dtype, device, batch, and head broadcasting remain the caller's responsibility; the generated snippet demonstrates the mask rule rather than guessing those runtime details.

Method and limitations

This utility is a deterministic mask constructor, not a runtime benchmark. It does not predict kernel choice, memory use, latency, or numerical behavior. Framework behavior can change by version, so verify the generated shape and mask convention against the linked primary documentation for the version you run. The fixtures exercise the same validation and counting functions as the interactive controls, including an invalid input and a truncated-preview case.