Attention

Core Concepts

Self-attention allows each token in a sequence to gather information from every other token. The result is that each token's representation is updated based on how much it “attends” to every other position.

Mechanism

Attention(Q, K, V) = softmax( Q K^T / sqrt(head_dim) ) V

where:
  Q  = query matrix  [T, n_heads * head_dim]
  K  = key matrix    [T, n_kv_heads * head_dim]
  V  = value matrix  [T, n_kv_heads * head_dim]
  T  = sequence length
  head_dim = 64 (for Qwen2.5-0.5B)

The softmax ensures each row of the attention weight matrix sums to 1 — every token distributes 100% of its attention across all positions.

Causal masking

Decoder-only models like Qwen2.5 apply a causal mask: token at position i can only attend to positions 0 through i. Future tokens are masked to negative infinity before the softmax, so they receive zero attention weight. This ensures the model is autoregressive — it cannot look ahead.

Real numbers (Qwen2.5-0.5B, “The cat sat on the mat.”)

Property	Value
Attention tensor shape	`[24, 14, 7, 7]` — (layers, heads, seq, seq)
Attention row sums	1.0000 (valid softmax)
Max abs error vs independent forward pass	0.00050 (floating-point rounding only)
Strongest link (layer 0, head 0)	“cat” attends to “The” with weight 0.864

TokenPrint visualization

In the Walkthrough Self-Attention chapter, the attention weight matrix for the example sentence is rendered as a heatmap. The strongest attention link is highlighted and labeled with the real weight value. You can select individual heads using the head grid controls.

In Generation mode, the active head's attention weights animate as each token is produced, showing where the model is focusing to make each prediction.