Q / K / V

Core Concepts

Inside each attention layer, the normalized residual stream is projected into three separate matrices — Query (Q), Key (K), and Value (V) — each with its own learned weight matrix. These projections are the input to the attention computation.

Projections

Projection	Weight shape	Output shape (7-token input)	Role
Q	`[896, 896]`	`[7, 896]`	What this token is looking for
K	`[128, 896]`	`[7, 128]`	What this token offers to be found by
V	`[128, 896]`	`[7, 128]`	The information to extract if attended to

Notice that K and V have much smaller weight matrices than Q. This is because Qwen2.5 uses Grouped Query Attention (GQA) with only 2 KV heads compared to 14 Q heads — the K and V projections are shared across groups of Q heads.

Splitting into heads

After the projection, each matrix is split into individual heads by reshaping:

Q: [T, d_model] -> [T, n_heads, head_dim]  = [7, 14, 64]
K: [T, d_kv]    -> [T, n_kv_heads, head_dim] = [7, 2, 64]
V: [T, d_kv]    -> [T, n_kv_heads, head_dim] = [7, 2, 64]

Each head operates independently on its own 64-dimensional slice. RoPE is then applied to Q and K within each head before the attention computation.

Scaling factor

Before the softmax, the dot products Q @ K^T are divided by sqrt(head_dim) = sqrt(64) = 8.0. Without this scaling, large head dimensions would push dot products into regions where the softmax gradient vanishes (the “attention collapse” problem).