Grouped Query Attention
Grouped Query Attention (GQA) reduces the memory cost of the KV cache by sharing Key and Value heads across groups of Query heads. Qwen2.5-0.5B uses 14 Q heads and 2 KV heads — each KV head is shared by 7 Q heads.
Motivation
In standard multi-head attention (MHA), every Q head has its own K and V head. During autoregressive decoding, the KV cache must store the K and V activations for every head at every layer at every past position. With GQA, only the KV heads are stored, reducing the cache size by a factor of n_heads / n_kv_heads = 14 / 2 = 7.
Topology (Qwen2.5-0.5B)
| Parameter | Value |
|---|---|
Q heads (num_heads) | 14 |
KV heads (num_kv_heads) | 2 |
| Head dimension | 64 |
| Group size (Q per KV) | 7 |
| Q projection shape | [896, 896] |
| K projection shape | [128, 896] |
| V projection shape | [128, 896] |
During the attention computation, each K and V head is expanded (repeated) 7 times before the dot product, so the shapes become compatible:
Q: [T, 14, 64] (14 independent heads) K: [T, 2, 64] -> expand -> [T, 14, 64] V: [T, 2, 64] -> expand -> [T, 14, 64]
TokenPrint visualization
In the 3D model, the GQA structure is visible in the attention geometry: 14 Q blades arranged in a fan, with 2 KV groups clearly differentiated by their reduced geometry. In the Self-Attention chapter of the Walkthrough, each head's attention heatmap can be selected individually to see how the 14 independent Q heads attend differently despite sharing K and V.