RMSNorm

Core Concepts

Root Mean Square Layer Normalization (RMSNorm) is the normalization operation used in Qwen2.5 and most modern transformer architectures. It stabilizes activations before attention and MLP sub-blocks.

Formula

RMSNorm(x) = x / RMS(x) * gamma

where:
  RMS(x) = sqrt( (1/d) * sum(x_i^2) )
  gamma   = learned scale vector  [shape: d_model]
  d       = hidden_size = 896

Unlike LayerNorm, RMSNorm does not subtract the mean and has no learned bias term. It only re-scales the input so its root-mean-square equals 1, then applies the learned gamma scale.

Where it appears

Each transformer layer has two RMSNorm operations:

• Input norm (input_layernorm) — applied to the residual stream before the attention sub-block.
• Post-attention norm (post_attention_layernorm) — applied to the residual stream before the MLP sub-block.

There is also a final model norm (model.norm) applied to the residual stream after all 24 layers, before the unembedding step.

Real numbers (Qwen2.5-0.5B-Instruct, layer 0)

TokenPrint visualization

In the 3D view, RMSNorm operations are rendered as narrow cylindrical collars around the transformer spine. In the Walkthrough RMSNorm chapter, the worked formula is displayed with real input values and real gamma weights for the example sentence.