Transformer Layers
A transformer model is a stack of identical layers, each refining the token representations through two sub-blocks: attention and MLP. The reference model has 24 layers.
The residual stream
The central abstraction is the residual stream — a sequence of vectors, one per token, that accumulates information as it passes through each layer. Each layer reads from the stream, computes a delta, and writes it back via addition:
x_out = x_in + Attention(RMSNorm(x_in)) x_out = x_out + MLP(RMSNorm(x_out))
The residual connection (the addition) is what makes deep transformers trainable — gradients flow directly through the additions, bypassing each sub-block.
Layer structure
Each of the 24 layers contains:
| Component | Tensor name pattern | Parameters (layer 0) |
|---|---|---|
| Input RMSNorm | model.layers.N.input_layernorm.weight | 896 |
| Q projection | model.layers.N.self_attn.q_proj.weight | 896 × 896 |
| K projection | model.layers.N.self_attn.k_proj.weight | 128 × 896 |
| V projection | model.layers.N.self_attn.v_proj.weight | 128 × 896 |
| Output projection | model.layers.N.self_attn.o_proj.weight | 896 × 896 |
| Post-attention RMSNorm | model.layers.N.post_attention_layernorm.weight | 896 |
| MLP gate projection | model.layers.N.mlp.gate_proj.weight | 4864 × 896 |
| MLP up projection | model.layers.N.mlp.up_proj.weight | 4864 × 896 |
| MLP down projection | model.layers.N.mlp.down_proj.weight | 896 × 4864 |
TokenPrint visualization
In the 3D view, each layer is represented as a horizontal slice of the transformer stack. The components within a layer are rendered as distinct 3D shapes:
- RMSNorm collars — narrow cylindrical rings at the top and bottom of each layer
- GQA blade array — the Q/K/V attention structure (14 Q blades, 2 KV groups)
- SwiGLU funnel — the MLP sub-block with gate, up, and down projections
- Residual splines — data wires connecting the residual stream through each layer