MLP / SwiGLU

Core Concepts

After the attention sub-block, each transformer layer applies a feed-forward MLP. Qwen2.5 uses SwiGLU, a gated activation function that produces stronger representations than a simple two-layer MLP.

SwiGLU formula

MLP(x) = down_proj( SiLU(gate_proj(x)) * up_proj(x) )

where:
  gate_proj: [896, 4864]    (hidden_size -> ffn_size)
  up_proj:   [896, 4864]    (hidden_size -> ffn_size)
  down_proj: [4864, 896]    (ffn_size -> hidden_size)

SiLU(x) = x * sigmoid(x)   (Swish activation)

The gate projection produces the gate signal; the up projection produces the candidate values. Element-wise multiplication after the SiLU gate selects which values pass through. The down projection compresses back to the residual stream dimension.

FFN expansion ratio

The intermediate dimension is 4,864, giving an expansion ratio of 4864 / 896 ≈ 5.43×. This is sized to the actual FFN dimension in the model config, not a rounded approximation.

TokenPrint visualization

The MLP sub-block is rendered as a tapered funnel shape in the 3D model. Three labeled prongs represent the gate, up, and down projections. The Walkthrough MLP chapter shows the SwiGLU formula with the real intermediate dimensions annotated and the gate vs. up projections distinguished geometrically.