Residual Streams

Core Concepts

The residual stream is the sequence of token representations that flows through the model from input to output. Each layer adds a small delta to it rather than rewriting it from scratch. This makes the gradient path short and the training stable.

Accumulation through layers

At each layer, two additions occur:

h_0      = embed(tokens)                    # [T, 896]
h_attn_1 = h_0    + attention_delta(h_0)   # layer 1 attn
h_mlp_1  = h_attn_1 + mlp_delta(h_attn_1) # layer 1 MLP
h_attn_2 = h_mlp_1 + attention_delta(h_mlp_1)
...
h_final  = model_norm(h_mlp_24)
logits   = h_final @ embed_T               # [T, vocab_size]

The stream at each step has shape [T, 896] throughout — the residual connection preserves dimensionality.

Measuring contributions

TokenPrint's Debugger mode measures the PCA-space residual delta per layer — how much the stream changes at each layer in projected space. This tells you which layers are doing the most work.

The measurement is the total residual change in PCA space, not a decomposition into attention vs. MLP contributions. The label in the UI states this explicitly.

Logit lens

The logit lens projects the residual stream at each intermediate layer through the final unembedding matrix to see what token the model would predict if it stopped there. TokenPrint computes this for every layer:

logit_lens[layer][position] = softmax( h_layer[position] @ embed_T )

The result is a matrix of probability distributions over the 151,936-token vocabulary, one per (layer, position) pair. This data drives the Softmax & Output chapter of the Walkthrough.