Embeddings
Before any transformer layer runs, each token ID is converted into a dense vector of real numbers by a lookup into the embedding matrix. This vector is the token's initial representation in the model's hidden space.
The embedding matrix
The embedding matrix has shape [151936, 896]vocab_size × hidden_size and contains 136,134,656 parameters — 27.5% of the total model parameter count, making it the largest single tensor.
In Qwen2.5, the embedding matrix is tied with the output unembedding matrix: the same weights are used to convert token IDs to vectors at the input and to convert hidden states back to logits at the output.
Hidden size
Every internal representation in the model is a vector of dimension 896 (the hidden size, also called d_model). After the embedding lookup, the sequence has shape [T, 896]seq_len × hidden_size where T is the number of tokens.
PCA projection in TokenPrint
The Embedding chapter in Walkthrough mode projects the 896-dimensional token embeddings into 3D using PCA (principal component analysis). The backend runs the PCA decomposition using real PyTorch tensor values and returns the 3D coordinates.
The projection is deterministic — the same input always produces the same 3D coordinates, verified by the geometry verification script. Semantically similar tokens cluster together in the PCA space: for example, apple and orange land almost on top of each other at the embedding layer.
Position encoding
Qwen2.5 does not use learned positional embeddings added to the token embeddings (as in the original transformer). Instead, it uses Rotary Position Encoding (RoPE), which is applied inside each attention layer to the Q and K projections. The embedding lookup itself is position- independent.