Tokens
A token is the atomic unit of text that a language model processes. Before any computation, the raw input string is converted into a sequence of integer IDs by a tokenizer. The model never sees characters — only these IDs.
Byte-pair encoding
Qwen2.5 uses a byte-pair encoding (BPE) tokenizer with a vocabulary of 151,936 tokens. BPE builds the vocabulary by iteratively merging the most frequent adjacent byte pairs in the training corpus. Common words become single tokens; rare words are split into subword pieces.
Example — the sentence “The cat sat on the mat.”:
| Token text | Vocabulary ID |
|---|---|
The | #785 |
cat | #9982 |
sat | #14524 |
on | #389 |
the | #279 |
mat | #4264 |
. | #13 |
These are real vocabulary IDs from the Qwen2.5 tokenizer. Common words like the (ID 279) and . (ID 13) have low IDs because they appeared early in the BPE merge sequence.
TokenPrint visualization
The Tokenization chapter in Walkthrough mode renders each token as a colored chip showing both the token text and its vocabulary ID. The chip color encodes the token's position in the sequence.
The language selector lets you switch between tokenizations of the same sentence in different languages — the number of tokens changes even for semantically equivalent text, which demonstrates how tokenizer vocabulary coverage affects efficiency.
Token IDs are read from a real tokenizer call on the backend. They are not estimated or approximated.
Special tokens
The Qwen2.5 tokenizer uses several special tokens that are not natural text:
| Token | Purpose |
|---|---|
<|im_start|> | Marks the start of a chat turn |
<|im_end|> | Marks the end of a chat turn |
<|endoftext|> | End-of-sequence marker |
These appear in the token stream during Generation mode and are visible in the top-k skyline panel when the model predicts end-of-generation.