Decoding Strategies

Core Concepts

Decoding is the strategy used to turn the model's predicted token probabilities into the next token. Every model returns a probability distribution over the vocabulary; the decoding strategy decides which token to actually emit.

Greedy decoding

Greedy decoding always picks the token with the highest probability. It is deterministic — the same prompt always produces the same output. The Generation mode uses greedy decoding by default, and the Browser GPT-2 engine supports greedy only.

Sampling

Sampling draws the next token randomly from the probability distribution. This produces more diverse and creative text than greedy, which can fall into repetitive loops. Options expose a random seed so runs are reproducible: the same seed reproduces the same token sequence.

Temperature

Temperature scales the logits before softmax. Logits are divided by the temperature value:

softmax(logits / T)

• T = 1 — the raw distribution, no change.
• T < 1 — sharpens the distribution, making high-probability tokens more likely.
• T > 1 — flattens the distribution, making low-probability tokens more likely.

Lower temperatures approach greedy behavior; higher temperatures approach uniform sampling.

Top-k and top-p

Top-k and top-p are truncation filters applied before selecting a token:

• Top-k restricts the choice to the k most probable tokens. The probabilities are renormalized over that subset. In TokenPrint the top-k panel shows the filtered candidates live during generation.
• Top-p (nucleus) keeps the smallest set of tokens whose cumulative probability reaches p. The cutoff adapts to the shape of the distribution.

Both are often combined with sampling — decode from the top-k (or top-p) set, sampled at a chosen temperature.

Sliding window

Sliding window decoding trims the KV cache to the last windowSize positions at each decode step. The model genuinely recomputes with a reduced visual context — the older positions fall out of the attention window. This bounds memory for long sequences at the cost of losing older context.

Speculative decoding

Speculative decoding is a self-speculative “blockwise” strategy: the model drafts draftGamma candidate tokens cheaply from its last distribution, then verifies all of them in a single batched forward pass, accepting the longest matching prefix. It produces the same result as greedy but with fewer forward passes, at the cost of a larger peak batch.

In TokenPrint