KV Cache

Core Concepts

During autoregressive generation, recomputing the Key and Value activations for all past tokens at each new step would be quadratic in the sequence length. The KV cache avoids this by storing the K and V activations from previous steps and reusing them.

Prefill vs. decode phases

TokenPrint's generation trace distinguishes two phases:

Phase	What is computed	Cache state	Example (39-token prompt)
`prefill`	All prompt tokens in one parallel pass	Built from scratch	39 positions computed, `cache_len = 0`
`decode`	One new token per step	Previous KV pairs read from cache	1 position computed, `cache_len = 39`

These are real phases from the actual PyTorch forward pass, not a simulation. The backend threads real past_key_values through the model loop.

Cache size

For each layer, the cache stores K and V tensors for the 2 KV heads at each past position. At position n, cache size per layer is:

cache_per_layer = n_kv_heads * head_dim * n_past_tokens * dtype_bytes
                = 2 * 64 * n * 4       (float32)
                = 512 * n bytes

For 24 layers and a 32,768-token context: approximately 384 MB at float32.

TokenPrint visualization

In Generation mode, the KV cache is rendered as a spatial occupancy structure in the 3D scene. Filled cells indicate cached positions; the boundary between prefill and decode is visible as the cache grows token by token. The transport bar shows the current cache_len and n_positions from the real trace metadata.