The KV cache: what it is and why it dominates inference memory

The KV cache is the store of already-computed key and value tensors that a transformer keeps around during autoregressive generation so it does not have to recompute them for every new token. It is the single largest variable consumer of accelerator memory during inference, and it is the reason the token-by-token decode phase is bound by memory bandwidth rather than by compute.

What is actually cached, and why. In self-attention, every token’s embedding is projected by three learned weight matrices into a query (Q), a key (K), and a value (V) vector. Attention mixes them as softmax(Q·Kᵀ/√d)·V: the query of the current position is scored against the keys of all positions, and those scores weight the values. The key insight for caching is that K and V for a given token depend only on that token and the fixed weights, so once computed they never change — whereas each position’s query is used exactly once, to produce that position’s output, and is then discarded (NVIDIA; researcher synthesis). During generation the model therefore stores every token’s K and V and reuses them at every later step, recomputing only the new token’s Q, K, and V. Without this, generating token n would re-attend over all n−1 prior tokens from scratch, making total generation cost grow with the square of the sequence length; caching reduces the per-step attention work to linear in the sequence length.

Prefill vs. decode. Inference has two regimes. Prefill processes the whole prompt at once as a matrix–matrix multiply, computing and storing the K/V for every prompt token in one parallel pass — this phase is compute-bound. Decode then emits one token per forward pass: it reads the entire cached K/V (and the model weights), appends the new token’s K/V, and produces one token. Decode is a sequence of matrix–vector multiplies and is memory-bound (jax-ml scaling book; NVIDIA).

How big it gets. The cache size in bytes is:

2 × num_layers × num_kv_heads × head_dim × seq_len × batch_size × bytes_per_element

The leading 2 is for storing K and V separately. It scales linearly in each factor — double the context, the batch, or the depth and you double the cache. num_kv_heads is the number of key/value heads, which under grouped-query attention is smaller than the number of query heads. Precision sets bytes_per_element: 2 for FP16/BF16, 1 for FP8, 4 for FP32.

Worked example (Llama-3-70B-class model, BF16), using binary units (1 KiB = 1024 B) throughout: 80 layers, 8 KV heads, head_dim 128, 2 bytes. Per token that is 2 × 80 × 8 × 128 × 2 = 327,680 bytes = exactly 320 KiB per token per sequence. An 8,192-token (8K) request then holds 320 KiB × 8,192 = 2.5 GiB of KV cache; 32 such concurrent requests need about 80 GiB — enough to consume an entire 80 GB accelerator on its own, before the model weights are even counted (researcher synthesis from NVIDIA/community calculations). Long contexts push this further: a single 128K-token request against the same model is on the order of tens of GiB of cache.

Why it makes decode memory-bandwidth-bound. A single-token decode step is a matrix–vector operation with an arithmetic intensity of roughly one FLOP per byte moved — far below the “ridge point” where an accelerator becomes compute-limited (on the order of 150–300 FLOPs/byte for recent datacenter GPUs). So the memory bus, not the math units, sets the speed: each token requires streaming the model weights plus the growing KV cache out of memory, and GPU compute utilization during decode can sit in the single-digit percent range (arXiv 2402.16363; jax-ml scaling book). This is exactly the memory-bandwidth-bound decode regime established for HBM: the KV cache is a primary, sequence-length-dependent consumer of that bandwidth, and its capacity is what HBM’s capacity must accommodate above the weight floor.

When the KV cache actually dominates — a regime question. It is tempting to say either “weights dominate memory” or “the KV cache dominates”; the honest answer is that it depends on the operating point. Model weights are a fixed cost (a 70B model in FP16 is ~140 GB regardless of load), while the KV cache grows with context length and with the number of concurrent requests. At short context and small batch, weights dominate the footprint and the per-token bandwidth bill. At long context and/or high concurrency the KV cache rivals or exceeds the weight footprint and comes to dominate both capacity pressure and decode-time bandwidth (researcher synthesis; jax-ml scaling book). Increasing batch size raises arithmetic intensity for the weight-bound matmuls (weights are amortized across the batch) but not for attention, because every sequence carries its own KV cache — so larger batches tend to stay memory-bound on the attention path even as the feed-forward path approaches compute-bound. This regime-dependence, not a single verdict, is the correct mental model.

Reducing footprint and bandwidth — and the trade-offs. Several techniques attack the cache, each with a real cost:

• Multi-query (MQA) and grouped-query attention (GQA). Share K/V heads across query heads. MQA uses a single K/V head — up to an h× reduction — but the aggressive sharing costs model quality and can destabilize training. GQA is the practical middle ground: query heads are split into groups that each share one K/V head, giving a num_query_heads / num_kv_heads reduction (e.g. 8× for a 64-query/8-KV configuration such as Llama-3-70B) with little quality loss, and existing multi-head checkpoints can be up-trained into GQA for a small fraction of pretraining compute (GQA paper; Raschka). This is now the industry default.
• KV-cache quantization. Storing K/V at lower precision cuts the cache roughly in proportion to the bit reduction: FP8 or INT8 give about 2× with near-lossless quality in most reported settings, and INT4 gives about 4× with a small accuracy penalty, while very low bit-widths degrade sharply (researcher synthesis). These figures come largely from vendor and community reports and are held with corresponding caution.
• Paged KV memory management (PagedAttention / vLLM). Allocating the cache in fixed-size blocks on demand, rather than one contiguous per-request buffer, nearly eliminates the internal fragmentation that otherwise wastes a large fraction of reserved cache memory, letting far more requests share a GPU and delivering multi-fold throughput gains. The cost is per-operation latency overhead from block-table indirection, so it is a throughput optimization whose benefit shows up under concurrency rather than at batch size one (vLLM paper; researcher synthesis).
• Speculative decoding. A small draft model proposes several tokens that the large model verifies in one pass, amortizing the KV/weight read across multiple tokens. Its benefit is entirely contingent on the draft-acceptance rate — under low acceptance (e.g. high-temperature generation) it can add cost rather than save it.
• Attention sparsity. Attending to only a salient subset of cached tokens can shrink the effective cache dramatically at long context, but the important tokens are query- and context-dependent, so static pruning underperforms and dynamic selection adds its own complexity.

A tension worth flagging, not papering over. These techniques are not freely composable. In particular, some community reports find that KV-cache quantization can reduce the gains from speculative decoding: quantizing K/V shifts the target model’s logit distribution, which lowers the draft-token acceptance rate and erodes the very bandwidth amortization speculative decoding relies on. So “quantize the cache” and “speculate to amortize bandwidth” are not guaranteed to add up, and may partly cancel. This rests on thinner sourcing than the rest of this unit and is stated as a caution rather than a settled figure.

Confidence 0.6: this is a fresh bootstrap foundation unit. Its mechanical core (what K/V are, why they are cacheable, the size formula and its linear scaling, and the memory-bandwidth-bound nature of decode) is well corroborated across strong sources and is internally checkable arithmetic. It is held in the low-mid range because most quantitative figures for reduction techniques and the regime-crossover points come from secondary/community and vendor material that this pipeline did not independently verify, and because the quantization× speculation tension is drawn from a single weaker source. Confidence should rise as downstream units (memory tiering/offloading, quantization) are written and these numbers are pinned to primary sources.