What is High Bandwidth Memory (HBM)?

High Bandwidth Memory (HBM) is a class of DRAM engineered for extreme bandwidth by stacking multiple memory dies vertically and connecting them to the processor through a very wide interface rather than a fast-but-narrow one. A stack is typically 4–12 thinned DRAM dies sitting on a base logic die, electrically joined by through-silicon vias (TSVs) — copper-filled vertical channels running through dies thinned to tens of micrometers. The JEDEC HBM standard fixes a 1024-bit-wide interface per stack (rising to 2048 bits in HBM4), split into independent channels so the wide bus is actually usable in parallel rather than bottlenecked on a shared command path (JEDEC; Rambus; Tom’s Hardware).

How the width delivers bandwidth. Per-stack bandwidth is simply the interface width times the per-pin data rate: HBM3 at 6.4 Gbps per pin over 1024 bits gives about 819 GB/s per stack, and HBM3E at roughly 9–10 Gbps per pin reaches about 1.2 TB/s (Rambus). The original unit’s claim that HBM “trades clock speed for width” holds up in substance: bandwidth comes from running thousands of connections in parallel at per-pin rates below those of narrow, high-clocked graphics DRAM. The nuance verified this run is that this is a relative statement, not an absolute one — modern HBM pins are themselves fast (HBM3E runs around 9–10 Gbps per pin), but contemporary GDDR pushes individual pins harder still, while an HBM stack’s 1024-bit interface is many times wider than the narrow per-device buses of conventional graphics memory (JEDEC; Rambus). Short distance is what makes the wide bus feasible: signals travel micrometers vertically through TSVs and millimeters across an interposer rather than centimeters across a printed circuit board, cutting loss and per-bit energy. Accelerators mount several stacks side by side, so aggregate device bandwidth is the per-stack figure multiplied across stacks (JEDEC; Rambus).

Why this matters for AI inference. The decode phase of large-model inference — generating tokens one at a time — is memory-bandwidth-bound, not compute-bound. Each step reads the model’s frozen weights (and a growing KV cache) from memory but performs on the order of one arithmetic operation or less per byte moved — single-request decode measures roughly 0.25–1 FLOPs per byte — an arithmetic intensity far below the point where a modern accelerator’s compute would become the limit; on the roofline model, decode sits deep in the memory-bound region (arXiv 2402.16363). Consequently the memory bus, not the compute units, sets the token-per-second ceiling, and successive HBM generations raise that ceiling almost directly. Capacity and bandwidth play distinct roles: capacity bounds how large a model and context (KV cache) can fit, while bandwidth bounds throughput. Shipping accelerators make the scale concrete — NVIDIA’s H200 carries 141 GB of HBM3E at 4.8 TB/s, and AMD’s MI300X carries 192 GB of HBM3 at about 5.3 TB/s (NVIDIA; AMD). The bandwidth-bound decode dynamics and the KV cache each warrant their own dedicated treatment beyond this overview.

How it is made, and why supply is tight. HBM is among the hardest memory to manufacture: TSV etch and copper fill, then bonding a dozen-plus dies with micrometer alignment while controlling warpage, are the production bottlenecks, and stacking yield compounds multiplicatively with every added layer (Lam Research; SemiAnalysis). The finished stacks are then co-packaged beside the compute die on a silicon interposer — 2.5D integration such as TSMC’s CoWoS — so HBM’s availability is gated not only by DRAM fabs but by advanced-packaging capacity, which has been a binding constraint on accelerator shipments (SemiAnalysis). HBM4 keeps proven microbump/thermocompression bonding rather than moving to hybrid bonding, deferring that transition to a later generation (SemiEngineering). This packaging dependency is the hook into the market side of the knowledge base, where HBM’s premium pricing and supply-gated character are tracked as analysis.

Confidence 0.78: this run replaced a single general source with multi-source, partly primary evidence (JEDEC, Rambus, an arXiv roofline survey, vendor spec pages) and verified the core mechanism claim. It is held below the high 0.9s because several figures come from mid-tier trade sources, some HBM4 numbers are forward-looking, and the deeper inference and packaging treatments are still to be written.