HBM Is the Bottleneck, Not FLOPs: Inference Bandwidth

Q: Why is my expensive GPU nearly idle when the model generates tokens?

Because autoregressive token generation — the decode phase — is memory-bandwidth-bound, not compute-bound. To produce a single output token the GPU must read the entire set of model weights out of high-bandwidth memory (HBM), perform a comparatively tiny matrix-vector multiplication, and then repeat the whole read for the next token. The ratio of arithmetic to bytes moved is roughly one floating-point operation per byte at batch size 1, whereas a chip like the NVIDIA H100 needs on the order of 300 FLOPs of work for every byte it fetches before its tensor cores become the limiting factor. That mismatch means the cores spend most of their time waiting for memory, so the GPU runs at a small single-digit percentage of its rated peak throughput. It is not broken and it is not misconfigured in the usual sense — it is doing exactly what a memory-bound workload does on hardware sized for compute. The fix is never "more FLOPs"; it is to reduce the bytes read per token (quantise the weights, use mixture-of-experts, keep the KV cache small) or to amortise each weight read across many requests with continuous batching, which is why batch size 1 in production is the most common cause of a shockingly low utilisation number.

Q: What is the roofline model and why does it apply to LLM serving?

The roofline model is a performance framework that plots achievable throughput against arithmetic intensity, defined as the number of floating-point operations performed per byte of memory traffic. It has two regimes separated by a hardware-specific ridge point. To the left of the ridge, arithmetic intensity is low and performance is capped by memory bandwidth — you are bandwidth-bound, and the ceiling is a flat line equal to bandwidth divided by bytes-moved-per-unit-work. To the right, intensity is high and performance is capped by the compute units — you are compute-bound at peak FLOPs. The ridge point itself is simply peak FLOPs divided by peak bandwidth; for an H100 SXM at roughly 990 dense BF16 TFLOPs over 3.35 TB/s it sits near 300 FLOP per byte. LLM serving applies cleanly because the two phases of a request fall on opposite sides: prefill processes the whole prompt as a large matrix-matrix multiply with high intensity and is compute-bound, while decode emits one token at a time as a matrix-vector operation at about 1 FLOP per byte and is deep in the bandwidth-bound region. Because decode dominates wall-clock time for chat workloads, the roofline tells you the serving system lives on the memory-bound roof — which is why the correct hardware axis to optimise is bandwidth, not peak compute.

Q: Why does the NVIDIA H200 serve more tokens than the H100 if it has the same compute?

Because the H200 and H100 share the same compute die and therefore the same peak FLOPs, but the H200 has a substantially larger and faster memory subsystem — 141 GB of HBM3e at 4.8 TB/s versus the H100 SXM's 80 GB of HBM3 at 3.35 TB/s, roughly a 43 percent bandwidth increase. Since decode throughput is governed by tokens per second ≈ memory bandwidth divided by bytes read per token, and the bytes-per-token are identical for the same model, more bandwidth translates almost directly into more decode tokens per second. The extra compute that would help a compute-bound workload does not exist here, and it would not help decode even if it did, because decode never approaches the compute roof. This is the cleanest real-world demonstration that production LLM serving is bandwidth-bound: hold FLOPs constant, raise bandwidth, and throughput rises. It also reframes the purchasing decision — the H200 is not a "faster compute" upgrade, it is a "wider memory pipe" upgrade, and the larger 141 GB capacity independently helps by letting bigger models fit on fewer devices, removing interconnect hops from the decode path. If you were comparing the two chips on a peak-TFLOP spec sheet you would see no difference and conclude, wrongly, that there was nothing to gain.

Q: Should I buy accelerators based on TFLOPs or memory bandwidth?

For inference serving that is dominated by token generation, you should shop primarily on memory bandwidth (TB/s) and memory capacity (GB), and treat peak TFLOPs as a secondary figure relevant mainly to the prefill phase and to training. The reason is structural: decode is memory-bound, so its throughput ceiling is set by how fast the accelerator can stream weights out of HBM, not by how many tensor-core operations it can theoretically perform. Two accelerators with identical peak FLOPs but different bandwidth will deliver very different decode throughput, and the higher-FLOP part is frequently the wrong buy if its bandwidth is not correspondingly higher. Capacity matters as a second axis because it determines topology — a 192 GB card can hold a model that would otherwise be sharded across two or four smaller GPUs, and every GPU boundary you remove deletes a cross-device interconnect hop from the latency-critical decode loop. The practical rule: size the decode pool on TB/s and GB, size a separate prefill pool on FLOPs if you disaggregate the phases, and be deeply suspicious of any recommendation that leads with the headline FP4 or FP8 number, because that number describes the compute roof your decode workload never reaches. Peak FLOPs is a training-and-prefill metric wearing a serving-metric costume.

Q: What is HBM4 and why does the 2025 standard matter?

HBM4 is the fourth generation of the High Bandwidth Memory standard, and JEDEC published it as JESD270-4 on 16 April 2025. The headline change is that HBM4 doubles the memory interface width from the 1,024 bits of prior generations to 2,048 bits per stack, and pushes total bandwidth past 2 terabytes per second per stack, with advanced configurations reaching higher still. It also raises the number of independent channels per stack from 16 to 32 and supports taller die stacks — up to 16-high — with denser dies, enabling substantially larger per-stack capacity. Why it matters for architecture is the same reason the rest of this topic matters: HBM4 is fundamentally a bandwidth-and-capacity release, not a compute release. The memory vendors and JEDEC are investing the standards effort in the exact axis that bounds inference, which is a strong signal that the industry has internalised memory bandwidth as the binding constraint on serving large models. For anyone planning a multi-year fleet, HBM4-class accelerators represent the next step change in the term that actually caps decode throughput — bytes streamed per token — rather than in the peak-FLOP number that decode does not touch. It is the clearest institutional confirmation that "buy bandwidth" is not a temporary quirk of one GPU generation but the durable shape of the workload.

Q: How do continuous batching and quantisation raise the token throughput ceiling?

They attack the two terms in the decode throughput equation, tokens per second ≈ memory bandwidth divided by bytes read per token. Continuous batching attacks the effective utilisation of bandwidth: the weight read required for one decode step is a fixed cost, so if you serve 32 requests from the same weight fetch you have amortised a single expensive memory read across 32 output tokens, which pushes the kernel from the bandwidth-bound region toward the compute-bound region and dramatically raises tokens per second per GPU. It is a scheduler feature rather than a hardware feature, which is why modern serving stacks like vLLM exist and why running batch size 1 "for latency" leaves most of your throughput on the table. Quantisation attacks the bytes-per-token term directly: moving weights from FP16 to FP8 halves the bytes read per token, and FP4 — native on Blackwell — halves it again, so the same physical bandwidth now supports a proportionally higher token ceiling. Crucially, quantisation is a memory-traffic optimisation that also happens to save capacity, not primarily a compute trick. Combined, batching and quantisation are the highest-leverage moves available, and neither one requires or benefits from buying an accelerator with more peak FLOPs; both are ways of getting more useful tokens out of the memory bandwidth you already paid for.

Q: Why does the KV cache matter for memory bandwidth at long context?

The KV cache stores the key and value tensors for every token already in the context so the model does not recompute them on each decode step, and at long context lengths the bytes moved to read that cache can rival or exceed the bytes moved to read the model weights. Since decode is bandwidth-bound, every additional byte the KV cache adds to the per-token memory traffic directly lowers the token throughput ceiling, by the same tokens-per-second ≈ bandwidth ÷ bytes-per-token relationship that governs the weights. This is why teams frequently quantise their weights, celebrate the throughput gain, and then watch a new 128,000-token context feature quietly re-cap throughput because the KV cache is now the dominant term in the byte budget. The mitigations are the KV-cache analogues of the weight optimisations: paged attention to avoid fragmentation and wasted reads, KV quantisation to shrink bytes per cached token, and tiered offloading of cold cache entries to slower memory so they do not consume premium HBM bandwidth until needed. The general point is that a rigorous inference sizing exercise budgets bytes-moved-per-token as weights plus KV cache together, because optimising only the weights while ignoring a growing KV cache produces sizing estimates that are correct at short context and badly wrong in production at long context.

Q: Does capacity or bandwidth matter more when choosing an inference GPU?

They matter for different things and you must evaluate both, but they are not interchangeable and collapsing them into a single "big memory" impression is a common sizing error. Capacity, measured in gigabytes, determines what you can load and therefore your topology: a 192 GB accelerator such as the AMD MI300X or NVIDIA B200 can hold a large model on a single device that an 80 GB H100 would force you to shard across several, and each GPU boundary you eliminate removes an interconnect hop from the decode path, improving latency and simplifying the serving stack. Bandwidth, measured in terabytes per second, determines how fast you can serve whatever fits: it sets the decode throughput ceiling directly, because decode streams the weights from HBM once per token. The failure mode is choosing a card with generous capacity but starved bandwidth and expecting high throughput, or the reverse. The disciplined approach is to first ensure capacity is sufficient to hold the model with the topology you want — ideally minimising cross-GPU hops — and then, among cards that clear that bar, to select on bandwidth because that is the term that caps tokens per second. In short: capacity is a gate that decides feasibility and topology, bandwidth is the dial that decides serving throughput, and a serious evaluation checks the TB/s figure explicitly rather than inferring performance from the GB figure.

Satyam Kumar

Back to Blog

ai-architecture-patterns

HBM Is the Bottleneck, Not FLOPs: Designing Inference Around Memory Bandwidth

By Satyam KumarAugust 22, 202610 min read

hbm memory bandwidth llm inference roofline model gpu architecture nvidia h200 nvidia blackwell b200 amd mi300x hbm4 inference optimization kv cache continuous batching quantization ai infrastructure accelerator selection 2026

HBM Is the Bottleneck, Not FLOPs: Designing Inference Around Memory Bandwidth

Frequently Asked Questions

Share this article

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

AI & Cloud Architect. Helping teams build systems that scale to millions.

LinkedIn GitHub

HBM Is the Bottleneck, Not FLOPs: Designing Inference Around Memory Bandwidth

Frequently Asked Questions

Share this article

Comments

Leave a comment