Skip to content

LLM VRAM & GPU Sizing Calculator

How much GPU memory does it take to serve a large language model, and how many H100s, H200s, or A100s do you actually need? Serving VRAM is not one number — it is model weights + KV cache + overhead, and the KV cache grows with both your context length and your concurrency. Set your model, precision, context, and batch below and the calculator returns the VRAM breakdown and the GPU count in real time. It is an estimate for capacity planning, updated for 2026 hardware.

LLM Serving VRAM & GPU Sizing Calculator

Model weights
141 GB
70.6B @ FP16
KV cache (total)
42.9 GB
2.7 GB/req × 16
Total VRAM needed
214 GB
incl. ~29.6 GB overhead
NVIDIA H100 needed
3 GPUs
89% of 240 GB

On 3× NVIDIA H100 this configuration fits about 28 concurrent 8,192-token requests before VRAM is exhausted. The weights alone (141 GB) exceed one NVIDIA H100, so you need tensor parallelism across GPUs regardless of batch size.

Estimate only. Real usage varies with the serving framework (paged attention reduces KV waste), activation memory, CUDA-graph capture, speculative decoding, and MoE routing. Throughput (tokens/sec) is a separate calculation from VRAM.

The formula the calculator uses

Three terms decide serving VRAM, and getting the second one wrong is the usual planning mistake:

  • Weights = parameters × bytes-per-parameter. FP16 is 2 bytes, FP8/INT8 is 1, INT4 is 0.5. A 70B model is ~141 GB in FP16, ~35 GB in INT4.
  • KV cache = 2 × layers × KV-heads × head-dim × bytes, per token, then × context length × concurrent requests. This is the term that explodes at long context and high batch — often larger than the weights.
  • Overhead = activations, CUDA graphs, and framework working memory — roughly 15% plus a couple of gigabytes.

When the weights alone exceed one GPU’s VRAM, you need tensor parallelism across cards no matter how small the batch. When weights fit but the KV cache for your target concurrency does not, you add GPUs for memory, not compute — which is why inference is memory-bound, not FLOP-bound.

How to reduce the VRAM (and the GPU bill)

  • Quantize the weights (FP8/INT4) — 2–4× smaller weights, with an accuracy cost you must evaluate.
  • Quantize the KV cache (FP8) — halves the term that dominates at high concurrency.
  • Use paged attention (vLLM-style) — removes KV fragmentation so you can pack more concurrent requests into the same VRAM.
  • Cap context where the product allows — KV cache is linear in context length.
  • Right-size concurrency — and read GPU utilization and the idle fleet before over-provisioning.

Frequently asked questions

How much VRAM do I need to serve an LLM?

Total serving VRAM is model weights plus KV cache plus framework overhead. Weights are the parameter count times bytes per parameter (2 for FP16, 1 for FP8/INT8, 0.5 for INT4). The KV cache is 2 x layers x KV-heads x head-dim x bytes per token, multiplied by your context length and by the number of concurrent requests. Overhead for activations and the CUDA/serving framework typically adds 15 percent plus a couple of gigabytes. A 70B model in FP16 needs about 141 GB for weights alone, so it does not fit on a single 80 GB H100 and requires tensor parallelism across at least two GPUs before you add any KV cache.

What is the KV cache and why does it dominate at long context?

The KV cache stores the key and value tensors for every token already processed, so the model does not recompute them each step. Its size grows linearly with context length and with the number of concurrent requests, which is why memory, not compute, is usually the binding constraint for long-context or high-concurrency serving. For a large model at 128k context with a big batch, the KV cache can exceed the size of the weights themselves. Grouped-query attention (GQA), which shares KV heads across query heads, is the main architectural lever that keeps this cache affordable.

Does quantization reduce the GPU count I need?

Yes, for the weights. Moving from FP16 to INT4 cuts weight memory by 4x, so a 70B model drops from about 141 GB to about 35 GB of weights, which can change a two-GPU deployment into a single-GPU one. But quantizing weights does not shrink the KV cache unless you also quantize the cache itself (FP8 KV cache halves it). At high concurrency the KV cache, not the weights, sets the GPU count, so quantization helps less than people expect. Quantization also has an accuracy cost that varies by model and task and must be evaluated, not assumed.

When do I need more than one GPU for an LLM?

You need multiple GPUs in two situations. First, when the weights alone exceed a single GPU’s VRAM — a 70B FP16 model at 141 GB cannot fit on an 80 GB card, so tensor parallelism is mandatory regardless of batch size. Second, when weights fit but the KV cache for your target concurrency and context pushes total VRAM past one card. The calculator distinguishes these: it flags when weights alone force multi-GPU, and it reports how many concurrent requests a given fleet can hold before VRAM is exhausted.

How accurate is this calculator?

It is a first-order estimate suitable for capacity planning and GPU-count decisions, using the standard weights-plus-KV-cache-plus-overhead model with real architecture numbers for each preset. Actual VRAM depends on the serving framework — paged attention (vLLM) reduces KV fragmentation and waste, CUDA-graph capture and speculative decoding add memory, and mixture-of-experts models route differently — so treat the output as a planning baseline and validate with a load test before committing hardware. Throughput in tokens per second is a separate calculation driven by memory bandwidth, not covered here.

Does this work for how to size storage for a 40,000-GPU AI data center?

This tool sizes the VRAM and GPU count for serving a single model configuration, which is the per-node building block of a large fleet. Data-center-scale sizing then multiplies by the number of model replicas you run for throughput and availability, and adds the separate dimensions of interconnect bandwidth, power and cooling density, and storage for model weights, checkpoints, and datasets. For the fleet-level architecture — power, cooling, networking, and storage tiers behind tens of thousands of GPUs — see the AI infrastructure sizing and data-center capacity articles linked below.

What is tensor parallelism and how does it change the math?

Tensor parallelism splits each layer’s weight matrices across multiple GPUs so a model too large for one card can run, with the GPUs exchanging activations over high-speed interconnect (NVLink). The total VRAM requirement is roughly unchanged — you still need enough aggregate memory for weights plus KV cache plus overhead — but it is now spread across the fleet, and interconnect bandwidth becomes a performance factor. The calculator’s GPU count is the aggregate memory answer; whether you realize it as tensor parallelism, pipeline parallelism, or replicas depends on your latency and throughput goals.

Why is memory the bottleneck for inference instead of compute (FLOPs)?

Autoregressive decoding generates one token at a time, reading the entire model’s weights and the full KV cache from memory for every single token. That makes decoding memory-bandwidth-bound: the accelerator’s FLOPs sit largely idle while it waits on memory. This is why VRAM capacity sets what you can serve and memory bandwidth sets how fast, while raw FLOPs rarely bind. It is also why accelerator choice is primarily a memory decision — see the HBM bandwidth article linked below.