AI Infrastructure Sizing: GPU, Memory & Storage for LLM Workloads (2026)

Q: How much GPU memory do I need for a 70B parameter LLM?

At BF16 precision (the standard for serving in 2026), a 70-billion parameter model requires approximately 140 GB of GPU memory for the model weights alone. You also need to account for the KV cache, which at 32 concurrent requests with 32K context can add another 40–80 GB. A single H200 (141 GB) can serve the model at BF16 with moderate concurrency. At INT4 quantisation, the same model fits in approximately 35 GB, making it servable on a single H100 or even an L40S, but with some quality trade-offs depending on the use case.

Q: What is the difference between HBM3 and HBM3e for AI workloads?

HBM3 provides memory bandwidth of approximately 3.35 TB/s (as used in the H100), while HBM3e provides 4.8 to 8.0 TB/s (as used in H200 and B200). Since LLM inference is memory-bandwidth-bound during autoregressive decoding, this bandwidth difference directly translates to faster token generation. An H200 with HBM3e generates tokens roughly 1.4x faster than an H100 with HBM3 for the same model, independent of any compute improvements. For latency-sensitive applications like conversational AI, HBM3e delivers a meaningful user experience improvement.

Q: How much does it cost to run LLM inference in production?

Production LLM inference costs vary dramatically based on model size, concurrency, and GPU selection. A RAG-powered internal assistant using a quantised 13B model on 2x L40S GPUs costs approximately £2,500 to £4,000 per month. A customer-facing deployment using a 70B model on 4x H200 GPUs costs £15,000 to £25,000 per month. Batch document processing on mixed GPU tiers costs £5,000 to £8,000 per month. The most impactful cost optimisation strategies are right-sizing quantisation to your quality bar, using spot instances for training and batch work, and implementing autoscaling for inference.

Q: Should I use NVIDIA or AMD GPUs for LLM workloads?

NVIDIA remains the default choice for enterprise AI in 2026 due to superior software ecosystem maturity — CUDA, TensorRT-LLM, and vLLM all have first-class NVIDIA support. AMD MI300X is a compelling alternative for teams with engineering depth, offering 192 GB of HBM3 at 15 to 25 percent lower cost than equivalent NVIDIA hardware. The ROCm software stack has matured considerably, with native support in PyTorch and vLLM. Choose AMD if you have the engineering capacity to handle occasional compatibility issues and want to reduce hardware costs. Choose NVIDIA if you want the most mature ecosystem with the broadest library support.

Q: How should I plan GPU capacity for growth?

Plan for 3x your current load within 12 months. Enterprise AI adoption consistently grows faster than teams expect once initial use cases prove successful. Design infrastructure for horizontal scaling from day one using model replicas behind a load balancer and stateless serving containers. Budget GPU memory with 30 to 50 percent headroom above current model requirements to accommodate model upgrades without infrastructure changes. Account for multi-model serving — a typical production deployment serves 3 to 8 models simultaneously. Conduct monthly infrastructure reviews that right-size based on actual utilisation data, as most organisations over-provision by 2 to 3x in their first year.

Q: What storage do I need for AI and LLM workloads?

AI storage operates across three tiers. Tier 1 is NVMe SSD for model weights and checkpoints — plan for 500 GB to 2 TB per GPU node. Tier 2 is high-throughput SSD or distributed filesystem (Lustre, GPFS, WekaFS) for training datasets, requiring at least 1 GB/s sequential read throughput per GPU. Tier 3 is object storage (S3, GCS, Azure Blob) for logs, artefacts, and long-term storage — budget 5 to 20 TB per AI project, growing at approximately 500 GB per month. Cold-start performance depends on Tier 1 speed: a 140 GB model loads in 20 seconds from a Gen5 NVMe drive versus 47 seconds from a standard drive.

Satyam Kumar

Back to Blog

ai-engineering

AI Infrastructure Sizing: GPU, Memory, and Storage for LLM Workloads (2026)

By Satyam KumarApril 13, 202615 min read

ai infrastructure sizing gpu for llm llm infrastructure requirements ai hardware guide h100 vs h200 gpu memory llm ai infrastructure cost llm gpu requirements 2026 nvidia h100 ai ai workload sizing enterprise ai infrastructure gpu selection guide

AI Infrastructure Sizing: GPU, Memory, and Storage for LLM Workloads (2026)

Frequently Asked Questions

Share this article

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

AI & Cloud Architect. Helping teams build systems that scale to millions.

LinkedIn GitHub

AI Infrastructure Sizing: GPU, Memory, and Storage for LLM Workloads (2026)

Frequently Asked Questions

Share this article

Comments

Leave a comment