FP4 Inference: NVFP4 vs MXFP4 in Production (2026)

Q: What is the difference between NVFP4 and MXFP4?

Both use the same 4-bit E2M1 element, but they scale it differently. MXFP4, defined in the OCP Microscaling specification, shares one power-of-two E8M0 scale across a block of 32 values. NVFP4, NVIDIA's format, shares an FP8 E4M3 scale across a block of 16 values and adds an FP32 scale per tensor. The finer, more precise NVFP4 scale keeps more accuracy at a cost of 4.5 bits per value instead of 4.25, and it only gets native compute acceleration on Blackwell GPUs.

Q: Does FP4 really double inference throughput compared with FP8?

Only on hardware with native FP4 tensor cores and only when the workload is compute-bound at high batch. NVIDIA's DGX B200 specification lists 72 petaflops of dense FP4 against 36 petaflops of dense FP8, which is the source of the 2x figure. On Hopper and earlier GPUs there is no FP4 tensor core, so serving stacks run weight-only W4A16, which saves memory bandwidth but performs the multiply at 16-bit. Decode-bound traffic gains from the halved weight reads on any GPU; the compute doubling is Blackwell-only.

Q: Can I serve an NVFP4 model on an H100?

Yes, but not as true FP4. The vLLM Model Optimizer documentation states that on a GPU without a supported native FP4 GEMM kernel, vLLM falls back to weight-only W4A16 execution through Marlin, logs a warning and may lose throughput on compute-heavy workloads. You get the roughly 1.8x memory reduction against FP8 and none of the tensor-core gain, while paying the full accuracy cost of 4-bit weights. Pin the linear backend and fail readiness on fallback so this does not happen silently.

Q: How much accuracy do you lose going from FP8 to NVFP4?

NVIDIA's published post-training results on DeepSeek-R1-0528 show drops of one point or less on MMLU-Pro (85 to 84), GPQA Diamond (81 to 80) and LiveCodeBench (77 to 76), with MATH-500 unchanged at 98 and AIME 2024 two points higher in NVFP4. Those are vendor numbers on a large reasoning model. Smaller models lose more because they have less parameter redundancy, and exact-match tasks such as structured extraction and tool-call formatting surface small perplexity changes as hard failures, so measure on your own evaluation set.

Q: Why did OpenAI ship gpt-oss in MXFP4 rather than NVFP4?

The gpt-oss-120b model card states the mixture-of-experts weights were post-trained with MXFP4 quantization so the 117-billion-parameter model fits a single 80 GB GPU such as an NVIDIA H100 or AMD MI300X. MXFP4 is the open OCP format co-authored by AMD, Arm, Intel, Meta, Microsoft, NVIDIA and Qualcomm, so a checkpoint in it can be read on any vendor's hardware. For a public release intended to run everywhere, portability outweighs the extra accuracy NVFP4's finer scale would give on Blackwell.

Q: Which layers should stay at higher precision when quantising to FP4?

Start by excluding the embedding layer, the output projection (lm_head) and usually the attention projections, because a handful of outlier channels in those layers can dominate a 16-value or 32-value block and wreck the shared scale. NVIDIA's NVFP4 pretraining paper kept a few layers at higher precision even while training in 4-bit. Quantise the large feed-forward and expert matrices first, where the memory saving is, then try to include the excluded layers only with evaluation evidence.

Q: When should a team not move to FP4 inference?

Skip FP4 when traffic is prefill-dominated, such as long-prompt RAG or classification, because prefill is compute-bound and FP8 already saturates Hopper tensor cores; when accuracy headroom is thin or the task is scored by exact match; when the fleet is mixed or older and the business case assumed a compute doubling rather than a memory saving; and for fine-tuned models whose fine-tune has not been re-validated at low precision. In each case fix batching and quantise the KV cache first.

Q: Does quantising weights to FP4 help with long-context inference?

Less than teams expect. At long context the KV cache, not the weights, is the dominant data streamed through HBM at each decode step, and quantising weights does nothing for the cache. A 4-bit model with a 16-bit KV cache is half a migration. Quantise the KV cache to FP8, consider offloading tiers for very long contexts, and then measure whether FP4 weights still move tokens per second on your actual context-length distribution.

Satyam Kumar

Back to Blog

architecture

FP4 Inference in Production: The Fourth Bit Lives in the Scale, So Choose the Block Format and the Kernel Before You Choose the Model (2026)

By Satyam KumarOctober 20, 202616 min read

fp4 inference nvfp4 mxfp4 4-bit quantization blackwell fp4 ocp microscaling llm quantization fp8 vs fp4 vllm nvfp4 tensorrt model optimizer llm compressor gpt-oss mxfp4 low precision inference decode throughput inference cost per token gpu memory bandwidth

FP4 Inference in Production: The Fourth Bit Lives in the Scale, So Choose the Block Format and the Kernel Before You Choose the Model (2026)

Frequently Asked Questions

Share this article

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

AI & Cloud Architect. Helping teams build systems that scale to millions.

LinkedIn GitHub

FP4 Inference in Production: The Fourth Bit Lives in the Scale, So Choose the Block Format and the Kernel Before You Choose the Model (2026)

Frequently Asked Questions

Share this article

Comments

Leave a comment