How to Benchmark AI Accelerators Honestly (MLPerf 2026)

Q: Why do AI accelerator benchmarks from different vendors disagree?

Each vendor measures at the operating point where it wins: a particular model, precision, batch size, concurrency and latency limit, often with tuned or unreleased software. Without accuracy and latency constraints stated, numbers are not comparable. Compare systems only at the same operating point, ideally one taken from your own production traffic and service level objective.

Q: What is MLPerf Inference v6.1?

It is the MLCommons inference benchmark round published on 16 September 2026. It had a record 30 submitters, added an end-to-end RAG benchmark and an edge agentic inference benchmark with multi-turn context, and added speculative decoding support in the interactive scenario. MLCommons reported up to a 5.7x performance gain compared with one year earlier.

Q: What is MLPerf Endpoints and how is it different from MLPerf Inference?

MLPerf Endpoints measures a served endpoint, including model, hardware, serving software and network, through the API applications call. It sweeps concurrency and publishes a Pareto frontier with TTFT at P90, system throughput, throughput per watt and per-user interactivity. MLPerf Inference measures systems in fixed scenarios at predefined operating points. Endpoints v1.0 opened for submissions in October 2026.

Q: What is the difference between MLPerf Offline, Server and Interactive scenarios?

Offline sends all queries at once and measures maximum throughput with no latency target. Server and Interactive send queries on a Poisson arrival schedule for at least 600 seconds and require percentile latency limits; Interactive has tighter limits, for example 1,500 ms TTFT and 15 ms TPOT for DeepSeek-R1. Comparing one vendor's Offline number with another's Server number is meaningless.

Q: What is goodput in LLM inference?

Goodput is the rate of requests or tokens that met your latency service level objective and passed your accuracy check. Throughput counts everything the system produced, including answers that came back too late or wrong. Pricing accelerators on goodput instead of raw throughput often reverses the ranking, because a faster system may only be faster outside your latency limit.

Q: How do I calculate cost per million tokens for an accelerator?

Measure the output tokens per second that meet your SLO at your real concurrency, multiply by 3,600 to get good tokens per hour, then divide the hourly cost by that figure in millions. Add power if you own the hardware and divide by expected utilisation, because idle capacity is paid for too. In our example, $40 per hour at 32.4 million good tokens is about $1.23 per million.

Q: Should I trust datasheet TFLOPS when choosing an accelerator?

Use it only to check the hardware class. A figure such as 2,070 FP4 TFLOPS sparse on NVIDIA Jetson Thor is accurate, but it assumes 4-bit floating point and structured sparsity that most models do not use, and it says nothing about memory bandwidth, software maturity or latency. Token-by-token decoding is usually limited by memory bandwidth, not arithmetic.

Q: When should I not run my own accelerator benchmark?

When you are not buying hardware. If you consume a managed model API at modest volume, benchmark the endpoint at your concurrency for TTFT and cost per million tokens and stop. Also avoid bake-offs between systems your software stack cannot run without a migration; price that migration honestly first, because a better leaderboard result does not pay for porting kernels.

Satyam Kumar

Back to Blog

architecture

A Benchmark Is a Claim About an Operating Point: How to Benchmark AI Accelerators Honestly in 2026

By Satyam KumarOctober 27, 202610 min read

ai accelerator benchmarking mlperf inference v6.1 mlperf endpoints gpu benchmark comparison inference goodput cost per million tokens ttft tpot latency mlcommons eu ai gigafactories tender ai infrastructure procurement tokens per second per user fp4 sparse tflops inference tco llm serving benchmark gpu procurement pareto frontier concurrency

A Benchmark Is a Claim About an Operating Point: How to Benchmark AI Accelerators Honestly in 2026

Frequently Asked Questions

Share this article

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

AI & Cloud Architect. Helping teams build systems that scale to millions.

LinkedIn GitHub

A Benchmark Is a Claim About an Operating Point: How to Benchmark AI Accelerators Honestly in 2026

Frequently Asked Questions

Share this article

Comments

Leave a comment