LLM Throughput & Latency Calculator
How many tokens per second will your LLM actually serve, and are you limited by memory bandwidth or compute? Decode speed is set by how fast the GPU can read the model’s weights from memory — not by its FLOPs — so a single request is bandwidth-bound, while time-to-first-token is compute-bound on the prompt. Set your model, GPU, batch, and utilization below for single-stream speed, TTFT, and batched aggregate throughput, with the bottleneck named. It is a first-order roofline for planning.
LLM Throughput & Latency Calculator
First-order roofline. Decode speed = weights ÷ (memory bandwidth × MBU): every output token reads the whole model from memory, so bandwidth, not FLOPs, sets single-stream speed. Batching amortizes those weight reads across requests and raises aggregate throughput until you hit the compute ceiling. Real numbers vary with KV-cache traffic, attention kernels, speculative decoding, chunked prefill, and framework efficiency (MBU/MFU) — treat as a planning baseline.
How the roofline works
- Decode (per token) is memory-bound: time per token ≈ weight bytes ÷ (memory bandwidth × MBU). Every token reads the whole model, so bandwidth sets single-stream speed.
- Prefill (TTFT) is compute-bound: it processes the full prompt in parallel, so TTFT ≈ 2 × params × input tokens ÷ (FLOPs × MFU).
- Batching amortizes the weight read across N requests, so aggregate throughput rises with batch until it hits the compute ceiling — the crossover the calculator reports as the bottleneck.
This is why HBM bandwidth, not FLOPs, is the real bottleneck, and why the accelerator choice for inference is a bandwidth decision first.
How to serve faster
- Quantize the weights — fewer bytes per token read means proportionally faster decode.
- Pick higher bandwidth — H200 over H100 raises single-stream speed directly.
- Batch to fill compute — raises aggregate throughput and cuts cost per token, at the price of per-user speed.
- Check it fits first — use the VRAM calculator, then price it with the cost calculator.
Frequently asked questions
How many tokens per second can a GPU generate for an LLM?
Single-stream decoding speed is set by memory bandwidth, not FLOPs: every output token requires reading the entire set of model weights from memory, so tokens per second is roughly the memory bandwidth times its utilization divided by the weight size. A 70B model in FP16 is about 141 GB of weights; an H100 with roughly 3.35 TB/s of bandwidth at 70 percent utilization reads that in about 60 milliseconds, giving on the order of 16 tokens per second for a single request. Smaller or quantized models read fewer bytes and run faster; larger models run slower. Aggregate throughput across many concurrent requests is much higher because batching amortizes each weight read over many sequences — the calculator reports both the single-stream speed and the batched aggregate.
Why is LLM inference memory-bandwidth-bound instead of compute-bound?
During autoregressive decoding the model produces one token at a time, and for each token it must read all of its weights and the KV cache out of high-bandwidth memory. The actual arithmetic per token — roughly two floating-point operations per parameter — is tiny relative to the bytes moved, so the accelerator finishes the math long before the data arrives and sits idle waiting on memory. That makes decode a memory-bandwidth problem: the bottleneck is how fast you can stream weights, not how many FLOPs the chip can do. This is why accelerator selection for inference is primarily a bandwidth decision, why quantization (fewer bytes per weight) speeds decoding, and why batching — which reuses each weight read across many requests — is the main lever for raising throughput.
What is time to first token (TTFT) and what determines it?
Time to first token is the latency from sending a request to receiving the first output token, and it is dominated by the prefill phase — processing the entire input prompt to build the initial KV cache. Unlike decoding, prefill processes all input tokens in parallel and is compute-bound, so TTFT scales with prompt length and inversely with the GPU’s usable FLOPs. A long prompt raises TTFT; more or faster compute lowers it. This is the opposite regime from per-token decoding, which is bandwidth-bound, which is why a system can have fast token generation but slow TTFT on long prompts, or vice versa. Techniques like chunked prefill and prefix caching reduce TTFT for repeated or long contexts.
How does batching change LLM throughput?
Batching is the single biggest lever for aggregate throughput because it amortizes the expensive part — reading the model weights from memory — across many requests. In a single decode step the GPU reads all the weights once and, if it is serving a batch of N sequences, produces N tokens from that one read, so aggregate tokens per second rises almost linearly with batch size while the memory bottleneck holds. This continues until you exhaust the compute ceiling (where the arithmetic can no longer keep up) or run out of VRAM for the KV cache. That is why production serving uses continuous or in-flight batching to keep the batch full: single-stream speed per user is modest, but aggregate throughput and cost-efficiency climb steeply with concurrency. The calculator shows the crossover by reporting the bottleneck as either memory bandwidth or compute.
What are MBU and MFU, and what values are realistic?
MBU (model bandwidth utilization) is the fraction of the GPU’s peak memory bandwidth actually achieved during decoding, and MFU (model FLOPs utilization) is the fraction of peak compute achieved during prefill. Neither reaches 100 percent because of kernel overheads, memory access patterns, and synchronization. Well-optimized inference stacks often reach roughly 60 to 80 percent MBU for decoding and 30 to 50 percent MFU for prefill, though the exact numbers depend on the model, sequence length, batch size, and framework. The calculator exposes both as sliders so you can match your measured efficiency; using peak specs without a utilization factor will overstate real throughput significantly, which is a common planning error.
Does tensor parallelism increase throughput linearly?
Adding GPUs via tensor parallelism increases the aggregate memory bandwidth and compute available, so first-order throughput scales with GPU count — but not perfectly linearly, because splitting each layer across GPUs requires exchanging activations over the interconnect on every step. With fast interconnect (NVLink) the communication overhead is modest and scaling is good; over slower links it erodes quickly, and beyond a certain degree the communication cost outweighs the added bandwidth. Tensor parallelism is also often necessary rather than optional: when a model’s weights exceed a single GPU’s VRAM, you must split it regardless of throughput. The calculator scales bandwidth and compute by GPU count as an upper bound; real speedup is somewhat lower due to interconnect overhead.
How do I hit a latency target like 30 tokens per second per user?
Work from the single-stream decode math: per-user tokens per second is bandwidth times utilization divided by weight size, so to raise it you either reduce the bytes read per token or increase bandwidth. Concretely, that means quantizing the weights (FP8 or INT4 cuts the bytes read by 2 to 4 times and speeds decoding proportionally), choosing a higher-bandwidth GPU (H200 over H100, for instance), using a smaller model, or reducing the batch size so each user gets more of the throughput. There is a direct tension with cost: large batches maximize aggregate throughput and cost-efficiency but reduce per-user speed, while small batches give each user faster tokens at higher cost per token. The calculator’s per-user figure at a given batch lets you find the batch size that meets your latency target, and you can pair it with the cost calculator to see the price of that choice.
What does this throughput calculator leave out?
It is a first-order roofline that models the dominant effect — weight bandwidth for decode, prefill compute for TTFT — but omits several second-order factors. It does not separately model KV-cache read traffic, which adds to per-token bytes at long context and high batch; attention-kernel efficiency and memory access patterns; speculative decoding, which can raise effective tokens per second; chunked prefill and its interaction with decode; MoE routing, where only a subset of parameters activate per token; or interconnect overhead in multi-GPU setups. It also does not size VRAM (whether the model even fits) — use the VRAM calculator for that. Treat the output as a planning baseline for choosing hardware and batch size, then validate with a benchmark on your actual model and serving stack.