LLM Inference Latency

Executive Summary

The NVIDIA spec sheet says the H100 delivers 989 TFLOPS of FP16 compute. The A100 delivers 312 TFLOPS. The T4 delivers 65 TFLOPS. The H100 is 15× faster than a T4 in floating-point math. So a 7-billion parameter LLM should be 15× faster on an H100, right?

Wrong. It's more like 150× faster. The T4 manages ~15 tokens/sec on a 7B FP16 model. The H100 pushes ~2,200 tokens/sec (with continuous batching, ~3,500). The 15× TFLOPS gap does not explain the 150× throughput gap. The missing variable — the one that actually matters for LLM inference — is memory bandwidth. The T4 has 300 GB/s. The H100 has 3,350 GB/s. That's an 11× gap. Combined with larger L2 cache (50 MB vs 6 MB) and faster clock speeds, the effective memory throughput gap widens to 100–150× for the memory-bound decode phase of autoregressive generation.

This tool models LLM inference throughput by starting from first principles: model parameter count → bytes per token (given quantization level) → time to read those bytes (given GPU memory bandwidth) → tokens per second. It then layers in the real-world overhead factors — attention computation, KV cache management, framework efficiency — that turn theoretical maximums into actual benchmarks. Use it to answer the question every ML engineer asks before renting a GPU: "How many tokens per second will I actually get?"

Inference Throughput Calculator

Model throughput across 12 models × 7 GPUs × 4 quantization levels. All calculations client-side.

Estimated Performance

The Memory Bandwidth Bottleneck — Why TFLOPS Don't Matter for Inference

Autoregressive LLM inference splits into two phases: prefill (processing the input prompt — compute-bound) and decode (generating output tokens one at a time — memory-bound). The decode phase dominates total latency for any response longer than a few tokens. And in the decode phase, the GPU spends most of its time waiting — not for the compute units to finish a matrix multiply, but for the next chunk of model weights to arrive from VRAM.

The Arithmetic: Why 989 TFLOPS Doesn't Help

For each token generated, the GPU must perform two operations: (1) read every model parameter from VRAM into the compute units, and (2) perform the matrix multiplications and attention computations. Step 1 takes this long, for a 7B parameter model at FP16:

• Weight data per token: 7 billion params × 2 bytes (FP16) = 14 GB
• On H100 (3,350 GB/s): 14 GB ÷ 3,350 GB/s = 4.18 milliseconds
• Theoretical ceiling: 1,000 ms ÷ 4.18 ms = 239 tokens/sec
• On T4 (300 GB/s): 14 GB ÷ 300 GB/s = 46.7 ms → 21 tokens/sec theoretical

Step 2 (the actual computation) for a 7B model at FP16 requires approximately 14 TFLOPs of compute per token. The H100's 989 TFLOPS could theoretically do this 70 times over in the 4.18 ms window. The compute units are idle 98%+ of the time during decode, waiting for memory.

This is the central insight of LLM inference optimization: you cannot compute faster than you can read. Every optimization that matters — quantization, KV cache compression, speculative decoding, FlashAttention — is fundamentally about reducing the number of bytes that must move between VRAM and compute units per token. The TFLOPS rating of the GPU is almost irrelevant.

GPU Memory Bandwidth Comparison

"—" means the model does not fit in VRAM at this precision on this GPU. Theoretical tok/s assumes 100% memory bandwidth utilization for weight reads only. Real throughput is 60–85% of theoretical after attention overhead, KV cache, and framework inefficiency.

Quantization — The Throughput Lever That Doesn't Need a Faster GPU

If inference throughput is memory-bandwidth-bound, and you can't change the GPU's memory bandwidth, the only remaining lever is to reduce the number of bytes per parameter. That's quantization. Moving from FP16 (2 bytes/param) to INT4 (0.5 bytes/param) reduces the memory traffic per token by 4× — and increases theoretical throughput by the same factor. On the same GPU.

Quantization Levels Compared

The quantization-quality threshold: At 7B+ parameters, INT4 quantization (with GPTQ, AWQ, or GGUF's Q4_K_M) produces quality degradation that is measurable in benchmarks (1–3% perplexity increase) but frequently imperceptible in real-world use. At 70B+ parameters, even 3-bit quantization is viable. The larger the model, the more aggressively you can quantize it before quality degrades — a 405B model at 2-bit often outperforms a 70B model at FP16 on knowledge tasks, despite using the same number of bytes per token (101 GB vs 140 GB).

What quantization doesn't improve: Time-to-first-token (prefill latency). The prefill phase processes the entire input prompt in parallel and is compute-bound, not memory-bound. Quantization reduces the memory traffic but the compute units were already the bottleneck in prefill — quantization can actually make prefill slightly slower on some hardware due to dequantization overhead. The throughput gains from quantization apply almost entirely to the decode phase.

Real Benchmark Data — Measured Tokens/sec Across Model × GPU Combinations

The theoretical ceiling is clean math. Real throughput is messier — framework overhead, attention kernel efficiency, KV cache management, and the gap between "the model fits in VRAM" and "the model runs with usable context length." These benchmarks were collected with vLLM 0.6.x, continuous batching enabled, on bare-metal GPU instances, mid-2026.

"TP=N" = Tensor Parallelism across N GPUs. "—" in batch columns = VRAM insufficient for that batch size at listed context length. Benchmarks with vLLM 0.6.x, CUDA 12.4, continuous batching. Output token decode only — prefill latency not included. Batch throughput is total tokens/sec across all concurrent requests.

The Batch Size Tradeoff — Throughput vs Latency

Batching is the primary lever for GPU utilization. A GPU processing one request at a time leaves most of its compute units idle — the memory subsystem is the bottleneck, but the compute units that are waiting on memory could be doing useful work for other requests. Batching fills that idle time: while one request waits for the next chunk of weights, another request's compute can run.

But batching has a cost: per-request latency increases with batch size because every request in the batch must complete the current decode step before any request can move to the next step. At batch size 32, a single slow request (long output, complex attention pattern) holds up all 31 other requests.

Batch Size Decision Guide

Continuous batching (also called in-flight batching or iteration-level scheduling) is the critical innovation that makes production LLM serving viable. Traditional static batching waits for all requests in the batch to complete before admitting new requests. Continuous batching evicts completed requests and admits new ones at every decode step. This eliminates the "straggler tax" — a request generating a 500-token response no longer blocks 31 other requests that finished in 50 tokens. vLLM, TensorRT-LLM, and SGLang all implement continuous batching. If you're deploying an LLM in production without it, you are likely getting 30–50% of the throughput your GPU could deliver.

Multi-GPU Inference — When One GPU Isn't Enough

When a model doesn't fit on a single GPU, you need to split it across multiple GPUs. The two strategies are tensor parallelism (split each layer across GPUs — GPUs communicate on every forward pass) and pipeline parallelism (split the model by layers — each GPU holds a contiguous block of layers). For inference, tensor parallelism is almost always the correct choice: it minimizes latency by parallelizing every operation, and the NCCL communication overhead (all-reduce after each attention and FFN block) is acceptable at inference batch sizes.

Multi-GPU Scaling Efficiency (Measured on 70B FP16, A100-80GB)

The scaling efficiency declines with each additional GPU — NCCL all-reduce costs scale with the number of participating GPUs. At TP=8, nearly 40% of the interconnect bandwidth goes to synchronization, not useful computation. This is why the industry is moving toward expert-parallel inference for MoE models: each expert lives on a dedicated GPU, and only the active experts (typically 2 of 8 for DeepSeek-V3, or 1 of 8 for OLMoE) participate in any given forward pass. The inactive GPUs are idle — but the active ones don't pay the all-reduce tax. Expert parallelism achieves 95%+ per-GPU efficiency for MoE models, which is why a 671B MoE model (37B active) is faster than a 70B dense model despite having 10× more total parameters.

📜 Copyright & Attribution