Lab

Inference throughput, estimated

Prefill is compute-bound — it's roughly 2×params×tokens FLOPs, capped by peak GPU throughput and how much of it you actually realize (MFU). Decode is memory-bandwidth-bound — every step re-reads the model weights and the growing KV cache, so throughput comes down to bytes moved per token, not FLOPs. Every constant that affects the answer is an adjustable input below, not a hidden assumption.

Model

GPU

Precision

Tensor-parallel degree

Batch size (concurrent sequences)

Sequence length (prompt + context)

Advanced assumptions

Prefill MFU

GQA ratio (KV heads / attn heads)

Time to first token

—

Decode throughput

—

Inter-token latency

—

Memory footprint

—

Params / GPU (weights, TP-sharded)

KV cache / GPU at max length

Prefill FLOPs (batch)

Bytes read per decode step / GPU

This is a first-order estimate for capacity planning intuition, not a substitute for benchmarking a real serving stack (vLLM / SGLang / TensorRT-LLM) — it ignores scheduling overhead, attention-kernel efficiency, network/PCIe transfer time for disaggregated prefill-decode, and speculative decoding. GPU specs are vendor-quoted dense figures; treat B200 numbers as approximate. Tensor-parallel communication overhead (an AllReduce per layer — see the AllReduce visualizer) is not modeled here.