Reiner Pope – The math behind how LLMs are trained and served

What It Covers

Reiner Pope, CEO of chip startup Maddox and former Google TPU architect, delivers a blackboard lecture explaining the mathematical foundations of LLM training and inference. Using roofline analysis, he quantifies how batch size, memory bandwidth, compute throughput, KV cache, sparsity, and parallelism strategies determine API pricing, model latency, and why AI architectures have evolved the way they have.

Key Questions Answered

• •Batch Size Economics: The cost per token follows a hyperbolic curve at small batch sizes because weight-loading costs are not amortized across users. At batch size one, costs approach near-infinite relative to large batches. The crossover point — where memory bandwidth and compute are balanced — occurs at roughly 300 times the model's sparsity ratio. For DeepSeek with 1-in-8 expert activation, this means approximately 2,400 concurrent sequences before compute, not memory, becomes the binding constraint.
• •Optimal Batch Size Formula: The hardware-determined balance point between memory-bound and compute-bound operation equals FLOPS divided by memory bandwidth (approximately 300 on most GPU generations, stable across H100 to B100) multiplied by the sparsity ratio of active-to-total parameters. For a dense model this is roughly 300 sequences; for a sparse MoE model like DeepSeek with 32 of 256 experts active, it scales down proportionally, making sparse models far more economical to serve at realistic traffic volumes.
• •KV Cache as the Persistent Bottleneck: Unlike weight matrices, KV cache cannot be amortized across batch elements or sharded efficiently across pipeline stages. Each sequence requires its own full context-length KV cache, scaling linearly with both batch size and context length. This makes long-context inference disproportionately expensive and explains why Gemini charges 50% more above 200K tokens — that threshold corresponds to the crossover where KV fetch time exceeds weight fetch time on their hardware configuration.
• •Decode vs. Prefill Pricing: Output tokens cost roughly five times more than input tokens across major APIs because decode is heavily memory-bandwidth-bound while prefill is compute-bound. During prefill, the memory cost per token drops as context length increases, since weight-loading is amortized across many tokens processed in parallel. The five-times price gap directly reveals the ratio of memory-bound to compute-bound operation, making API pricing a readable signal of a provider's hardware efficiency profile.
• •Expert Parallelism Constrains Rack Size: Mixture-of-experts layers require all-to-all communication patterns, where any GPU may need to send tokens to any other GPU. This pattern fits perfectly within a single NVLink rack (72 GPUs on Blackwell) but degrades sharply across rack boundaries because inter-rack bandwidth via scale-out networks runs approximately eight times slower than intra-rack NVLink. This physical constraint is why frontier inference deployments maximize expert parallelism within one rack before considering pipeline parallelism across racks.
• •Pipeline Parallelism Solves Weight Storage, Not KV Cache: Distributing model layers across multiple racks via pipeline parallelism reduces per-rack weight storage proportionally, but KV cache memory requirements remain constant per GPU regardless of pipeline depth. As pipeline stages increase, KV cache becomes the dominant memory consumer, eliminating the capacity benefit. The practical implication: pipelining is useful when model weights exceed single-rack capacity, but adding more stages beyond two or three yields diminishing returns for inference workloads.
• •Overtraining Ratio Derivable from API Traffic: By equating training compute cost with inference compute cost — a heuristic that holds when total cost curves cross — and assuming roughly 50 million output tokens per second for a frontier model over a two-month deployment window, the implied pretraining token count reaches approximately 150 trillion tokens. Chinchilla-optimal for a 100-billion active-parameter model is around 2 trillion tokens, suggesting current frontier models are overtrained by a factor of roughly 100x relative to compute-optimal training.