The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten

What It Covers

Philip Kiely and Ali Taha from Baseten walk through the full inference engineering stack — from cache-aware routing and prefill/decode disaggregation to speculative decoding, quantization strategies, and hardware parallelism — explaining how providers stack optimizations to achieve 4–10x speed improvements over baseline deployments, while covering video diffusion constraints and the convergence of training and inference workflows.

Key Questions Answered

• •Cache-Aware Routing: When a long prompt (e.g., 200,000 tokens) arrives, route it to a replica that already holds a matching KV cache to skip prefill entirely. Baseten checks for cached inputs before assigning a prefill worker. For repeated agentic or coding workloads, this alone delivers significant latency reduction. Disaggregating prefill and decode across separate GPU sets further prevents decode throughput from being bottlenecked by heavy prefill computation on the same hardware.
• •Speculative Decoding Gains and Limits: A small draft model — typically one-sixteenth the size of the target model, around 1B parameters — predicts 3 tokens per step, which the full model verifies in one forward pass. Acceptance rates are traffic-dependent: a coding-specific speculator achieves near-perfect acceptance on coding prompts but degrades on unrelated tasks. For shared endpoints, general-purpose speculators trained on public coding and agentic datasets are the practical default, delivering roughly a 2x decode speed improvement.
• •Quantization Error Cancellation: Baseten's research intern Joshua demonstrated mathematically that quantizing more layers can produce better output fidelity than quantizing fewer, because quantization errors across layers can cancel out. By predicting which layers have offsetting error distributions — one skewed right, the next left — and selectively quantizing those, Baseten achieved 20% higher quantization than NVIDIA's baseline on GLM 5.2 while improving KL divergence between quantized and full-precision logit distributions, yielding both better throughput and better model fidelity simultaneously.
• •Stacking Optimizations for 4–10x Gains: A baseline GLM 5.2 deployment on Hoppers without optimization runs at roughly 30–40 tokens per second. Moving from BF16 to INT8 adds ~30–40%; INT8 to NVFP4 adds another ~30–40% (combined ~2x). A well-trained speculator adds another ~2x. Prefill/decode disaggregation adds a further ~2x under sufficient traffic. Stacking these three optimizations reaches approximately 8–10x over baseline, with the best-configured deployments hitting 300–400 tokens per second on latency-optimized parallelism settings.
• •Dedicated vs. Serverless Inference Threshold: The decision to move from pay-per-token serverless APIs to dedicated GPU deployments is driven by three factors: reliability requirements, traffic volume (millions of tokens per hour make hourly billing substantially cheaper), and workload specificity. Dedicated deployments unlock traffic-specific speculative decoders, custom quantization precision, tuned batch sizes, and parallelism strategies optimized for latency versus throughput — none of which are feasible on shared endpoints where traffic patterns are unknown and hardware is contended.
• •Video Diffusion Attention Bottleneck: Generating five seconds of 480p video at 16 fps produces roughly 35,000 attention tokens after latent compression. Because diffusion attention is quadratic, doubling duration quadruples compute, making full-attention generation beyond ~10 seconds practically infeasible on current hardware. Sparse attention (top 12.5% of tokens) reduces compute but degrades quality. Autoregressive video models would enable streaming generation and allow LLM-style inference optimizations like speculative decoding, but no open-source autoregressive video model currently matches diffusion quality.
• •Training–Inference Convergence: Inference engineers at Baseten now routinely write training pipelines. Two workflows drive this: first, speculative decoder (Eagle-head) training requires running inference on the base model to extract hidden states as training data. Second, when NVFP4 post-training quantization (PTQ) degrades model quality, quantization-aware distillation — running full-precision and NVFP4 model versions in parallel and aligning their logit distributions — recovers fidelity. NVIDIA's quantization distillation paper formalizes this approach, and RL training rollout speed directly depends on inference engine throughput, making inference optimization a training bottleneck.