How to Engineer AI Inference Systems with Philip Kiely - #766

What It Covers

Philip Kiely, Head of AI Education at Base Ten, explains inference engineering as a discipline requiring expertise across CUDA programming, distributed systems, and applied research. He covers the maturity cycle from per-token APIs to dedicated deployments, hardware generations, and why inference optimization becomes critical at scale for agentic AI workloads.

Key Questions Answered

• •Inference research-to-production timeline: New inference techniques move from research paper to production implementation in hours, not weeks. Base Ten's team implemented a PoloQuant CUDA kernel 31 hours after the paper published. Engineers should monitor inference research continuously, as techniques dismissed a year ago can become immediately viable when model scales shift.
• •Product maturity deployment cycle: Companies follow a predictable inference path: start with per-token closed APIs, then move to hyperscaler provisioned throughput (AWS Bedrock, GCP Vertex), then dedicated inference providers like Base Ten with weight-owned models, then in-house platforms. The trigger is typically cost overruns or capacity constraints, not company size.
• •KV cache prefix optimization: A single token difference early in a prompt sequence invalidates the entire KV cache, eliminating reuse benefits. Engineers should structure system prompts and chat templates to maximize shared prefixes across requests, ensuring cache hits. This applies regardless of whether inference runs on owned infrastructure or third-party providers.
• •Task-model matching for agentic speed: Agents making hundreds of model calls per user action require specialized runtimes per task type. Running named entity recognition on a frontier LLM costs significantly more versus a specialized runtime. Base Ten's NER runtime runs in 1 millisecond versus 500 milliseconds on a small LLM — a 500x difference that eliminates visible latency in agent pipelines.
• •Hopper GPU staying power: H100 rental prices are higher now than a year ago despite Blackwell availability. Hopper GPUs remain dominant because open-source models from Chinese labs are optimized for Hopper architecture due to export controls, and smaller models (1–8B parameters) run efficiently on MIG-partitioned Hopper slices without requiring full Blackwell NVL72 systems.