How to Engineer AI Inference Systems with Philip Kiely - #766
Episode
54 min
Read time
2 min
Topics
Remote Work, Artificial Intelligence, Software Development
AI-Generated Summary
Key Takeaways
- ✓Inference research-to-production timeline: New inference techniques move from research paper to production implementation in hours, not weeks. Base Ten's team implemented a PoloQuant CUDA kernel 31 hours after the paper published. Engineers should monitor inference research continuously, as techniques dismissed a year ago can become immediately viable when model scales shift.
- ✓Product maturity deployment cycle: Companies follow a predictable inference path: start with per-token closed APIs, then move to hyperscaler provisioned throughput (AWS Bedrock, GCP Vertex), then dedicated inference providers like Base Ten with weight-owned models, then in-house platforms. The trigger is typically cost overruns or capacity constraints, not company size.
- ✓KV cache prefix optimization: A single token difference early in a prompt sequence invalidates the entire KV cache, eliminating reuse benefits. Engineers should structure system prompts and chat templates to maximize shared prefixes across requests, ensuring cache hits. This applies regardless of whether inference runs on owned infrastructure or third-party providers.
- ✓Task-model matching for agentic speed: Agents making hundreds of model calls per user action require specialized runtimes per task type. Running named entity recognition on a frontier LLM costs significantly more versus a specialized runtime. Base Ten's NER runtime runs in 1 millisecond versus 500 milliseconds on a small LLM — a 500x difference that eliminates visible latency in agent pipelines.
- ✓Hopper GPU staying power: H100 rental prices are higher now than a year ago despite Blackwell availability. Hopper GPUs remain dominant because open-source models from Chinese labs are optimized for Hopper architecture due to export controls, and smaller models (1–8B parameters) run efficiently on MIG-partitioned Hopper slices without requiring full Blackwell NVL72 systems.
What It Covers
Philip Kiely, Head of AI Education at Base Ten, explains inference engineering as a discipline requiring expertise across CUDA programming, distributed systems, and applied research. He covers the maturity cycle from per-token APIs to dedicated deployments, hardware generations, and why inference optimization becomes critical at scale for agentic AI workloads.
Key Questions Answered
- •Inference research-to-production timeline: New inference techniques move from research paper to production implementation in hours, not weeks. Base Ten's team implemented a PoloQuant CUDA kernel 31 hours after the paper published. Engineers should monitor inference research continuously, as techniques dismissed a year ago can become immediately viable when model scales shift.
- •Product maturity deployment cycle: Companies follow a predictable inference path: start with per-token closed APIs, then move to hyperscaler provisioned throughput (AWS Bedrock, GCP Vertex), then dedicated inference providers like Base Ten with weight-owned models, then in-house platforms. The trigger is typically cost overruns or capacity constraints, not company size.
- •KV cache prefix optimization: A single token difference early in a prompt sequence invalidates the entire KV cache, eliminating reuse benefits. Engineers should structure system prompts and chat templates to maximize shared prefixes across requests, ensuring cache hits. This applies regardless of whether inference runs on owned infrastructure or third-party providers.
- •Task-model matching for agentic speed: Agents making hundreds of model calls per user action require specialized runtimes per task type. Running named entity recognition on a frontier LLM costs significantly more versus a specialized runtime. Base Ten's NER runtime runs in 1 millisecond versus 500 milliseconds on a small LLM — a 500x difference that eliminates visible latency in agent pipelines.
- •Hopper GPU staying power: H100 rental prices are higher now than a year ago despite Blackwell availability. Hopper GPUs remain dominant because open-source models from Chinese labs are optimized for Hopper architecture due to export controls, and smaller models (1–8B parameters) run efficiently on MIG-partitioned Hopper slices without requiring full Blackwell NVL72 systems.
Notable Moment
Kiely describes how text-to-speech models have a hard ceiling of roughly 80–100 tokens per second for real-time audio output. Beyond that threshold, the optimization goal flips entirely — engineers should increase batch size for concurrent streams or reduce hardware costs rather than chasing higher token throughput.
Episode Transcript
Mean, it might be the fastest timeline in the world. If you think about medicine, for example, it can take decades for research to reach a pharmacy. If you think about, you know, physics or engineering, it can take years to apply a new concept. Even within AI, if you want to train a model off of a new technique, it can still take weeks or months to find the exact right way to sort of express that technique. But with inference, the timeline is often hours. A new model architecture comes out. You have to figure out how to support it day zero. Alright, everyone. Welcome to another episode of the TwiML AI podcast. I am your host, Sam Sherrington. Today, I'm joined by Philip Kiley. Philip is head of AI education at Base ten. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Phillip, great to see you again, and welcome to the podcast. Hey, Sam. Thanks for having me on. Absolutely. Absolutely. We met at, the recent GTC conference where you were signing copies of that book that you have there, over your, I guess, your left shoulder. Yeah. The And I've got one here on my desk in front of center now. There you go. That was a really fun conference. I I felt like, like a rock star, you know, just, like, handing out. We also had honestly, I think part of it was we had free ice cream at the booth as well. Uh-huh. And, I gave away a lot of books, but I did run the numbers at the end of the show. The ice cream was more popular. You know, I I thought, it'd be good to get you on and talk through, like, you know, a little bit about, you know, inference engineering, you know, what what you're writing about in the book, but also kinda what you're seeing beyond the book and, what is really kinda making a difference from, you know, inference perspective nowadays. So before we get to that, though, I'd love to have you talk a little bit about, you know, your journey, like, what brought you to, you know, Base 10 and inference engineering. So back in 2022, this was ten months before ChatGPT was launched to the public. I had a a really strong and novel thesis about the AI industry. That strong and novel thesis was I need a job. And so I, I applied for a bunch of jobs and I ended up choosing between Base ten and a laundry delivery startup. And it was, like, a really tough decision, because it's hey. It's it's a good the other one's a good company to show around. But ultimately, I I decided to go work on some some really interesting problems in the ML space with a tiny startup at the time called called Base ten. And I've been at it for …
Get the full transcript (8,612 words) + summary by email — free
One-time email with the complete transcript and AI summary of this episode. No account needed.
One email, no spam. We’ll also show you what SignalCast does.
You just read a 3-minute summary of a 51-minute episode.
Get The TWIML AI Podcast summarized like this every Monday — plus up to 2 more podcasts, free.
Pick Your Podcasts — FreeKeep Reading
More from The TWIML AI Podcast
Do AI Tokenomics Matter More Than Model Benchmarks? with Chris Potts - #776
Sep 9 · 59 min
Latent Space
The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten
Aug 3
More from The TWIML AI Podcast
World Models and the Future of Spatial AI with Justin Johnson - #775
Sep 1 · 66 min
Software Engineering Daily
How LLMs Are Reshaping Recommendation Systems
Aug 18
Books, tools, and gear mentioned in this episode
SignalCast may earn commission on purchases via these links.
Tools
by Amazon Web Services
“Companies follow a predictable inference path: start with per-token closed APIs, then move to hyperscaler provisioned throughput (AWS Bedrock, GCP Vertex), then dedicated inference providers like Base Ten with weight-owned models”
by Google Cloud
“Companies follow a predictable inference path: start with per-token closed APIs, then move to hyperscaler provisioned throughput (AWS Bedrock, GCP Vertex), then dedicated inference providers like Base Ten with weight-owned models”
company
“Philip Kiely, Head of AI Education at Base Ten, explains inference engineering as a discipline requiring expertise across CUDA programming, distributed systems, and applied research.”
More from The TWIML AI Podcast
We summarize every new episode. Want them in your inbox?
Do AI Tokenomics Matter More Than Model Benchmarks? with Chris Potts - #776
World Models and the Future of Spatial AI with Justin Johnson - #775
Why the Next AI Breakthrough May Come from Physics with Max Welling - #774
Why Image Generation Needs More Than Bigger Models with Fatih Porikli - #773
Why Models Are AI’s Next Training Dataset with Damian Borth - #772
Similar Episodes
Related episodes from other podcasts
Latent Space
Aug 3
The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten
Software Engineering Daily
Aug 18
How LLMs Are Reshaping Recommendation Systems
How I AI
Sep 7
Build your own company brain: the enterprise AI playbook from Stripe’s engineering team | Sharadh Krishnamurthy
Lenny's Podcast
Aug 9
The playbook for building high talent density teams | Adam Ward, Head of Talent at Cursor
The Vergecast
Jul 16
The one AI detector people actually trust
Explore Related Topics
This podcast is featured in Best AI Podcasts (2026) — ranked and reviewed with AI summaries.
Read this week's AI & Machine Learning Podcast Insights — cross-podcast analysis updated weekly.
You're clearly into The TWIML AI Podcast.
Every Monday, we deliver AI summaries of the latest episodes from The TWIML AI Podcast and 192+ other podcasts. Free for one show.
Start My Monday DigestNo credit card · Unsubscribe anytime