#324 Sharon Zhou: Inside AMD's Plan to Build Self-Improving AI
Episode
46 min
Read time
2 min
Topics
Productivity, Fundraising & VC, Artificial Intelligence
AI-Generated Summary
Key Takeaways
- ✓Catastrophic Forgetting Prevention: When fine-tuning models without access to original pre-training data, reintroducing as little as 1% of pre-training data during post-training significantly reduces catastrophic forgetting. This allows models to reconnect with earlier representations. Developers doing heavy post-training workloads should monitor this actively, as even small fine-tuning tasks can compound into larger degradation over time.
- ✓Kernel Optimization Economics: Speeding up a single matrix multiplication kernel — which executes billions or trillions of times inside one model — can translate to hundreds of billions of dollars in savings at frontier scale. Even for smaller deployments, a 10x kernel speed improvement is economically equivalent to purchasing 10x more GPU compute hardware.
- ✓Verifiable Rewards for RL Training: AMD's kernel generation pipeline uses GPU profiler output as a verifiable reward signal for reinforcement learning, similar to how math correctness verified ChatGPT's reasoning training. Because profiler speed metrics are objective and non-subjective, they feed directly back into post-training loops without requiring costly human preference labeling.
- ✓AMD's Open-Source Advantage: AMD's ROCm software stack is open-source, unlike NVIDIA's CUDA. This means language models can train directly on ROCm documentation and code, enabling AI agents to learn AMD-specific kernel writing more effectively. Developers building on AMD hardware benefit from this transparency when using AI coding tools like Cursor to generate optimized kernels.
- ✓Kernel Engineering Skill Gap: Writing optimized GPU kernels requires simultaneous expertise in GPU architecture specifics and model mathematics — a combination rare enough to bottleneck even frontier labs. A complex kernel can take a non-expert months and an expert several weeks to write manually, making AI-assisted kernel generation a high-leverage productivity multiplier across the entire AI development stack.
What It Covers
Sharon Zhou, VP of AI at AMD and Stanford PhD graduate, explains how AMD uses AI agents and reinforcement learning to autonomously generate and optimize low-level GPU kernel code, enabling language models to run faster on AMD hardware while reducing the rare human expertise bottleneck in kernel engineering.
Key Questions Answered
- •Catastrophic Forgetting Prevention: When fine-tuning models without access to original pre-training data, reintroducing as little as 1% of pre-training data during post-training significantly reduces catastrophic forgetting. This allows models to reconnect with earlier representations. Developers doing heavy post-training workloads should monitor this actively, as even small fine-tuning tasks can compound into larger degradation over time.
- •Kernel Optimization Economics: Speeding up a single matrix multiplication kernel — which executes billions or trillions of times inside one model — can translate to hundreds of billions of dollars in savings at frontier scale. Even for smaller deployments, a 10x kernel speed improvement is economically equivalent to purchasing 10x more GPU compute hardware.
- •Verifiable Rewards for RL Training: AMD's kernel generation pipeline uses GPU profiler output as a verifiable reward signal for reinforcement learning, similar to how math correctness verified ChatGPT's reasoning training. Because profiler speed metrics are objective and non-subjective, they feed directly back into post-training loops without requiring costly human preference labeling.
- •AMD's Open-Source Advantage: AMD's ROCm software stack is open-source, unlike NVIDIA's CUDA. This means language models can train directly on ROCm documentation and code, enabling AI agents to learn AMD-specific kernel writing more effectively. Developers building on AMD hardware benefit from this transparency when using AI coding tools like Cursor to generate optimized kernels.
- •Kernel Engineering Skill Gap: Writing optimized GPU kernels requires simultaneous expertise in GPU architecture specifics and model mathematics — a combination rare enough to bottleneck even frontier labs. A complex kernel can take a non-expert months and an expert several weeks to write manually, making AI-assisted kernel generation a high-leverage productivity multiplier across the entire AI development stack.
Notable Moment
When asked whether faster kernel optimization might relieve the global chip shortage pressure, Zhou flatly dismissed the idea — stating that demand for compute is effectively infinite and no organization has reached a point where efficiency gains reduce their appetite for more hardware.
Episode Transcript
Catastrophic forgetting is definitely a problem, especially in post training and when you don't have access to the original pre training data. How much of that is focused on improving the design of AMD hardware and how much of it it is focused on general model development, which AMD is not doing. Right? AMD does not develop its own models. I think people want infinite chips, Greg. So no. It doesn't relieve the pressure. I am Sharon. I'm the VP of AI at AMD, and I I think about self improving AI, self improving LLMs, which we'll get into later. But my background comes from AI research. So I used to be an AI researcher at Stanford where I did my PhD with Andrew Ng. I taught there as adjunct in generative AI back before all this chat GPT stuff. And after Stanford, I started a start up, an, AI infrastructure start up doing post training of language models, actually on AMD GPUs. This was started a couple months before ChatGPT launched. And most recently over last, several months, we have transitioned now to AMD. So my team and I are are there now, and very excited to enable more people to use, compute and to get access to compute because that really is the limiting factor, one of the big limiting factors, for developing AI, and being able to enable more people to steer these models. So that's what I'm really excited about. And, yeah, that's that's why I'm here. Yeah. And and I do wanna talk about self improving AI. Can you start by defining what we're talking about when we, talk about self improvement? Are we talking about models that that rewrite their own code or something, like, refining their own training data? Yeah. I think that's exactly it. It is a broad category, but, essentially, it's the idea of these models being able to edit any part of themselves to improve themselves. So whether that be the data, whether that be the actual model architecture, whether that be, how they evaluate themselves. And, actually, the part that I'm working on is below all of that, and it's actually how fast they actually run on the GPUs themselves. So they are writing the kernel code that under you know, that underlies these models, to run faster on these GPUs, and to run effectively on them. Yeah. And on new hardware too. So, that's that's been really exciting to see. Yeah. I I just read about kernel evolve. Yes. Who who is doing kernel evolve? I've forgotten. Is that Google? Not at all. Is is that your guys' work? So there's a lot of different, pieces of work around, you know, kernel generation and, being able to use LLMs to generate these kernels. So we're doing some of that, but I think that work might have been from Metta. But there's work across, Yeah. I think there's work across the board, across the industry, that is very important, …
Get the full transcript (7,345 words) + summary by email — free
One-time email with the complete transcript and AI summary of this episode. No account needed.
One email, no spam. We’ll also show you what SignalCast does.
You just read a 3-minute summary of a 43-minute episode.
Get Eye on AI summarized like this every Monday — plus up to 2 more podcasts, free.
Pick Your Podcasts — FreeKeep Reading
More from Eye on AI
86% of What Coding Agents Do Is Just Reading — Not Solving | Alexander Whedon of Subquadratic
Sep 8 · 54 min
The AI Breakdown
The Right Way to Worry About AI
Aug 7
More from Eye on AI
From 10 Drones a Month to Nearly 100,000 — Inside Ukraine's Largest Drone Manufacturer | Marko Kushnir, General Cherry
Sep 3 · 38 min
20VC (20 Minute VC)
20VC: The Best AI Companies Have Unique Data Acquisition Strategies | Will Simile Kill Kalshi, Polymarkets and NASDAQ | How to Sign Fortune 500 Companies As Customers in Weeks with Joon Sung Park, Simile
Aug 1
Books, tools, and gear mentioned in this episode
SignalCast may earn commission on purchases via these links.
Tools
by AMD
“AMD's ROCm software stack is open-source, unlike NVIDIA's CUDA. This means language models can train directly on ROCm documentation and code, enabling AI agents to learn AMD-specific kernel writing more effectively.”
“Developers building on AMD hardware benefit from this transparency when using AI coding tools like Cursor to generate optimized kernels.”
More from Eye on AI
We summarize every new episode. Want them in your inbox?
86% of What Coding Agents Do Is Just Reading — Not Solving | Alexander Whedon of Subquadratic
From 10 Drones a Month to Nearly 100,000 — Inside Ukraine's Largest Drone Manufacturer | Marko Kushnir, General Cherry
In 5 to 10 Years, Using Weapons Without AI Will Be Considered Unethical | Yaroslav Azhnyuk, The Fourth Law
Inside Ukraine's Azov Drone R&D: The Engineer Building AI Weapons 18 km From the Front Line | Alexander Palamarchuk
95% of AI Agent Projects Fail to Reach Production. Here's Why | Manoj Saxena, TrustWise
Similar Episodes
Related episodes from other podcasts
The AI Breakdown
Aug 7
The Right Way to Worry About AI
20VC (20 Minute VC)
Aug 1
20VC: The Best AI Companies Have Unique Data Acquisition Strategies | Will Simile Kill Kalshi, Polymarkets and NASDAQ | How to Sign Fortune 500 Companies As Customers in Weeks with Joon Sung Park, Simile
Beyond Biotech
Apr 30
How Epic Bio is leveraging CRISPR without cutting DNA
Practical AI
Apr 16
Open Source Self-Driving with Comma AI
Huberman Lab
Nov 17
How to Speak Clearly & With Confidence | Matt Abrahams
Explore Related Topics
This podcast is featured in Best AI Podcasts (2026) — ranked and reviewed with AI summaries.
Read this week's AI & Machine Learning Podcast Insights — cross-podcast analysis updated weekly.
You're clearly into Eye on AI.
Every Monday, we deliver AI summaries of the latest episodes from Eye on AI and 192+ other podcasts. Free for one show.
Start My Monday DigestNo credit card · Unsubscribe anytime