#323 David Ha: Why Model Merging Could Be the Next AI Breakthrough
Episode
57 min
Read time
2 min
Topics
Productivity, Investing, Startups
AI-Generated Summary
Key Takeaways
- ✓Model Merging Without Weights: Sakana AI's ABMCTS system, presented as a NeurIPS spotlight, combines closed proprietary models like OpenAI, Google, and DeepSeek without accessing their weights. Instead, it uses Monte Carlo tree search to prompt multiple models simultaneously, evaluates their responses, and iteratively refines the most promising conversational branches to achieve state-of-the-art results on benchmarks like ARC-AGI.
- ✓Quality Diversity Over Elitism: When running evolutionary search across thousands of agent-generated solutions, selecting only top-performing results causes premature local minima. Sakana AI's Shinkai Evolve framework instead combines high-performing solutions with low-scoring but highly novel ones, demonstrating dramatically improved sample efficiency — reaching optimal solutions significantly earlier than pure elitist selection strategies used in systems like Google's AlphaEvolve.
- ✓LLM-Squared for Algorithm Discovery: Sakana AI's DiscoPop paper demonstrated that frontier LLMs can generate thousands of candidate training algorithms for other LLMs, with evolutionary search selecting and breeding the best-performing ones. The resulting algorithm achieved state-of-the-art performance on LLM fine-tuning tasks, establishing a replicable pipeline where AI systems autonomously improve AI training efficiency without human-designed algorithmic proposals.
- ✓AI Scientist v2 with Tree-Based Idea Branching: Unlike v1, which required a user-supplied code template, AI Scientist v2 autonomously generates research ideas from a general prompt, branches into multiple idea variations using tree search, self-evaluates via a calibrated reviewer LLM, and produced three papers submitted to an ICLR workshop — one scoring above the acceptance threshold in a blind evaluation approved by ethics boards.
- ✓Noisy World Models Force Generalizable Agent Skills: Ha's earlier world model research found that higher-fidelity simulations make it easier for agents to exploit simulation bugs for unlimited scores rather than learning transferable skills. Deliberately introducing noise — widening the gap between simulated and real environments — forces agents to develop more robust, generalizable behaviors applicable in actual deployment conditions.
What It Covers
David Ha, co-founder of Sakana AI, explains how evolutionary algorithms combined with large language models can merge frontier AI models, generate novel scientific ideas, and potentially push beyond the boundaries of existing human knowledge through collective intelligence systems and open-ended search strategies.
Key Questions Answered
- •Model Merging Without Weights: Sakana AI's ABMCTS system, presented as a NeurIPS spotlight, combines closed proprietary models like OpenAI, Google, and DeepSeek without accessing their weights. Instead, it uses Monte Carlo tree search to prompt multiple models simultaneously, evaluates their responses, and iteratively refines the most promising conversational branches to achieve state-of-the-art results on benchmarks like ARC-AGI.
- •Quality Diversity Over Elitism: When running evolutionary search across thousands of agent-generated solutions, selecting only top-performing results causes premature local minima. Sakana AI's Shinkai Evolve framework instead combines high-performing solutions with low-scoring but highly novel ones, demonstrating dramatically improved sample efficiency — reaching optimal solutions significantly earlier than pure elitist selection strategies used in systems like Google's AlphaEvolve.
- •LLM-Squared for Algorithm Discovery: Sakana AI's DiscoPop paper demonstrated that frontier LLMs can generate thousands of candidate training algorithms for other LLMs, with evolutionary search selecting and breeding the best-performing ones. The resulting algorithm achieved state-of-the-art performance on LLM fine-tuning tasks, establishing a replicable pipeline where AI systems autonomously improve AI training efficiency without human-designed algorithmic proposals.
- •AI Scientist v2 with Tree-Based Idea Branching: Unlike v1, which required a user-supplied code template, AI Scientist v2 autonomously generates research ideas from a general prompt, branches into multiple idea variations using tree search, self-evaluates via a calibrated reviewer LLM, and produced three papers submitted to an ICLR workshop — one scoring above the acceptance threshold in a blind evaluation approved by ethics boards.
- •Noisy World Models Force Generalizable Agent Skills: Ha's earlier world model research found that higher-fidelity simulations make it easier for agents to exploit simulation bugs for unlimited scores rather than learning transferable skills. Deliberately introducing noise — widening the gap between simulated and real environments — forces agents to develop more robust, generalizable behaviors applicable in actual deployment conditions.
Notable Moment
Ha revealed that making a world model more realistic and detailed can backfire: agents exploit simulation imperfections to achieve artificially high scores without learning real skills. A noisier, less perfect simulation paradoxically produces agents with stronger, more transferable capabilities in real-world environments.
Episode Transcript
Let's say you have 10 agents and you give them the same problem, and then you come up with solutions. What are you combining at that point? And is it being done in pairs? I mean, does each offspring have two parents, or can you combine more than one? Our first paper on model merging is does require access to the weights of the model. It's an impractical assumption because, in reality, most of the frontier models are closed. And even if we did have access to the weights, they probably do not share architectures. This episode is brought to you by Tastytrade. On ION AI, we talk a lot about how artificial intelligence is changing how people analyze information, spot patterns, and make more informed decisions. Markets are no different. The edge increasingly comes from having the right tools, the right data, and the ability to understand risk clearly. That's one of the reasons I like what Tastytrade is building. With Tastytrade, you can trade stocks, options, futures, and crypto all in one platform with low commissions, including zero commissions on stocks and crypto so you keep more of what you earn. The platform is packed with advanced charting tools, back testing, strategy selection, and risk analysis tools that help you think in probabilities rather than guesses. They've also introduced an AI powered search feature that can help you discover symbols aligned with your interests, which is a smart way to explore markets more intentionally. For active traders, there are tools like active trader mode, one click trading, and smart order tracking. And if you're still learning, Tastytrade offers dozens of free educational courses plus live support from their trade desk reps during trading hours. If you're serious about trading in a world increasingly shaped by technology, check out Tastytrade. Visit tastytrade.com to start your trading journey today. I'm going to myself. Tastytrade Inc is a registered broker dealer and member of FINRA, NFA, and SIPC. I'm interested in, evolution neuroevolution, which and you're coauthor on a book that just came out. I've talked to Risto and Sebastian. But I'm I'm interested, but I don't wanna talk about the book so much. I wanna talk about, neuroevolution or evolutionary strategies and what you're doing at, Sakana or Sakana, in in those realms. And it it seems like you've got a few layers. I mean, you had the model merging, which we can talk about. And, and then, you know, and evolutionary AI generally. But then AI, a scientist, takes those principles and applies it to research. So can we start by talking about, evolutionary AI or or, evolutionary strategies and and your work in artificial life, which is part of all of that and and and how you came to that area, particularly since you you came from finance, which doesn't sound related. Yeah. Oh, it it is quite related, I think. You know? I I feel like having lived went through the the financial crisis in 2008, it …
Get the full transcript (8,318 words) + summary by email — free
One-time email with the complete transcript and AI summary of this episode. No account needed.
One email, no spam. We’ll also show you what SignalCast does.
You just read a 3-minute summary of a 54-minute episode.
Get Eye on AI summarized like this every Monday — plus up to 2 more podcasts, free.
Pick Your Podcasts — FreeKeep Reading
More from Eye on AI
86% of What Coding Agents Do Is Just Reading — Not Solving | Alexander Whedon of Subquadratic
Sep 8 · 54 min
The Founders Podcast
#432 The Mind of Napoleon
Sep 5
More from Eye on AI
From 10 Drones a Month to Nearly 100,000 — Inside Ukraine's Largest Drone Manufacturer | Marko Kushnir, General Cherry
Sep 3 · 38 min
Odd Lots
Rory Johnston on Why His $200 Oil Prediction Didn't Turn Out Right
Jun 26
Books, tools, and gear mentioned in this episode
SignalCast may earn commission on purchases via these links.
Tools
- Shinkai EvolveBy guest
by Sakana AI
“Sakana AI's Shinkai Evolve framework instead combines high-performing solutions with low-scoring but highly novel ones, demonstrating dramatically improved sample efficiency.”
- AI Scientist v2By guest
by Sakana AI
“AI Scientist v2 autonomously generates research ideas from a general prompt, branches into multiple idea variations using tree search, self-evaluates via a calibrated reviewer LLM, and produced three papers submitted to an ICLR workshop.”
- ABMCTSBy guest
by Sakana AI
“Sakana AI's ABMCTS system, presented as a NeurIPS spotlight, combines closed proprietary models like OpenAI, Google, and DeepSeek without accessing their weights.”
company
- Sakana AIBy guest
“David Ha, co-founder of Sakana AI, explains how evolutionary algorithms combined with large language models can merge frontier AI models.”
“Sakana AI's ABMCTS system combines closed proprietary models like OpenAI, Google, and DeepSeek without accessing their weights.”
“Sakana AI's ABMCTS system combines closed proprietary models like OpenAI, Google, and DeepSeek without accessing their weights.”
“Sakana AI's ABMCTS system combines closed proprietary models like OpenAI, Google, and DeepSeek without accessing their weights.”
More from Eye on AI
We summarize every new episode. Want them in your inbox?
86% of What Coding Agents Do Is Just Reading — Not Solving | Alexander Whedon of Subquadratic
From 10 Drones a Month to Nearly 100,000 — Inside Ukraine's Largest Drone Manufacturer | Marko Kushnir, General Cherry
In 5 to 10 Years, Using Weapons Without AI Will Be Considered Unethical | Yaroslav Azhnyuk, The Fourth Law
Inside Ukraine's Azov Drone R&D: The Engineer Building AI Weapons 18 km From the Front Line | Alexander Palamarchuk
95% of AI Agent Projects Fail to Reach Production. Here's Why | Manoj Saxena, TrustWise
Similar Episodes
Related episodes from other podcasts
The Founders Podcast
Sep 5
#432 The Mind of Napoleon
Odd Lots
Jun 26
Rory Johnston on Why His $200 Oil Prediction Didn't Turn Out Right
Investing for Beginners
May 18
How AI Is Changing Investing— with David Trainer
In Good Company with Nicolai Tangen
May 15
HIGHLIGHTS: Aliko Dangote - Founder and CEO of the Dangote Group
Odd Lots
Apr 21
Brad Jacobs on His Big Bet on Building Insulation
Explore Related Topics
This podcast is featured in Best AI Podcasts (2026) — ranked and reviewed with AI summaries.
Read this week's Investing & Markets Podcast Insights — cross-podcast analysis updated weekly.
You're clearly into Eye on AI.
Every Monday, we deliver AI summaries of the latest episodes from Eye on AI and 192+ other podcasts. Free for one show.
Start My Monday DigestNo credit card · Unsubscribe anytime