#490 – State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI
Read time
2 min
Topics
Productivity, Remote Work, Investing
AI-Generated Summary
Key Takeaways
- ✓Chinese Open Model Strategy: DeepSeek trained their model for approximately $5 million at cloud rates, while Olmo 3 spent around $2 million for cluster rental including engineering issues. Chinese companies release open weight models primarily to gain international distribution where users won't pay for API subscriptions to Chinese services due to security concerns, creating influence through free access rather than direct revenue.
- ✓Pretraining Cost Economics: Training costs represent a small fraction compared to serving costs for hundreds of millions of users. A thousand GPU rental costs roughly $100 daily, while frontier labs operate millions of GPUs. Companies now optimize for smaller, more efficient models because recurring serving costs reach billions of dollars, making model size reduction more valuable than raw capability gains through larger pretraining runs.
- ✓Reinforcement Learning Scaling: Post-training through reinforcement learning with verifiable rewards unlocked major capability gains in 2025, enabling tool use, multi-step reasoning, and better code generation. AI2's November model used five days of RL training, then ran another 3.5 weeks in December for notable improvements, demonstrating that RL scaling provides more cost-effective intelligence gains than expanding pretraining compute at current model sizes.
- ✓Data Quality Over Quantity: Olmo 3 achieved better performance with less training data than predecessors by focusing on data quality and mixing ratios. Labs train classifiers on samples from different sources like GitHub, Stack Exchange, and Wikipedia, then use linear regression to determine optimal dataset composition based on target evaluations. Synthetic data includes OCR extraction from PDFs yielding trillions of tokens, not just AI-generated content.
- ✓Architecture Convergence: Modern frontier models remain fundamentally similar to GPT-2 architecture with incremental tweaks like mixture of experts, multi-head latent attention, and group query attention. The differentiation comes from systems optimization including FP8 and FP4 training, distributed compute management across 10,000-100,000 GPUs, and post-training algorithms rather than novel architectural paradigms. Converting between model architectures requires only adding specific components to the base transformer.
What It Covers
Sebastian Raschka and Nathan Lambert analyze the 2025 AI landscape following DeepSeek's breakthrough, comparing Chinese and US model development, examining scaling laws across pretraining and inference, discussing open versus closed models, and evaluating the technical architecture evolution from GPT-2 to current frontier models like Claude Opus 4.5 and GPT-5.
Key Questions Answered
- •Chinese Open Model Strategy: DeepSeek trained their model for approximately $5 million at cloud rates, while Olmo 3 spent around $2 million for cluster rental including engineering issues. Chinese companies release open weight models primarily to gain international distribution where users won't pay for API subscriptions to Chinese services due to security concerns, creating influence through free access rather than direct revenue.
- •Pretraining Cost Economics: Training costs represent a small fraction compared to serving costs for hundreds of millions of users. A thousand GPU rental costs roughly $100 daily, while frontier labs operate millions of GPUs. Companies now optimize for smaller, more efficient models because recurring serving costs reach billions of dollars, making model size reduction more valuable than raw capability gains through larger pretraining runs.
- •Reinforcement Learning Scaling: Post-training through reinforcement learning with verifiable rewards unlocked major capability gains in 2025, enabling tool use, multi-step reasoning, and better code generation. AI2's November model used five days of RL training, then ran another 3.5 weeks in December for notable improvements, demonstrating that RL scaling provides more cost-effective intelligence gains than expanding pretraining compute at current model sizes.
- •Data Quality Over Quantity: Olmo 3 achieved better performance with less training data than predecessors by focusing on data quality and mixing ratios. Labs train classifiers on samples from different sources like GitHub, Stack Exchange, and Wikipedia, then use linear regression to determine optimal dataset composition based on target evaluations. Synthetic data includes OCR extraction from PDFs yielding trillions of tokens, not just AI-generated content.
- •Architecture Convergence: Modern frontier models remain fundamentally similar to GPT-2 architecture with incremental tweaks like mixture of experts, multi-head latent attention, and group query attention. The differentiation comes from systems optimization including FP8 and FP4 training, distributed compute management across 10,000-100,000 GPUs, and post-training algorithms rather than novel architectural paradigms. Converting between model architectures requires only adding specific components to the base transformer.
Notable Moment
Nathan Lambert reveals he exclusively uses extended thinking modes across multiple models, running five simultaneous GPT-5 pro queries for different research tasks like finding papers or checking equations. He finds the non-thinking GPT-5 model has higher error rates and poor tone, refusing to use it despite speed advantages, demonstrating how power users prioritize marginal intelligence gains over convenience.
Episode Transcript
The following is a conversation all about the state of the art in artificial intelligence, including some of the exciting technical breakthroughs and developments in AI that happened over the past year and some of the interesting things we think might happen this upcoming year. At times, it does get super technical, but we do try to make sure that it remains accessible to folks outside the field without ever dumbing it down. It is a great honor and pleasure to be able to do this kind of episode with two of my favorite people in the AI community, Sebastian Raschka and Nathan Lambert. They are both widely respected machine learning researchers and engineers who also happen to be great communicators, educators, writers, and Twitterers, ex posters. Sebastian is the author of two books I highly recommend for beginners and experts alike. First is build a large language model from scratch and build a reasoning model from scratch. I truly believe in the machine learning computer science world, the best way to learn and understand something is to build it yourself from scratch. Nathan is the post training lead at the Allen Institute for AI and author of the definitive book on reinforcement learning from human feedback. Both of them have great x accounts, great substacks. Sebastian has courses on YouTube. Nathan has a podcast, and everyone should absolutely follow all of those. And now a quick few second mention of each sponsor. Check them out in the description or at lexfredeman.com/sponsors. It is, in fact, the best way to support this podcast. We got a bunch of great sponsors. Box for intelligent content management, Quo for your phone system, like calls, text, contacts for your business, Uplift desk, the desk I'm sitting behind, and my favorite office desk. Thin, for customer service AI agents, Shopify for selling stuff online, CodeRabbit for AI powered code review, element for electrolytes, and, of course, our longtime friend, perplexity. For curiosity driven knowledge exploration, choose wisely, my friends. And now onto the full add reads. I try to make them interesting, but if you do skip, please still check out the sponsors. I enjoy their stuff. Maybe you will too. To get in touch with me, for whatever reason, go to lexfreeman.com/contact. If you, can't tell, I'm trying to have a bit of a pep in my step at the moment because I had a long night, didn't get much sleep at all. So I am running on fumes, delirious, happy, unsure of what is reality and what is a dream. In fact, we could, right now, be living inside of a dream. I have been going through a lot. I have been working insane hours, so much going on. I'm so overwhelmed. Of course, as always, truly grateful and happy to be alive, but have not been able to publish as many episodes as I would like, so there's a bunch of sponsors we have to catch up on. Your support truly …
Get the full transcript (54,013 words) + summary by email — free
One-time email with the complete transcript and AI summary of this episode. No account needed.
One email, no spam. We’ll also show you what SignalCast does.
Get Lex Fridman Podcast summarized like this every Monday — plus up to 2 more podcasts, free.
Pick Your Podcasts — FreeKeep Reading
More from Lex Fridman Podcast
#501 – DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux
Aug 26 · 321 min
Latent Space
[LIVE] Anthropic Distillation & How Models Cheat (SWE-Bench Dead) | Nathan Lambert & Sebastian Raschka
Feb 26
More from Lex Fridman Podcast
#500 – Khabib Nurmagomedov: Dagestan, MMA, UFC, Islam, Conor, Fedor & Football
Aug 12
Software Engineering Daily
SED News: Restricted Models, IDE Wars, and the DeepMind Mafia
Jul 7
Books, tools, and gear mentioned in this episode
SignalCast may earn commission on purchases via these links. As an Amazon Associate, SignalCast earns from qualifying purchases.
Tools
“SPONSORS: Fin”
“SPONSORS: Box”
“SPONSORS: Quo”
“SPONSORS: Perplexity”
“SPONSORS: Shopify”
“SPONSORS: CodeRabbit”
Gear
Products
“evaluating the technical architecture evolution from GPT-2 to current frontier models like Claude Opus 4.5 and GPT-5”
More from Lex Fridman Podcast
We summarize every new episode. Want them in your inbox?
#501 – DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux
#500 – Khabib Nurmagomedov: Dagestan, MMA, UFC, Islam, Conor, Fedor & Football
#499 – Gary Gallagher: American Civil War, Slavery, Lincoln, Grant & Lee
#498 – Anthony Kaldellis: Roman Empire, Byzantine Empire, Rise & Fall of Empires
#497 – Biggest Mysteries in Physics: Antimatter, Dark Energy & ToE – Don Lincoln
Similar Episodes
Related episodes from other podcasts
Latent Space
Feb 26
[LIVE] Anthropic Distillation & How Models Cheat (SWE-Bench Dead) | Nathan Lambert & Sebastian Raschka
Software Engineering Daily
Jul 7
SED News: Restricted Models, IDE Wars, and the DeepMind Mafia
The TWIML AI Podcast
Feb 26
AI Trends 2026: OpenClaw Agents, Reasoning LLMs, and More with Sebastian Raschka - #762
The AI Breakdown
Aug 31
How to Navigate the Next Wave of AI Competition
a16z Podcast
Aug 22
Martin Casado on Where the Value Is Going in AI
Explore Related Topics
This podcast is featured in Best Tech Podcasts (2026) — ranked and reviewed with AI summaries.
Read this week's Investing & Markets Podcast Insights — cross-podcast analysis updated weekly.
You're clearly into Lex Fridman Podcast.
Every Monday, we deliver AI summaries of the latest episodes from Lex Fridman Podcast and 192+ other podcasts. Free for one show.
Start My Monday DigestNo credit card · Unsubscribe anytime