Success without Dignity? Nathan finds Hope Amidst Chaos, from The Intelligence Horizon Podcast

Hello, and welcome back to the Cognitive Revolution. Today, I'm sharing a special cross post from my recent appearance on the Intelligence Horizon podcast with hosts Owen Zhang and Will Sanok Dufalo. Owen and Will will soon be graduating from Yale College. And as you'll hear, they've clearly spent much of their senior year thinking deeply about the current state of AI, where we're headed, and what it means for all of us. And I was really impressed, not only with the quality of their questions, but their ability to challenge me with follow ups that effectively steel man the most relevant counter arguments. We start with the fact that while AI timelines have compressed dramatically over the last five years, genuine experts still disagree radically on critical questions. Having established what I hope is appropriate epistemic humility, I then go on to call it how I see it. In short, the singularity is near. Interpretability science proves that AIs are developing increasingly sophisticated world models. And with reinforcement learning scaling now clearly working, AIs are no longer simply imitating humans and likely won't be limited by what we know for much longer. The potential upside of this is, of course, incredible. The value that I've got from using AI to navigate what humans have discovered about how cancer works and how to treat it has been invaluable. And the prospect that we might cure the majority of human diseases in just the next decade or so is obviously extremely exciting. That said, the risks are also very real, and they will remain serious for as long as we lack a solid understanding of how AI's work and why they do what they do. My PDoom remains somewhere in the ten to ninety percent range. And yet, at the same time, I've become at least a little bit more optimistic that we might actually build robustly good AIs. Because scaling laws at least seem to imply that powerful AIs can only be created with massive resources. The three companies competing at the frontier today are at least reasonably responsible actors, and our best alignment techniques are working better than I had expected. Given these fundamentals, it seems at least plausible that a defense in-depth strategy, which combines techniques like Goodfire's intentional design, Redwood's AI control, improved cybersecurity through formal verification of software, and various forms of pandemic preparedness could collectively be enough to keep society on the rails. We touch on a number of other topics as well, including The US China rivalry and why, especially in the context of the Department of War's recent attack on anthropic, which I'm sad to say has us looking more and more like China all the time. I would rather bet on figuring out a way to cooperate with our fellow humans than bet everything on AI researchers ability to steer AI advances in a way that will ultimately work for us humans. I appreciate Owen and Will for allowing me to cross …