How Afraid of the A.I. Apocalypse Should We Be?

Shortly after JAD GPT was released, it felt like all anyone could talk about, at least if you were in AI circles, was the risk of rogue AI. You began to hear a lot of talk about researchers discussing their their p doom, the probability they gave to AI destroying or fundamentally displacing humanity. In May 2023, a group of the world's top AI figures, including Sam Altman and Bill Gates and Geoffrey Hinton, signed on to a public statement that said, mitigating the risk of extinction from AI should be a global priority alongside other societal scale risks, such as pandemics and nuclear war. And then nothing really happened. The signatories, or many of them at least, of that letter, raced ahead releasing new models and new capabilities. Your share price, your valuation became a whole lot more important in Silicon Valley than your PDoom, but not for everyone. Eliezer Yudkowsky was one of the earliest voices warning loudly about the existential risk posed by AI. He was making this argument back in the February, many years before Chatt GPT hit the scene. He's been in this community of AI researchers, influencing many of the people who build these systems. In some cases, inspiring them to get into this work in the first place, yet unable to convince him to stop building the technology he thinks will destroy humanity. He just released a new book, co written with Nate Suarez, called If Anyone Builds It, Everyone Dies. Now he's trying to make this argument to the public, a last ditch effort to, at least in his view, rouse us to save ourselves before it is too late. I come into this conversation taking AI risk seriously. If we're going to invent superintelligence, it is probably gonna have some implications for us, but also being skeptical of the scenarios I often see by which these takeovers are said to happen. So I wanted to hear what the godfather of these arguments would have to say. As always, my email, Ezra Klein show at n y times dot com. Eliezer Yudkowsky, welcome to the show. Thanks for having me. So I wanted to start with something that you say early in the book. This is not a technology that we craft. It's something that we grow. What do you mean by that? Well, it's the difference between a planter and the plant that grows up within it. We craft the AI growing technology and then the technology grows the AI, you know, like central original large language models before doing a bunch of clever stuff that they're doing today. The central question is, what probability have you assigned to the true next word of the text? As we tweak each of these billions of parameters well, actually, it was just like millions back then. As we tweak each of these millions of parameters, does the probability assigned to the correct token go up? And this is what teaches the AI to …