Are We About to Lose Control of AI? | AI Reality Check

Last week, Anthropic released a report with a scary sounding title, when AI builds itself, and it came accompanied by a scary animation that shows machines replicating themselves exponentially like cells in a petri dish. Now, the body of the report itself keeps these dark vibes going. I wanna read you some actual quotes here from the intro to the report. They say, for most of AI's history, humans drove every step in its development cycle, but at Anthropic, we are delegating a growing share of AI development to AI systems themselves, which is speeding up our work. Taken far enough and given enough compute, this trend points to an AI system capable of fully autonomous designing and developing its own successor. This is called recursive self improvement. We are not there yet, and recursive self improvement is not inevitable, but it could come sooner than most institutions are prepared for. A little bit later, they then add, AI that can build itself would be a major development in the history of technology, one that could bring enormous good for the world in science, health care, and beyond. But full recursive self improvement also might increase the risks of human humans losing control over AI systems. Now, if you look at the the headlines generated in response to this report, most of them focused on a section of the report that, seemed to call for a worldwide pause on AI development to avoid the scenario of humans losing control. But if you read that section closer, you see that's not actually what the report says. Here's the actual wording. If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think they would likely be a good thing. But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe. So in other words, Anthropic is saying, we'll only slow down if everyone else around the world does too. Otherwise, we have no choice but to continue with our efforts at full speed. Now, look, this is pretty grim stuff. Is basically saying that we are potentially hurtling towards a world of AI that improves itself rapidly until we lose control over it, and they're saying there is nothing that we can do about it except maybe continuing to publish solemn reports with fancy animations, and I guess also, cash in on our stock options after an IPO. Alright. So here's the key question. Are these fears justified? Well, it's Thursday, which means it's time for an AI reality check episode of the show, which is a good opportunity to go looking for some measured answers, and that is exactly what we are going to do. As always, I'm Cal Newport, and this is Deep Questions, the show for people seeking depth in a distracted world. Alright. So how, how much should we actually be afraid of recursive self improvement? …