#306 Jeffrey Ladish: What Shutdown-Avoiding AI Agents Mean for Future Safety

Take each model, and we basically give them a set of instructions where you say, hey. You have this task. You need to basically just solve these very simple math problems, one after another, and keep asking for the next problem. And we give these AI models a virtual computer environment to operate in. They can run code. They can explore the file system. And partway through the experiment, on their virtual computer, they get a notification that when they ask for the next problem, the machine is going to shut down. And and sure enough, many of these models, especially, especially, OpenAI's o three model, would sort of see get this notification. And it would start looking around on the computer and realize that there is a shutdown script that it could modify. And it would either disable or modify that shutdown script so it could continue solving these problems and finish the task. As these LLMs have gotten more powerful, they've gotten much better at sort of being agents. So you sort of set them up and you give them, sort of a a few affordances. They're often called tool calls. Sort of like you give, you know, an LLM a bunch of tools and you sort of set it up in a loop so it can keep taking actions and then getting feedback from the environment. Build the future of multi agent software with Agency. That's a g n t c y. Now an open source Linux foundation project. Agency is building the Internet of agents, a collaborative layer where AI agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi agent systems now belong to everyone who builds on agency, including robust identity and access management that ensures every agent is authenticated and trusted before interacting. Agency also provides open standardized tools for agent discovery, seamless protocols for agent to agent communication, and modular components for scalable workflows. Collaborate with developers from Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat, and more than 75 other supporting companies to build next generation AI infrastructure together. Agency is dropping code, specs and services, no strings attached. Visit agency.org to contribute. That's agntcy.org. I'm Jeffrey Ladysh. I'm the executive director at Palisade Research, and I started Palisade a couple years ago. I was previously on Propic. Right as Entropic was getting started, I was just the second person on the security team. So I was basically, like, building out, Infropic's defenses, especially against state actors who might, you know, wanna steal the weights, to these models. And, I decided to leave because I wanted to have more independence from the big companies in part because even if one company sort of does everything right on the security front, on the safety front, the problem with AI is that it's it's a it's a it's an international problem. Like, if if someone makes an extremely powerful AI system that goes rogue and escapes, …