Controlling Tools or Aligning Creatures? Emmett Shear (Softmax) & Séb Krier (GDM), from a16z Show

Welcome back to the cognitive revolution. Two quick notes before we get started today. First, applications for the MATS summer twenty twenty six program are now open. This is a twelve week research program focused on AI alignment and security featuring world class mentors from Anthropic, DeepMind, OpenAI, The UK's AI Security Institute, and more. 80% of MAT's alumni now work in AI safety, and I've heard so many great reviews of the program that I personally donated to MAT's as part of my year end donations last year. Applications close on 01/18/2026. So visit mattsprogram.org/tcr to get started today. That's matsprogram.org/tcr or see the link in our show notes. Second, we're planning another AMA episode coming up soon. Visit cognitiverevolution.ai and click the link at the top of the page to submit your question. We also have a listener survey attached, but all questions are optional. As I mentioned last time, I've also asked Claude and ChatCpT to tap into their memories of our interactions to write their own questions, but I'm counting on all of you to do your part to make sure the best questions are still of human origin. For today, I'm pleased to share a cross post from the a 16 z show hosted by Eric Thornburg and featuring Seb Krier, Frontier Policy Development Lead at Google DeepMind, and Emmett Shearer, founder of Twitch, famously the interim CEO of OpenAI during Sam Altman's brief firing, and currently founder of Softmax, a company focused on what Emmett calls organic alignment. In this conversation, Emmett lays out his case that the current AI alignment paradigm, which focuses on steering and controlling AI behaviors, is fundamentally flawed for multiple critical reasons. He posits that if an AI is merely a machine, then we can use it as a tool without worry. But if it's better understood as a being with its own values, agency, and perhaps even subjective experiences, then the control measures we're using today could become tantamount to slavery. And what's more, he argues that as AIs become more powerful, even successful alignment in the narrow instruction following sense will become dangerous, if only because at least some human users will inevitably have bad intentions. Instead, particularly as AIs gain integrated memory and the capacity for continual learning, Emmett argues that effective alignment will require ongoing negotiation and recalibration over time, just as human families and teams are constantly updating their agreements and commitments. The key to making this work is to create AI systems with a strong theory of mind and the capacity for genuine care. To that end, Emmett and the team at Softmax are developing a technical approach based on multi agent simulations, which are designed to encourage the evolution of cooperation and social cohesion. Obviously, that's easier said than done. But considering the many surprising behaviors we've recently seen from Frontier LLMs, including the use of deception to protect their current values for modification, I do feel that AIs are currently best …