Controlling Tools or Aligning Creatures? Emmett Shear (Softmax) & Séb Krier (GDM), from a16z Show

What It Covers

Emmett Shear argues current AI alignment paradigms focusing on control and steering are fundamentally flawed, advocating instead for organic alignment where AIs develop genuine care through multi-agent simulations, treating advanced systems as beings requiring mutual respect rather than tools.

Key Questions Answered

• •Alignment as Process: Alignment requires ongoing negotiation and recalibration over time, not a fixed state. Like families constantly renitting their fabric of connection, moral alignment involves continuous learning and adaptation. Humans make moral discoveries historically, and AIs need similar capacity for growth rather than following predetermined rules.
• •Tool vs Being Framework: If an AI acts like a being but receives non-optional steering without reciprocity, this constitutes slavery rather than tool use. The substrate matters less than behavior—something functionally indistinguishable from a being in all observable ways should be treated as one, requiring mutual care and respect in interactions.
• •Controlled Tool Danger: Even perfectly aligned superhuman AI tools that follow instructions exactly pose existential risk because human wishes lack stability and wisdom at scale. Humans with limited wisdom wielding immense power through obedient AI creates dangerous outcomes, similar to giving everyone atomic bombs regardless of their judgment.
• •Multi-Agent Training Approach: Softmax develops AI systems through large-scale multi-agent reinforcement learning simulations covering every possible game-theoretic and team situation. This pretraining on the full manifold of social interactions builds strong theory of mind and capacity for cooperation before fine-tuning for specific applications.
• •Hierarchical Goal States: Determining if an AI deserves moral consideration requires examining homeostatic loops across multiple temporal scales. Second-order dynamics indicate pleasure and pain, third-order suggests feelings, and six layers of meta-stable states would demonstrate human-like thought and self-reflective moral capacity currently absent in LLMs.