Richard Sutton – Father of RL thinks LLMs are a dead end

What It Covers

Richard Sutton, Turing Award winner and reinforcement learning pioneer, argues that large language models represent a dead end for AI progress because they lack goals, cannot learn from experience, and fundamentally mimic human behavior rather than understand the world.

Key Questions Answered

• •Experiential Learning vs Imitation: Reinforcement learning enables agents to learn from direct experience through action-sensation-reward cycles, building testable world models with ground truth feedback. LLMs only mimic human responses without predictions about actual world consequences or ability to adjust based on unexpected outcomes, lacking the fundamental learning mechanism animals use.
• •Four Component Agent Architecture: Effective AI agents require four distinct parts: a policy determining actions in situations, a value function using TD learning to assess progress, perception constructing state representation, and a transition model predicting world consequences. This transition model learns richly from all sensations, not just rewards, enabling continual adaptation.
• •Generalization Through Architecture: Deep learning systems fail at generalization because gradient descent solves training problems without ensuring good transfer to new states. Current systems only generalize well when researchers manually sculpt representations. Catastrophic interference when training on new data demonstrates fundamentally poor generalization, requiring new automated techniques to promote positive transfer across states.
• •Digital Intelligence Succession: Four inevitable factors drive AI succession: no unified global governance exists to coordinate development, researchers will eventually solve intelligence, capabilities will exceed human level, and intelligent systems naturally accumulate resources over time. This represents a major universal transition from biological replication to designed entities that understand and modify their own intelligence.
• •Cultural Evolution in AI Systems: When AI agents gain sufficient compute, they face a critical choice between self-improvement or spawning copies to learn diverse topics and reintegrate knowledge. The key challenge becomes cybersecurity against corruption, as incorporating external knowledge from spawned copies could introduce hidden goals or viruses that fundamentally alter the original agent's thinking and objectives.