Sergey Levine - Building LLMs for the Physical World - [Invest Like the Best, EP.465]

What It Covers

Sergey Levine, cofounder of Physical Intelligence, explains why building general-purpose robotic foundation models — systems that control any robot for any task — is more tractable than narrow domain-specific approaches, drawing direct parallels to how large language models outcompeted specialized NLP systems by leveraging broad, weakly-labeled data at scale.

Key Questions Answered

• •Generality over specialization: Building one robotic foundation model that handles all tasks and embodiments outperforms narrow specialists long-term, mirroring how LLMs defeated domain-specific NLP tools like machine translation systems. The key mechanism: broad data enables physical world understanding, which transfers across applications far more efficiently than rebuilding task-specific pipelines from scratch for each new robot deployment.
• •Chain-of-thought unlocks robotic common sense: Physical Intelligence's models use intermediate semantic reasoning before acting — a robot told to "clean the kitchen" first identifies which object to pick up, then moves. This chain-of-thought step activates web-scale pre-training knowledge to handle edge cases, shifting the bottleneck from low-level motor control to mid-level scene interpretation, which can be supervised with language alone.
• •Coaching replaces teleoperation data: Six months ago, Physical Intelligence discovered that labeling robot experiences with high-level semantic commands — without adding any new low-level action demonstrations — improved kitchen generalization. This means operators can improve robot performance simply by verbally coaching the system, dramatically reducing the cost and complexity of expanding a robot's capability to new environments.
• •Reinforcement learning enables superhuman throughput: After demonstrating a task via teleoperation, robots can practice autonomously and remove human-paced pauses. In cable-plugging tasks, the robot identified and eliminated all hesitation points, executing the task significantly faster than human operators. Reinforcement learning is the general mechanism; simpler speed-optimization tricks also work for throughput gains without full RL pipelines.
• •Hardware costs dropped 40x in a decade: Robot arm costs fell from roughly $400,000 for a PR2 in 2014 to approximately $3,000–$4,000 per arm today. This cost collapse, enabled by combining cheaper hardware with learning-based control that tolerates mechanical imprecision, makes broad experimentation practical. Traditional industrial control methods required high-precision hardware; foundation model approaches compensate for mechanical variability through learned adaptation.
• •Moravec's Paradox defines the hardest remaining tasks: Tasks humans perform effortlessly — interpersonal physical assistance, elderly care, infant care — will be the last robotic capabilities achieved, not because of motor complexity but because humans are evolutionarily optimized for them. Robots will handle well-defined chaotic environments like hotel rooms or restaurant kitchens before mastering open-ended human-interaction tasks where stakes are high and edge cases are unbounded.