How World Foundation Models Will Advance Physical AI With NVIDIA’s Ming-Yu Liu - Ep. 240

What It Covers

NVIDIA's Mingyu Liu explains world foundation models, deep learning systems that simulate physics and predict futures to train and verify physical AI agents like humanoid robots and autonomous vehicles through the new Cosmos development platform.

Key Questions Answered

• •Three verification applications: World models test policy checkpoints in simulation before physical deployment, initialize policy models with pretrained physics understanding to reduce training data needs, and enable real-time future rollout simulation for decision-making before robots act.
• •Cosmos platform components: NVIDIA releases open-weight models free for commercial use, including diffusion-based and autoregressive architectures, video tokenizers for transformer processing, post-training scripts for custom camera configurations, and video curation toolkits leveraging GPU-accelerated libraries for data processing.
• •Autoregressive versus diffusion tradeoffs: Autoregressive models predict tokens sequentially like GPT, run faster due to existing optimizations, but struggle with video compression accuracy. Diffusion models generate token sets simultaneously, produce more coherent outputs with better quality, but require different integration approaches.
• •Early industry adoption focus: Self-driving car companies and humanoid robot manufacturers including One X, Huobi, Li Auto, and others partner with NVIDIA to apply world models for testing scenarios difficult to replicate physically and validating agent behavior across diverse environments.