How World Foundation Models Will Advance Physical AI With NVIDIA’s Ming-Yu Liu - Ep. 240

Hello, and welcome to the NVIDIA AI podcast. I'm your host, Noah Kravitz. NVIDIA CEO Jensen Huang recently keynoted the CES Consumer Electronics Show conference in Las Vegas, Nevada. Amongst the many exciting announcements Jensen talked about was NVIDIA Cosmos. Cosmos is a development platform for world foundation models, which I think we're all gonna be talking a lot about in the coming months and years. What is a world foundation model? Well, thankfully, we've got an expert here to tell us all about it. Mingyu Liu is vice president of research at NVIDIA. He's also an I triple e fellow, and he's here to tell us all about world foundation models, how they work, what they mean, and why we should care about them going forward. So without further ado, Mingyu, thank you so much for joining the NVIDIA AI podcast, and welcome. It's great to be here. So let's start with the basics, if you would. What is a world foundation model? Sure. So world foundation models are deep learning based space time visual simulator that can help us look into the future. It can simulate physics. It can simulate people's intentions and activities. It's like data science of AI. Imagine many different environments and can simulate the future. So we can make good decision based on the simulation. We can leverage work foundation models, imagination, and simulation capability to help train physical AI agents. We can also leverage this capability to help the agent make good decision during the inference time. We can generate a virtual world based on text prompts, image prompts, video prompts, action prompts, and the layer combinations. So we call it a a world foundation model because it can generate many different worlds and also because it can be customized through different physical AI setups Right. To become a customized world model. Right? So different physical AI have different number of cameras in different locations. So we want the world foundation model to be customizable for different physical AI setups so they can use in their settings. So I wanna ask you kind of how a world model is similar or different to an LLM and other types of models. But I think first, I wanna back up a step and ask you, how is a world model similar or different to a model that generates video? Because my understanding and please correct me where I'm wrong. My understanding is that you can prompt a world model to generate a video, but that video is generated based on the things you were talking about based on, understanding of, you know, physics and other things in the physical world, and it's a different process. So I don't know what the best way is to kind of unpack it for the listeners, but one place to start might be, how does a world model differentiate from an LLM or a generative AI video model? So world model is different to LN, in the sense that …