What are the key takeaways from this Software Engineering Daily episode?

Key insights include: **Two-Tower Architecture:** Physical AI models split into a low-frequency reasoning/planning tower and a high-frequency action tower that can control 30–40 motors simultaneously at millisecond intervals. Both towers share a latent space for high-bandwidth context exchange, avoiding the bottleneck of decoding and re-encoding outputs between separate models running on distributed hardware.; **Three Distinct Compute Requirements:** Building physical AI requires three separate computing environments: a GPU cluster for model training, a ray-tracing GPU for physics simulation and reinforcement learning validation, and an edge device for deployment. Developers cannot use one unified cluster as with LLMs — each stage demands fundamentally different hardware architectures and workflows.; **Real-to-Sim Data Pipeline:** NVIDIA built a dedicated data curation tool called Curator specifically because standard LLM training pipelines cannot handle multimodal video, sensor, and motor data at the required throughput. Capturing real-world environments with cameras and LiDAR, then converting them into editable 3D simulations, allows augmentation of lighting, road width, and object placement for post-training.

What did Amit Goel discuss on Software Engineering Daily?

NVIDIA's Amit Goel explains physical AI to software developer Matt Merrill, covering how world models process multimodal sensor data to control robots and autonomous vehicles, the two-tower architecture separating reasoning from action, and how developers can enter robotics for under $1,000 using open-source tools and simulation environments. Key topics include: **Two-Tower Architecture:** Physical AI models split into a low-frequency reasoning/planning tower and a high-frequency action tower that can control 30–40 motors simultaneously at millisecond intervals. Both towers share a latent space for high-bandwidth context exchange, avoiding the bottleneck of decoding and re-encoding outputs between separate models running on distributed hardware.; **Three Distinct Compute Requirements:** Building physical AI requires three separate computing environments: a GPU cluster for model training, a ray-tracing GPU for physics simulation and reinforcement learning validation, and an edge device for deployment. Developers cannot use one unified cluster as with LLMs — each stage demands fundamentally different hardware architectures and workflows..

How long is this episode of Software Engineering Daily?

This episode is 56 minutes long. SignalCast provides an AI-generated summary so you can get the key insights in about 3 minutes.

Software Engineering Daily

Physical AI at NVIDIA

October 20, 2026

56 min episode · 2 min read

Amit Goel

Episode

56 min

Read time

2 min

Topics

Remote Work, Startups, Fundraising & VC

AI-Generated Summary

Published Oct 20, 2026

Key Takeaways

✓Two-Tower Architecture: Physical AI models split into a low-frequency reasoning/planning tower and a high-frequency action tower that can control 30–40 motors simultaneously at millisecond intervals. Both towers share a latent space for high-bandwidth context exchange, avoiding the bottleneck of decoding and re-encoding outputs between separate models running on distributed hardware.
✓Three Distinct Compute Requirements: Building physical AI requires three separate computing environments: a GPU cluster for model training, a ray-tracing GPU for physics simulation and reinforcement learning validation, and an edge device for deployment. Developers cannot use one unified cluster as with LLMs — each stage demands fundamentally different hardware architectures and workflows.
✓Real-to-Sim Data Pipeline: NVIDIA built a dedicated data curation tool called Curator specifically because standard LLM training pipelines cannot handle multimodal video, sensor, and motor data at the required throughput. Capturing real-world environments with cameras and LiDAR, then converting them into editable 3D simulations, allows augmentation of lighting, road width, and object placement for post-training.
✓Cosmos 3 Edge Runs On-Robot: NVIDIA's Cosmos 3 Edge is a 4-billion-parameter world model optimized to run in real time on the Jetson Thor — a passport-sized computer delivering 2,000 teraops of compute. This enables on-device planning and counterfactual reasoning, where the model evaluates up to 10 possible future scenarios and selects the safest path without cloud round-trips.
✓Entry Point for Software Developers: Developers can begin physical AI experimentation using a laptop with a 4090 GPU plus a simulator at no hardware cost. Adding a physical robotic arm — such as the SO-101, available as a 3D-printable kit — brings total startup cost to roughly $1,000. Hugging Face hosts pre-trained checkpoints matched to specific hardware, eliminating the need to customize models from scratch.

What It Covers

NVIDIA's Amit Goel explains physical AI to software developer Matt Merrill, covering how world models process multimodal sensor data to control robots and autonomous vehicles, the two-tower architecture separating reasoning from action, and how developers can enter robotics for under $1,000 using open-source tools and simulation environments.

Key Questions Answered

•Two-Tower Architecture: Physical AI models split into a low-frequency reasoning/planning tower and a high-frequency action tower that can control 30–40 motors simultaneously at millisecond intervals. Both towers share a latent space for high-bandwidth context exchange, avoiding the bottleneck of decoding and re-encoding outputs between separate models running on distributed hardware.
•Three Distinct Compute Requirements: Building physical AI requires three separate computing environments: a GPU cluster for model training, a ray-tracing GPU for physics simulation and reinforcement learning validation, and an edge device for deployment. Developers cannot use one unified cluster as with LLMs — each stage demands fundamentally different hardware architectures and workflows.
•Real-to-Sim Data Pipeline: NVIDIA built a dedicated data curation tool called Curator specifically because standard LLM training pipelines cannot handle multimodal video, sensor, and motor data at the required throughput. Capturing real-world environments with cameras and LiDAR, then converting them into editable 3D simulations, allows augmentation of lighting, road width, and object placement for post-training.
•Cosmos 3 Edge Runs On-Robot: NVIDIA's Cosmos 3 Edge is a 4-billion-parameter world model optimized to run in real time on the Jetson Thor — a passport-sized computer delivering 2,000 teraops of compute. This enables on-device planning and counterfactual reasoning, where the model evaluates up to 10 possible future scenarios and selects the safest path without cloud round-trips.
•Entry Point for Software Developers: Developers can begin physical AI experimentation using a laptop with a 4090 GPU plus a simulator at no hardware cost. Adding a physical robotic arm — such as the SO-101, available as a 3D-printable kit — brings total startup cost to roughly $1,000. Hugging Face hosts pre-trained checkpoints matched to specific hardware, eliminating the need to customize models from scratch.

Notable Moment

Goel reveals that world models perform live counterfactual reasoning at inference time — not just during training. NVIDIA's Alpamayo self-driving model evaluates 10 distinct future trajectories simultaneously during operation, scores each against a safety objective, and selects the optimal path in real time rather than following a single predicted route.

Know someone who'd find this useful?

Episode Transcript

In recent years, the main impact of AI has been on the digital world and most notably software engineering. However, its longer term effect on the physical world may prove even more profound, reshaping how cars drive themselves, how factories run, and how robots move through homes and warehouses. World models are at the center of the movement to bring AI into the physical space. These models take in observations of the current state of the world and predict what comes next, which lets a robot plan its next move or car anticipate the road ahead. NVIDIA has been building a family of these models called Cosmos and sits at the frontier of physical AI and robotics. Amit Goel has been at NVIDIA for fifteen years, and he is currently the head of robotics ecosystem and edge AI product. In this episode, Amit joins Matt Merrill to discuss what separates physical AI from digital AI, how world models plan and predict, the two tower architecture that lets a robot reason and act at once, why training on video demanded a new data pipeline, and how any developer can start building in this space. Matt Merrill is a software engineering leader with over twenty years of experience building and scaling software teams across enterprise and product focused organizations. His background is in back end development, cloud architecture, and distributed systems design. He currently architects and delivers software products and leads a team of engineers at DEBT Agency. You can learn more about his work at code.theothermattm.com. Hello, everybody. My name is Matt Merrill, and I am here today with Amit Goel from NVIDIA who is head of robotics and edge AI. And I'm really excited to be here to talk to you about physical AI. So why don't I turn it over to you, Amit, to introduce yourself? Hey, Matt. Great to be here. I'm Amit Goel. I head up our robotics ecosystem at NVIDIA, which means my team gets to work with all the developers who are doing some amazing work with AI and robotics. And I also lead our product team that builds the computers for running the AI on the device on the edge. It's called the Jetson family of products. I've been at NVIDIA for fifteen years. We're very excited to talk about physical AI today. Alright. So I am super excited to talk about this because I think that a lot of folks listening here are probably, at this point, pretty familiar with the software side of Gen AI and LLMs. And what I kind of wanna do today is just start from zero. Right? Like... And that's actually... I'm not playing a role here. I purposely didn't dive into this deep so that I could kind of be fresh. So when NVIDIA or anybody says physical AI, what do you mean? What makes it different from LLMs and coding agents that most of our listeners are using every day? Yeah. The big difference between LLMs …

Get the full transcript (9,861 words) + summary by email — free

One-time email with the complete transcript and AI summary of this episode. No account needed.

One email, no spam. We’ll also show you what SignalCast does.

Browse all Software Engineering Daily transcripts →