Open Source Self-Driving with Comma AI

What It Covers

Harald Sch, CTO at Comma AI, explains how OpenPilot — the most popular open source robotics project on GitHub — uses end-to-end machine learning and a diffusion-based world model simulator to deliver highway autonomy across supported vehicles, while outlining three unsolved problems blocking full autonomous driving: controls, reinforcement learning, and continual learning.

Key Questions Answered

• •End-to-End Training Architecture: Comma AI trains models directly from hundreds of millions of miles of human driving data, skipping intermediate detection layers like lane-line segmentation or traffic-light classifiers entirely. The model takes raw camera video as input and outputs two values: longitudinal acceleration and road curvature. This minimal output design keeps the on-device model small enough to run on a phone-grade chip.
• •Diffusion Simulator as Training Environment: Rather than classical depth-reprojection simulators, Comma AI now trains its driving policy inside a machine-learning-generated video simulator built on diffusion models. The critical differentiator is input-response accuracy — if the simulator is told the car turns left 10 degrees, it must produce video that precisely reflects that turn, not just photorealistic footage, making it viable for robotics training.
• •Compute Gap and Its Practical Ceiling: Comma AI's current device runs roughly 100 times less compute than a Tesla FSD computer. Despite this gap, highway performance is comparable because capability gains require exponential compute increases for marginal real-world improvements. A planned external GPU add-on targeting 100x more compute is projected to roughly double detection reliability in nuanced situations like ambiguous traffic lights.
• •Continual Learning as an Unsolved Requirement: OpenPilot currently uses classical optimization to learn vehicle-specific parameters — tire stiffness, friction coefficients — live during each drive. Inflating tires or driving in rain changes vehicle dynamics that the system must adapt to in real time. Standard neural network approaches cannot yet handle this live adaptation, making continual learning one of three critical unsolved problems for production autonomy.
• •Open Source as a Functional Requirement, Not Just Philosophy: Supporting hundreds of car models requires community contributors to reverse-engineer each vehicle's CAN bus signals. A closed-source stack would make this ecosystem impossible to scale. Comma AI treats open sourcing the car-interface layer as a structural necessity, while also holding a philosophical position that device owners should have full visibility into and control over software running on hardware they purchase.