Ethan He

AI Summary

→ WHAT IT COVERS Ethan He, formerly of xAI's Grok Imagine team, traces the full technical stack of building video generation models from zero — covering data pipelines, VAE tokenization, diffusion training costs, audio-video alignment, and his thesis that video model quality gains now derive primarily from language model intelligence, pointing toward video agents as the next major category. → KEY INSIGHTS - **Video model bootstrap sequence:** Building a production video model requires first training an image model, because image-text pairs are denser and cheaper to acquire than video-text pairs. Internet videos lack natural text alignment — YouTube titles rarely describe visual content — so synthetic captions must be generated via VLM, with human labelers instructed to describe footage in enough detail that a blind person could reconstruct it mentally. Image models then serve as the foundation for video fine-tuning. - **VAE compression tradeoffs:** Video transformers cannot train on raw pixels — a 1000×1000 image alone produces one million tokens, making attention computationally impossible. VAEs compress inputs into continuous latent spaces using patch-based encoding (typically 16×16 patches). Temporal compression ratios like 8×8×4 reduce sequence length fourfold versus frame-by-frame encoding, but introduce latency that breaks real-time interactivity. Frame-by-frame VAEs preserve responsiveness at the cost of four times larger context windows. - **Video model training costs rival mid-scale LLMs:** Storing one billion five-megabyte videos requires five petabytes of storage — approximately $230K per month on AWS S3 — plus comparable storage for pre-computed VAE features. AWS charges egress fees on top. Model parameters for open video models like LTX reach 19B dense, with MoE variants targeting 20B active and hundreds of billions total. Token counts during training reach tens of trillions, matching mid-scale language model training runs in compute cost. - **Step distillation cuts inference from 100 steps to 4–8:** Production video models use step distillation to reduce generation from 100+ diffusion steps to 4–8 steps without retraining from scratch. A distilled student model learns to replicate the teacher's output distribution in fewer steps, because the teacher's distribution is simpler than the raw internet distribution the teacher originally learned. Cosmos ships four-step and eight-step distilled variants; image-to-image transfer tasks can run in a single step. - **Language models supply most video quality gains:** The prompt rewriter — typically a larger language model like Mixtral — contributes more to output quality than the diffusion model itself. Video diffusion models interpret prompts literally: "a cat" produces a static cat on a white background. The language model expands sparse user intent into detailed scene descriptions. Thinking budgets, tool calling, and web search now extend this further, meaning reasoning model improvements directly translate into better video outputs without retraining the diffusion component. - **Video extension solves long-horizon generation via full history context:** Most video models generate isolated clips of a few seconds with no memory of prior content. Grok Imagine's video extension feature conditions each new clip on the complete history of previously generated video tokens, maintaining character voice consistency and object continuity across extended sequences. Naive implementation causes context window explosion — five seconds of video in Cosmos produces roughly 50–60K tokens — requiring selective context retrieval mechanisms like reference images or frame compression heuristics such as Frame Pack. - **Video agents are the near-term production unlock:** By end of 2025, video agents — reasoning models orchestrating diffusion models, video editors, FFmpeg, and other deterministic tools — will reach production-grade quality suitable for commercial distribution in advertising. The agent layer handles long-horizon tasks (generating one-minute videos, iterative refinement, layout control) that diffusion models cannot execute from a single prompt. Enterprise budgets will follow once agents cross the usability threshold, creating an exponential adoption curve similar to the transition from GitHub Copilot to fully autonomous coding agents. → NOTABLE MOMENT He argues that the majority of video generation quality improvements now originate from language model advances rather than diffusion architecture improvements — a position he describes as a "black pill" for researchers who have built careers in generative media. This conviction drove him to leave xAI specifically to focus on language model research, treating video as a downstream beneficiary. 💼 SPONSORS None detected 🏷️ Video Generation, World Models, Diffusion Models, Video Agents, Multimodal AI, xAI Grok, Inference Optimization