Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]

What It Covers

Neil Movva, founder of Sail Research, explains how his "token factory" targets background AI agents running hours or days rather than real-time chatbots. By scavenging underpriced chips, tolerating 95% uptime data centers, and optimizing software kernels for throughput over latency, Sail aims to reduce inference costs by 10x or more.

Key Questions Answered

• •Latency vs. Throughput Trade-off: GPU hardware forces a fundamental choice between low latency and high throughput. Batching more users' work together maximizes GPU utilization but increases per-request wait time. Sail deliberately optimizes for throughput, accepting slower individual responses in exchange for dramatically lower cost per token — a viable strategy only when users aren't actively waiting on results.
• •Scavenger Chip Strategy: Rather than competing with Anthropic or OpenAI for Blackwell allocations, Sail buys underpriced chips from AMD, TPUs, Trainium, and emerging vendors like Etch or D-Matrix. Because other providers lack kernel expertise for non-NVIDIA architectures, Sail exploits the perception gap — treating "no bad chips, only bad pricing" as a core acquisition principle to build aggregate compute capacity.
• •Distributed 1-Megawatt Data Centers: Inference workloads, unlike training, don't require co-located clusters. Sail targets small, distributed one-megawatt facilities — roughly eight refrigerator-sized racks — that larger operators ignore. These facilities accept 95% uptime, skip redundant fiber and diesel generators, and can integrate intermittent solar and wind power, cutting overhead costs that traditional data centers treat as non-negotiable requirements.
• •KV Cache as the Primary Inefficiency: The KV cache — which stores every token in an active conversation — frequently exceeds model weight size and remains largely uncompressed. DeepSeek's published research shows roughly one order-of-magnitude compression progress annually, suggesting two or more orders of magnitude of remaining inefficiency. Targeting KV cache compression, not compute scaling, represents the highest-leverage optimization available in current inference stacks.
• •Background Agents as the Dominant Workload: Movva projects the inference mix shifting from roughly 50/50 real-time versus background today to 90% background within a few years. Background tasks — deep research across 10,000+ sources, autonomous cybersecurity pen-testing, proactive personal agents — consume tokens without human attention as the bottleneck, making token volume theoretically unbounded compared to human-in-the-loop workflows.
• •Open Source Distillation is Structurally Unavoidable: A growing percentage of GitHub repositories are generated by tools like Claude Code, meaning frontier model capabilities diffuse into open-source training data passively and continuously. Movva argues that training a frontier-class model solely on high-quality open-source code outputs is plausible today, making it structurally impossible for closed labs to permanently maintain capability advantages through access restrictions alone.