Neil Movva

AI Summary

→ WHAT IT COVERS Neil Movva, founder of Sail Research, explains how his "token factory" targets background AI agents running hours or days rather than real-time chatbots. By scavenging underpriced chips, tolerating 95% uptime data centers, and optimizing software kernels for throughput over latency, Sail aims to reduce inference costs by 10x or more. → KEY INSIGHTS - **Latency vs. Throughput Trade-off:** GPU hardware forces a fundamental choice between low latency and high throughput. Batching more users' work together maximizes GPU utilization but increases per-request wait time. Sail deliberately optimizes for throughput, accepting slower individual responses in exchange for dramatically lower cost per token — a viable strategy only when users aren't actively waiting on results. - **Scavenger Chip Strategy:** Rather than competing with Anthropic or OpenAI for Blackwell allocations, Sail buys underpriced chips from AMD, TPUs, Trainium, and emerging vendors like Etch or D-Matrix. Because other providers lack kernel expertise for non-NVIDIA architectures, Sail exploits the perception gap — treating "no bad chips, only bad pricing" as a core acquisition principle to build aggregate compute capacity. - **Distributed 1-Megawatt Data Centers:** Inference workloads, unlike training, don't require co-located clusters. Sail targets small, distributed one-megawatt facilities — roughly eight refrigerator-sized racks — that larger operators ignore. These facilities accept 95% uptime, skip redundant fiber and diesel generators, and can integrate intermittent solar and wind power, cutting overhead costs that traditional data centers treat as non-negotiable requirements. - **KV Cache as the Primary Inefficiency:** The KV cache — which stores every token in an active conversation — frequently exceeds model weight size and remains largely uncompressed. DeepSeek's published research shows roughly one order-of-magnitude compression progress annually, suggesting two or more orders of magnitude of remaining inefficiency. Targeting KV cache compression, not compute scaling, represents the highest-leverage optimization available in current inference stacks. - **Background Agents as the Dominant Workload:** Movva projects the inference mix shifting from roughly 50/50 real-time versus background today to 90% background within a few years. Background tasks — deep research across 10,000+ sources, autonomous cybersecurity pen-testing, proactive personal agents — consume tokens without human attention as the bottleneck, making token volume theoretically unbounded compared to human-in-the-loop workflows. - **Open Source Distillation is Structurally Unavoidable:** A growing percentage of GitHub repositories are generated by tools like Claude Code, meaning frontier model capabilities diffuse into open-source training data passively and continuously. Movva argues that training a frontier-class model solely on high-quality open-source code outputs is plausible today, making it structurally impossible for closed labs to permanently maintain capability advantages through access restrictions alone. → NOTABLE MOMENT Movva argues that losing access to TSMC would be far less catastrophic than geopolitical discourse suggests. He contends that Intel's best processes trail TSMC by at most two times on performance-per-watt — a gap far smaller than commonly assumed — and that successive TSMC nodes themselves deliver only modest incremental efficiency gains per generation. 💼 SPONSORS [{"name": "Ramp", "url": "https://ramp.com/invest"}, {"name": "WorkOS", "url": "https://workos.com"}, {"name": "Rogo (Felix)", "url": "https://rogo.ai/felix"}, {"name": "Vanta", "url": "https://vanta.com/invest"}, {"name": "Ridgeline", "url": "https://ridgelineapps.com"}] 🏷️ AI Inference, GPU Optimization, Open Source Models, Data Center Infrastructure, AI Agents, Semiconductor Supply Chain