AI Summary
→ WHAT IT COVERS Baseten CEO Tuhin Srivastava joins Sarah Guo and Elad to discuss how the AI inference market reached 30x growth in 12 months, why 95% of tokens served run on custom models, how compute scarcity shapes strategy, and what the path from open-source adoption to specialized model deployment looks like. → KEY INSIGHTS - **Custom model adoption:** 95% of tokens served on Baseten run on customer-modified models, not vanilla open-source weights. Companies customize for both quality and performance through quantization and post-training. Builders should validate product-market fit with best-in-class frontier models first, then invest in specialization once user signal is proven and worth optimizing. - **Compute scarcity reality:** Baseten runs 90 clusters across 18 clouds at mid-90% utilization with a daily 4PM capacity management meeting. Securing 1,024 B200s from a reputable provider now requires a 3-to-5 year contract with 20-30% TCV prepaid upfront. Companies with low cost of capital hold a structural advantage in acquiring GPU supply. - **Application layer defensibility:** Companies like Abridge build moats through proprietary user signal — clinician edits, workflow steps, downstream actions — that frontier labs cannot access. Enterprises should encode differentiation in multi-step workflows and post-train models on those reward signals rather than relying solely on closed-source API access for competitive advantage. - **Inference-post-training loop:** Baseten acquired post-training firm PaaS to close the loop between inference and model improvement. Inference generates data, evals surface reward functions, and post-training improves the model — creating a continuous cycle. Infrastructure companies should treat inference and post-training as paired problems, since quantization decisions and training choices directly affect inference performance. - **Jevons paradox in AI workloads:** As inference costs fall, customers extend agent run times and insert intelligence into more workflow steps rather than reducing usage. Developers consistently respond to lower costs by running longer, more complex tasks. Companies building on AI should plan infrastructure and pricing models around expanding consumption, not stable or declining token volumes. → NOTABLE MOMENT Srivastava revealed that Baseten's co-founder's seven-year-old child has learned to ask "is that a P0?" when a pager alert fires at home — illustrating how deeply an always-on operational culture becomes embedded across an infrastructure company scaling at this pace. 💼 SPONSORS None detected 🏷️ AI Inference, Custom Model Deployment, GPU Compute Scarcity, Post-Training Optimization, AI Application Layer