Baseten CEO Tuhin Srivastava on the AI Inference Crunch, Custom Models, and Building the Inference Cloud

What It Covers

Baseten CEO Tuhin Srivastava joins Sarah Guo and Elad to discuss how the AI inference market reached 30x growth in 12 months, why 95% of tokens served run on custom models, how compute scarcity shapes strategy, and what the path from open-source adoption to specialized model deployment looks like.

Key Questions Answered

• •Custom model adoption: 95% of tokens served on Baseten run on customer-modified models, not vanilla open-source weights. Companies customize for both quality and performance through quantization and post-training. Builders should validate product-market fit with best-in-class frontier models first, then invest in specialization once user signal is proven and worth optimizing.
• •Compute scarcity reality: Baseten runs 90 clusters across 18 clouds at mid-90% utilization with a daily 4PM capacity management meeting. Securing 1,024 B200s from a reputable provider now requires a 3-to-5 year contract with 20-30% TCV prepaid upfront. Companies with low cost of capital hold a structural advantage in acquiring GPU supply.
• •Application layer defensibility: Companies like Abridge build moats through proprietary user signal — clinician edits, workflow steps, downstream actions — that frontier labs cannot access. Enterprises should encode differentiation in multi-step workflows and post-train models on those reward signals rather than relying solely on closed-source API access for competitive advantage.
• •Inference-post-training loop: Baseten acquired post-training firm PaaS to close the loop between inference and model improvement. Inference generates data, evals surface reward functions, and post-training improves the model — creating a continuous cycle. Infrastructure companies should treat inference and post-training as paired problems, since quantization decisions and training choices directly affect inference performance.
• •Jevons paradox in AI workloads: As inference costs fall, customers extend agent run times and insert intelligence into more workflow steps rather than reducing usage. Developers consistently respond to lower costs by running longer, more complex tasks. Companies building on AI should plan infrastructure and pricing models around expanding consumption, not stable or declining token volumes.