390: When to Choose Local LLMs vs APIs

What It Covers

Arvid Kahl shares practical lessons from building PodScan on when to use local AI models versus remote APIs based on scale, cost, and privacy requirements.

Key Questions Answered

• •Scale threshold: Local models work for under a few hundred operations daily, but at thousands of operations per day, remote APIs become more cost-effective due to economies of scale that individual founders cannot replicate.
• •CPU viability: Small language models running on CPU can handle low-context tasks like yes-no decisions on short text in two to five minutes, eliminating API costs for async workflows without requiring GPU investment.
• •Hybrid approach: Start with APIs to validate market and understand scale, then add local GPU servers as fallback for privacy compliance and reliability, avoiding vendor lock-in while maintaining operational flexibility.