Building Snipd: The AI Podcast App for Learning

What It Covers

Kevin Ben Smith, founder of Snipd, explains how his four-person team built an AI-powered podcast app focused on learning and knowledge retention. The conversation covers their technical architecture using Python, Google Cloud, and Flutter, their transition from self-hosted models to API-based LLMs, and their product philosophy of making AI invisible while solving real user problems around podcast discovery and retention.

Key Questions Answered

• •Triple-tap snipping feature: Users capture podcast insights by triple-tapping headphones, triggering AI summarization that saves the moment with transcript and audio to a knowledge library. This addresses the core problem that 90-99% of podcast content gets forgotten within minutes of listening, transforming passive consumption into active learning without requiring users to take manual notes or screenshots.
• •Cost optimization through model tiering: Snipd uses cheaper models for initial processing of over one million transcribed podcasts, then pipes outputs to premium models as judges for quality control. For quote selection, they generate five candidates with a budget model, then use a superior model to choose the best one, balancing scale economics with output quality across their massive audio processing pipeline.
• •Streaming LLM corrections with regex: When streaming chat responses to users, Snipd cannot pipe streams through additional LLM layers for formatting corrections due to API limitations. Instead, they maintain countless regex patterns that correct formatting errors in real-time as tokens stream, ensuring clean user experience. This represents practical AI engineering where traditional programming complements LLM capabilities rather than replacing it entirely.
• •Speaker diarization through heuristics: Snipd improves open-source diarization models by applying podcast-specific rules: speakers appearing for under thirty seconds in hour-long episodes are likely ad voices, not guests or hosts. They combine embedding-based clustering with LLM orchestration to assign speaker names, achieving better accuracy than tools like Descript by leveraging domain knowledge about podcast structure and high-quality studio audio.
• •Discovery through voice interfaces: Voice AI enables hooking into existing podcast listening habits rather than creating new notification-based triggers like Duolingo. When episodes end, an AI companion can initiate two-to-three minute conversations that force active processing of key takeaways, dramatically improving retention and application of knowledge. This backgroundable interaction stays within users' existing flow rather than requiring separate app opens or text-based engagement.
• •Multimodal future with raw audio: Current pipelines using Whisper transcription plus separate diarization will be replaced by feeding raw audio files directly into multimodal LLMs like Gemini 2.0 Flash. These models will output transcripts, speaker labels, timestamps, and metadata in one pass. The transition awaits cost parity with current self-hosted pipelines, but represents the inevitable direction as transformers continue consuming specialized audio processing tasks.