404: The Transcription Challenge: Building Infrastructure That Scales With The World

Hey. It's Arvid, and this is the Bootstrap founder. Today, we'll talk about keeping up with an avalanche of audio data and how I build PodScan's transcription infrastructure. This episode is sponsored by paddle.com, my merchant of record payment provider of choice who's been helping me focus on PodScan from day one. They're taking care of all the little things related to money so that founders like you and me can focus on building the things that only we can build, like a massive pod cast transcription infrastructure. Paddle handles all the rest, sales tax, credit cards, those kind of things. Don't need to deal with it because they do. I highly recommend checking it out. So please go to paddle.com and take a look. Now when I started building the first prototype of PodScan, I very quickly realized that this was gonna be a different business than any that I've built before. The difference had everything to do with one fundamental challenge in this field. Unlike most software service businesses, the resources that I would need from the start wouldn't scale with the number of customers I had, but would scale with something completely out of my control. The number of new podcast episodes being released worldwide every single day. So no matter if I had one customer or a 100, if they wanted to track every podcast out there for a keyword, I needed to deal with this from day one. And that's hard because if you ever investigated the idea of stoicism, you will know that there are certain things you can control, that you should care about, and certain things that you cannot control, that you shouldn't fret about at all. That's kind of the idea. Like, a very rough description of stoicism here. But, you know, like, it's deal with the things you could deal with and don't whine about the others. So that's exactly what I did. I focused on what I could do to make transcribing every single podcast out there a reality and I didn't complain about the fact that there are tens of thousands millions of shows being released all the time with tens of thousands of shows being released every day. I that's kind of the framework here. I had to deal with it. I think I'm currently tracking 3,800,000 shows. And roughly every day, there's somewhere between 30 to 70,000 being released. Depends on the day of the week. And I wanna talk about this Herculean effort of building transcription infrastructure, how I got it from being extremely expensive to manageably cheap comparatively, what the trade offs were along the way, and how much of the development of new technologies has impacted the feasibility of this entire project for me. Now for my first prototype, I obviously didn't try to transcribe everything at once. Like, I knew that that didn't make sense to try it all. But I had found my source of podcast feed data, just a couple …