Kaizen! Mop-up job (Friends)
Episode
107 min
Read time
2 min
Topics
Fundraising & VC, Software Development, Product & Tech Trends
AI-Generated Summary
Key Takeaways
- ✓Blue-Green Deployment Strategy: Always run new infrastructure alongside existing systems rather than in-place replacements. When Pipely's CDN instances crashed under full traffic load, reverting required only adding DNS records back, taking 10-15 minutes instead of hours of emergency fixes and preventing user-facing downtime during the migration.
- ✓Cache Performance Transformation: Homepage cache hit ratio jumped from 18.8% to 98.5%, feeds improved from 96.8% to 99.5%, and news pages went from 52.6% to 83% after switching CDNs. The homepage response time dropped from 259 milliseconds to 0.3 milliseconds at p50, representing an 863x speed improvement for majority of users.
- ✓Home Lab Benchmarking Infrastructure: Testing Pipely on 100 gigabit home network with Threadripper 9970X CPU achieved 225,000 requests per second serving homepage content and 2,200 requests per second for 13MB master feed files. Network throughput reached 80 gigabits per second before CPU became the bottleneck, validating production scalability assumptions.
- ✓CDN Cost-Benefit Analysis: Self-hosting CDN on Fly.io provides control over cache behavior and eliminates frustration with third-party limitations. Storage costs remain manageable for MP3 files, but video (MP4) distribution still makes sense on YouTube due to transcoding complexity, multiple codec requirements, and bandwidth expenses that incumbents absorb.
- ✓Analytics Pipeline Modernization: Current system duplicates request data across S3, Postgres, and Honeycomb with delayed processing via cron jobs. Proposed ClickHouse implementation would centralize all request events in columnar format, enable real-time analytics with materialized views, and eliminate background job complexity while maintaining S3 backup for long-term storage.
What It Covers
The Changelog team reviews their Pipely CDN launch at Denver, troubleshoots performance issues from under-provisioned instances, and analyzes dramatic improvements in cache hit ratios (70% to 89.5%) and response times (863x faster homepage loads) after migrating from Fastly to their self-hosted solution.
Key Questions Answered
- •Blue-Green Deployment Strategy: Always run new infrastructure alongside existing systems rather than in-place replacements. When Pipely's CDN instances crashed under full traffic load, reverting required only adding DNS records back, taking 10-15 minutes instead of hours of emergency fixes and preventing user-facing downtime during the migration.
- •Cache Performance Transformation: Homepage cache hit ratio jumped from 18.8% to 98.5%, feeds improved from 96.8% to 99.5%, and news pages went from 52.6% to 83% after switching CDNs. The homepage response time dropped from 259 milliseconds to 0.3 milliseconds at p50, representing an 863x speed improvement for majority of users.
- •Home Lab Benchmarking Infrastructure: Testing Pipely on 100 gigabit home network with Threadripper 9970X CPU achieved 225,000 requests per second serving homepage content and 2,200 requests per second for 13MB master feed files. Network throughput reached 80 gigabits per second before CPU became the bottleneck, validating production scalability assumptions.
- •CDN Cost-Benefit Analysis: Self-hosting CDN on Fly.io provides control over cache behavior and eliminates frustration with third-party limitations. Storage costs remain manageable for MP3 files, but video (MP4) distribution still makes sense on YouTube due to transcoding complexity, multiple codec requirements, and bandwidth expenses that incumbents absorb.
- •Analytics Pipeline Modernization: Current system duplicates request data across S3, Postgres, and Honeycomb with delayed processing via cron jobs. Proposed ClickHouse implementation would centralize all request events in columnar format, enable real-time analytics with materialized views, and eliminate background job complexity while maintaining S3 backup for long-term storage.
Notable Moment
After launching the new CDN on stage in Denver, the team went hiking and nearly walked into a rattlesnake while discussing technical details. The close encounter paralleled their production incident: both situations involved being engrossed in conversation while missing warning signs, though the CDN's memory crashes lacked the rattlesnake's helpful warning rattle.
Episode Transcript
Welcome to Change Log and Friends, a weekly talk show about rattlesnake encounters. Thanks as always to our partners at fly.io. That's the public cloud built for developers who ship. We love Fly. You might too. Learn all about it at fly.io. Okay. What's Kaizen? Well, friends, I am here with a new friend of mine, Scott Beatson, CEO of Augment Code. I'm excited about this. Augment taps into your team's collective knowledge, your code base, your documentation, your dependencies. It is the most context aware developer AI, so you won't just code faster. You also build smarter. It's an ask me anything for your code. It's your deep thinking buddy. It's your stand flow antidote. Okay, Scott. So for the foreseeable future, AI assisted is here to stay. It's just a matter of getting the AI to be a better assistant. And in particular, I want help on the thinking part, not necessarily the coding part. Can you speak to the thinking problem versus the coding problem and the potential false dichotomy there? Couple of different points to make. You know, AIs have gotten good at making incremental changes at least when they understand customer software. So first in the biggest limitation that these AIs have today, they really don't understand anything about your code base. If you take GitHub Copilot, for example, it's like a fresh college graduate, understands some programming languages and algorithms, but doesn't understand what you're trying to do. And as a result of that, something like two thirds of the community on average drops off of the product, especially the expert developers. Augment is different. We use retrieval augmented generation to deeply mine the knowledge that's inherent inside your code base. So we are a co pilot that is an expert and they can help you navigate the codebase, help you find issues and fix them and resolve them over time much more quickly than you can, trying to tutor up a novice on your software. So you're often compared to GitHub Copilot. I gotta imagine that you have a hot take. What's your hot take on GitHub Copilot? I think it was a great 1.o product, and I think they've done a a huge, service in promoting AI, but I think the game has changed. We have moved from AIs that are new college graduates to, in effect, AIs that are now among you the best developers in your code base. And that difference is a profound one for software engineering in particular. You know, if you're writing a new application from scratch, you want a web page that'll play tic tac toe, piece of cake to crank that out. But if you're looking at, you know, tens of millions of line code base, like many of our customers, Lemonade is one of them. I mean, 10,000,000 line monorepo. As they move engineers inside and around that code base and hire new engineers, just the workload on senior developers to mentor people into …
Get the full transcript (19,217 words) + summary by email — free
One-time email with the complete transcript and AI summary of this episode. No account needed.
One email, no spam. We’ll also show you what SignalCast does.
You just read a 3-minute summary of a 104-minute episode.
Get The Changelog summarized like this every Monday — plus up to 2 more podcasts, free.
Pick Your Podcasts — FreeKeep Reading
Books, tools, and gear mentioned in this episode
SignalCast may earn commission on purchases via these links. As an Amazon Associate, SignalCast earns from qualifying purchases.
Tools
“The Changelog team reviews their Pipely CDN launch at Denver, troubleshoots performance issues from under-provisioned instances, and analyzes dramatic improvements in cache hit ratios (70% to 89.5%) and response times (863x faster homepage loads) after migrating from Fastly to their self-hosted solution.”
“Self-hosting CDN on Fly.io provides control over cache behavior and eliminates frustration with third-party limitations.”
by Amazon Web Services
“Current system duplicates request data across S3, Postgres, and Honeycomb with delayed processing via cron jobs.”
“The Changelog team reviews their Pipely CDN launch at Denver, troubleshoots performance issues from under-provisioned instances, and analyzes dramatic improvements in cache hit ratios (70% to 89.5%) and response times (863x faster homepage loads) after migrating from Fastly to their self-hosted solution.”
“Current system duplicates request data across S3, Postgres, and Honeycomb with delayed processing via cron jobs.”
by Google
“Storage costs remain manageable for MP3 files, but video (MP4) distribution still makes sense on YouTube due to transcoding complexity, multiple codec requirements, and bandwidth expenses that incumbents absorb.”
“Proposed ClickHouse implementation would centralize all request events in columnar format, enable real-time analytics with materialized views, and eliminate background job complexity while maintaining S3 backup for long-term storage.”
“Current system duplicates request data across S3, Postgres, and Honeycomb with delayed processing via cron jobs.”
Gear
by AMD
“Testing Pipely on 100 gigabit home network with Threadripper 9970X CPU achieved 225,000 requests per second serving homepage content and 2,200 requests per second for 13MB master feed files.”
More from The Changelog
We summarize every new episode. Want them in your inbox?
Forking Cal.com to closed source (Interview)
Postgres at PlanetScale (Interview)
Canary tokens and digital tripwires (Interview)
From open source hits to OpenAI (Interview)
MCP on Code Mode (Interview)
Similar Episodes
Related episodes from other podcasts
The Vergecast
Jun 22
Is the Steam Machine worth the wait?
The Vergecast
Aug 15
Vibe coding through the GPT-5 mess
Lenny's Podcast
Sep 8
How we built Grok Bot in a month | Roman Ugarte (SpaceXAI)
The AI Breakdown
Sep 1
OpenClaw 2.0 Shows Where AI Agents Are Going Next
Practical AI
Aug 25
AI Proficiency: From Users to Builders
Explore Related Topics
This podcast is featured in Best Cybersecurity Podcasts (2026) — ranked and reviewed with AI summaries.
Read this week's Software Engineering Podcast Insights — cross-podcast analysis updated weekly.
You're clearly into The Changelog.
Every Monday, we deliver AI summaries of the latest episodes from The Changelog and 192+ other podcasts. Free for one show.
Start My Monday DigestNo credit card · Unsubscribe anytime