Skip to main content
The Changelog

Kaizen! Mop-up job (Friends)

107 min episode · 2 min read
·
Scott Beatson

Episode

107 min

Read time

2 min

Topics

Fundraising & VC, Software Development, Product & Tech Trends

AI-Generated Summary

Key Takeaways

  • Blue-Green Deployment Strategy: Always run new infrastructure alongside existing systems rather than in-place replacements. When Pipely's CDN instances crashed under full traffic load, reverting required only adding DNS records back, taking 10-15 minutes instead of hours of emergency fixes and preventing user-facing downtime during the migration.
  • Cache Performance Transformation: Homepage cache hit ratio jumped from 18.8% to 98.5%, feeds improved from 96.8% to 99.5%, and news pages went from 52.6% to 83% after switching CDNs. The homepage response time dropped from 259 milliseconds to 0.3 milliseconds at p50, representing an 863x speed improvement for majority of users.
  • Home Lab Benchmarking Infrastructure: Testing Pipely on 100 gigabit home network with Threadripper 9970X CPU achieved 225,000 requests per second serving homepage content and 2,200 requests per second for 13MB master feed files. Network throughput reached 80 gigabits per second before CPU became the bottleneck, validating production scalability assumptions.
  • CDN Cost-Benefit Analysis: Self-hosting CDN on Fly.io provides control over cache behavior and eliminates frustration with third-party limitations. Storage costs remain manageable for MP3 files, but video (MP4) distribution still makes sense on YouTube due to transcoding complexity, multiple codec requirements, and bandwidth expenses that incumbents absorb.
  • Analytics Pipeline Modernization: Current system duplicates request data across S3, Postgres, and Honeycomb with delayed processing via cron jobs. Proposed ClickHouse implementation would centralize all request events in columnar format, enable real-time analytics with materialized views, and eliminate background job complexity while maintaining S3 backup for long-term storage.

What It Covers

The Changelog team reviews their Pipely CDN launch at Denver, troubleshoots performance issues from under-provisioned instances, and analyzes dramatic improvements in cache hit ratios (70% to 89.5%) and response times (863x faster homepage loads) after migrating from Fastly to their self-hosted solution.

Key Questions Answered

  • Blue-Green Deployment Strategy: Always run new infrastructure alongside existing systems rather than in-place replacements. When Pipely's CDN instances crashed under full traffic load, reverting required only adding DNS records back, taking 10-15 minutes instead of hours of emergency fixes and preventing user-facing downtime during the migration.
  • Cache Performance Transformation: Homepage cache hit ratio jumped from 18.8% to 98.5%, feeds improved from 96.8% to 99.5%, and news pages went from 52.6% to 83% after switching CDNs. The homepage response time dropped from 259 milliseconds to 0.3 milliseconds at p50, representing an 863x speed improvement for majority of users.
  • Home Lab Benchmarking Infrastructure: Testing Pipely on 100 gigabit home network with Threadripper 9970X CPU achieved 225,000 requests per second serving homepage content and 2,200 requests per second for 13MB master feed files. Network throughput reached 80 gigabits per second before CPU became the bottleneck, validating production scalability assumptions.
  • CDN Cost-Benefit Analysis: Self-hosting CDN on Fly.io provides control over cache behavior and eliminates frustration with third-party limitations. Storage costs remain manageable for MP3 files, but video (MP4) distribution still makes sense on YouTube due to transcoding complexity, multiple codec requirements, and bandwidth expenses that incumbents absorb.
  • Analytics Pipeline Modernization: Current system duplicates request data across S3, Postgres, and Honeycomb with delayed processing via cron jobs. Proposed ClickHouse implementation would centralize all request events in columnar format, enable real-time analytics with materialized views, and eliminate background job complexity while maintaining S3 backup for long-term storage.

Notable Moment

After launching the new CDN on stage in Denver, the team went hiking and nearly walked into a rattlesnake while discussing technical details. The close encounter paralleled their production incident: both situations involved being engrossed in conversation while missing warning signs, though the CDN's memory crashes lacked the rattlesnake's helpful warning rattle.

Know someone who'd find this useful?

Episode Transcript

Welcome to Change Log and Friends, a weekly talk show about rattlesnake encounters. Thanks as always to our partners at fly.io. That's the public cloud built for developers who ship. We love Fly. You might too. Learn all about it at fly.io. Okay. What's Kaizen? Well, friends, I am here with a new friend of mine, Scott Beatson, CEO of Augment Code. I'm excited about this. Augment taps into your team's collective knowledge, your code base, your documentation, your dependencies. It is the most context aware developer AI, so you won't just code faster. You also build smarter. It's an ask me anything for your code. It's your deep thinking buddy. It's your stand flow antidote. Okay, Scott. So for the foreseeable future, AI assisted is here to stay. It's just a matter of getting the AI to be a better assistant. And in particular, I want help on the thinking part, not necessarily the coding part. Can you speak to the thinking problem versus the coding problem and the potential false dichotomy there? Couple of different points to make. You know, AIs have gotten good at making incremental changes at least when they understand customer software. So first in the biggest limitation that these AIs have today, they really don't understand anything about your code base. If you take GitHub Copilot, for example, it's like a fresh college graduate, understands some programming languages and algorithms, but doesn't understand what you're trying to do. And as a result of that, something like two thirds of the community on average drops off of the product, especially the expert developers. Augment is different. We use retrieval augmented generation to deeply mine the knowledge that's inherent inside your code base. So we are a co pilot that is an expert and they can help you navigate the codebase, help you find issues and fix them and resolve them over time much more quickly than you can, trying to tutor up a novice on your software. So you're often compared to GitHub Copilot. I gotta imagine that you have a hot take. What's your hot take on GitHub Copilot? I think it was a great 1.o product, and I think they've done a a huge, service in promoting AI, but I think the game has changed. We have moved from AIs that are new college graduates to, in effect, AIs that are now among you the best developers in your code base. And that difference is a profound one for software engineering in particular. You know, if you're writing a new application from scratch, you want a web page that'll play tic tac toe, piece of cake to crank that out. But if you're looking at, you know, tens of millions of line code base, like many of our customers, Lemonade is one of them. I mean, 10,000,000 line monorepo. As they move engineers inside and around that code base and hire new engineers, just the workload on senior developers to mentor people into …

Get the full transcript (19,217 words) + summary by email — free

One-time email with the complete transcript and AI summary of this episode. No account needed.

One email, no spam. We’ll also show you what SignalCast does.

Browse all The Changelog transcripts →

You just read a 3-minute summary of a 104-minute episode.

Get The Changelog summarized like this every Monday — plus up to 2 more podcasts, free.

Pick Your Podcasts — Free

Keep Reading

Books, tools, and gear mentioned in this episode

SignalCast may earn commission on purchases via these links. As an Amazon Associate, SignalCast earns from qualifying purchases.

Tools

  • The Changelog team reviews their Pipely CDN launch at Denver, troubleshoots performance issues from under-provisioned instances, and analyzes dramatic improvements in cache hit ratios (70% to 89.5%) and response times (863x faster homepage loads) after migrating from Fastly to their self-hosted solution.
  • Self-hosting CDN on Fly.io provides control over cache behavior and eliminates frustration with third-party limitations.
  • by Amazon Web Services

    Current system duplicates request data across S3, Postgres, and Honeycomb with delayed processing via cron jobs.
  • The Changelog team reviews their Pipely CDN launch at Denver, troubleshoots performance issues from under-provisioned instances, and analyzes dramatic improvements in cache hit ratios (70% to 89.5%) and response times (863x faster homepage loads) after migrating from Fastly to their self-hosted solution.
  • Current system duplicates request data across S3, Postgres, and Honeycomb with delayed processing via cron jobs.
  • by Google

    Storage costs remain manageable for MP3 files, but video (MP4) distribution still makes sense on YouTube due to transcoding complexity, multiple codec requirements, and bandwidth expenses that incumbents absorb.
  • Proposed ClickHouse implementation would centralize all request events in columnar format, enable real-time analytics with materialized views, and eliminate background job complexity while maintaining S3 backup for long-term storage.
  • Current system duplicates request data across S3, Postgres, and Honeycomb with delayed processing via cron jobs.

Gear

  • by AMD

    Testing Pipely on 100 gigabit home network with Threadripper 9970X CPU achieved 225,000 requests per second serving homepage content and 2,200 requests per second for 13MB master feed files.

More from The Changelog

We summarize every new episode. Want them in your inbox?

Similar Episodes

Related episodes from other podcasts

Explore Related Topics

This podcast is featured in Best Cybersecurity Podcasts (2026) — ranked and reviewed with AI summaries.

Read this week's Software Engineering Podcast Insights — cross-podcast analysis updated weekly.

You're clearly into The Changelog.

Every Monday, we deliver AI summaries of the latest episodes from The Changelog and 192+ other podcasts. Free for one show.

Start My Monday Digest

No credit card · Unsubscribe anytime