Kaizen! Mop-up job (Friends)

What It Covers

The Changelog team reviews their Pipely CDN launch at Denver, troubleshoots performance issues from under-provisioned instances, and analyzes dramatic improvements in cache hit ratios (70% to 89.5%) and response times (863x faster homepage loads) after migrating from Fastly to their self-hosted solution.

Key Questions Answered

• •Blue-Green Deployment Strategy: Always run new infrastructure alongside existing systems rather than in-place replacements. When Pipely's CDN instances crashed under full traffic load, reverting required only adding DNS records back, taking 10-15 minutes instead of hours of emergency fixes and preventing user-facing downtime during the migration.
• •Cache Performance Transformation: Homepage cache hit ratio jumped from 18.8% to 98.5%, feeds improved from 96.8% to 99.5%, and news pages went from 52.6% to 83% after switching CDNs. The homepage response time dropped from 259 milliseconds to 0.3 milliseconds at p50, representing an 863x speed improvement for majority of users.
• •Home Lab Benchmarking Infrastructure: Testing Pipely on 100 gigabit home network with Threadripper 9970X CPU achieved 225,000 requests per second serving homepage content and 2,200 requests per second for 13MB master feed files. Network throughput reached 80 gigabits per second before CPU became the bottleneck, validating production scalability assumptions.
• •CDN Cost-Benefit Analysis: Self-hosting CDN on Fly.io provides control over cache behavior and eliminates frustration with third-party limitations. Storage costs remain manageable for MP3 files, but video (MP4) distribution still makes sense on YouTube due to transcoding complexity, multiple codec requirements, and bandwidth expenses that incumbents absorb.
• •Analytics Pipeline Modernization: Current system duplicates request data across S3, Postgres, and Honeycomb with delayed processing via cron jobs. Proposed ClickHouse implementation would centralize all request events in columnar format, enable real-time analytics with materialized views, and eliminate background job complexity while maintaining S3 backup for long-term storage.