481: Dev Horror Stories
Episode
43 min
Read time
2 min
Topics
Career Growth, Health & Wellness, Software Development
AI-Generated Summary
Key Takeaways
- ✓Database Recovery Strategy: When production data is lost, combine recent backups with transaction logs to rebuild state. After dropping a production database, recreating every transaction from logs since the last backup restored all data despite the backup being 24 hours old.
- ✓Case-Insensitive Search Performance: Using Postgres ILIKE for case-insensitive matching bypasses existing indexes, causing severe slowdowns at scale. Switching to citext column type maintains case-insensitive searching while preserving index performance, solving import bottlenecks when processing 10,000 records simultaneously.
- ✓Multi-Column Index Ordering: Index column order must match query WHERE clause column order exactly in Postgres. An index on name and location only works efficiently if queries use WHERE name AND location, not WHERE location AND name, making query planning critical.
- ✓Time Precision Bugs: Timestamp precision mismatches between Postgres and Ruby create flaky tests when subsecond values are converted to strings. Bugs appeared only when decimal portions were divisible by eight due to octal interpretation, requiring careful handling of time data transformations.
What It Covers
Sally and Joelle share developer horror stories from their careers, covering production database disasters, data corruption bugs, time zone issues, debugging challenges, and the scariest things encountered when joining new projects.
Key Questions Answered
- •Database Recovery Strategy: When production data is lost, combine recent backups with transaction logs to rebuild state. After dropping a production database, recreating every transaction from logs since the last backup restored all data despite the backup being 24 hours old.
- •Case-Insensitive Search Performance: Using Postgres ILIKE for case-insensitive matching bypasses existing indexes, causing severe slowdowns at scale. Switching to citext column type maintains case-insensitive searching while preserving index performance, solving import bottlenecks when processing 10,000 records simultaneously.
- •Multi-Column Index Ordering: Index column order must match query WHERE clause column order exactly in Postgres. An index on name and location only works efficiently if queries use WHERE name AND location, not WHERE location AND name, making query planning critical.
- •Time Precision Bugs: Timestamp precision mismatches between Postgres and Ruby create flaky tests when subsecond values are converted to strings. Bugs appeared only when decimal portions were divisible by eight due to octal interpretation, requiring careful handling of time data transformations.
Notable Moment
A developer discovered their time tracking application would double-pay an employee due to a bug triggered by the rare combination of a salary raise occurring during the same pay period as daylight savings time, caught only by accounting review.
Episode Transcript
Auto scaling is a simple concept. Automatically add resources when needed and automatically shut them down to avoid paying for excess capacity. How hard could that be? If you've set up auto scaling yourself, you know it's not that easy, especially using the native auto scalers on platforms like AWS and Heroku. That's why you need Judo Scale. Judo Scale is auto scaling as a service, and they make auto scaling simple and easy, as it should be. You can use Judo Scale on AWS, Heroku, Render, fly.io, and more. It's free for low traffic apps and unlimited plans start at $25 per month. Autoscale on easy mode at judoscale.com. Hello, and welcome to another episode of the Bike Shed, a weekly podcast from your friends at Thoughtbot about developing great software. I'm Sally Hall. And I'm Joelle Kinville. And together, we're here to share a bit of what we've learned along the way. So, Joelle, what's new in your world? So this is one of those things that's new ish. Recently, both you and I were at TopBot's Summit, which is our sort of off-site event where we get all of us who work remotely and spend a week doing things in person. And this was held in Amsterdam, a city that is famous for its canals. So after some, then I had an extra day to myself to explore the city, and I visited the Canal Museum in Amsterdam. It's one of those places that it's not like a big fancy museum. It's just sort of, one of the classic, like, canal row houses. And then you go in. And the way that they've set up their displays is it's a lot of sort of interactive dioramas. So we'll have a diorama of the city and they'll have projectors, and they can sort of project video on top of that or highlight certain areas. So it's a very sort of, like, multimedia show kind of thing. I've seen a couple of these in a few different museums, And particularly for smaller museums, I just absolutely love this, type of display. I love a small niche museum, and that sounds really fun. What was the your favorite thing you learned about canals? Thing that was really interesting to me is how a lot of the canals were sort of developed in the early modern period. So after Amsterdam started getting a lot of wealth, they're sort of cramped inside the old medieval city center, and they're like, hey, we've got all swamps around us. What if we kinda drained them and built these, like, luxurious neighborhoods that have canals going through them? So it's really a sort of a planned expansion of the city where they got more land, built the canals, and then built these row houses that were meant to be for the wealthy along them. And in a weird sort of way, to me, that parallels the development of Boston's Back Bay neighborhood Okay. Which is …
Get the full transcript (7,403 words) + summary by email — free
One-time email with the complete transcript and AI summary of this episode. No account needed.
One email, no spam. We’ll also show you what SignalCast does.
You just read a 3-minute summary of a 40-minute episode.
Get The Bike Shed summarized like this every Monday — plus up to 2 more podcasts, free.
Pick Your Podcasts — FreeKeep Reading
More from The Bike Shed
506: The Muppet Software Team
Jul 14 · 38 min
The Joe Rogan Experience
#2484 - David Cross
Apr 16
More from The Bike Shed
505: What is a “principal” or “staff” engineer?
Jul 7 · 34 min
The Full Ratchet
Investor Stories 460: The Math Behind Venture Returns, When Sizing Matters More Than Selection, and Growth Over Discounts (Wallach, Okike, Banks)
Feb 19
Books, tools, and gear mentioned in this episode
SignalCast may earn commission on purchases via these links.
Tools
by PostgreSQL
“Using Postgres ILIKE for case-insensitive matching bypasses existing indexes, causing severe slowdowns at scale.”
- Postgres citextRecommended
by PostgreSQL
“Switching to citext column type maintains case-insensitive searching while preserving index performance, solving import bottlenecks when processing 10,000 records simultaneously.”
More from The Bike Shed
We summarize every new episode. Want them in your inbox?
Similar Episodes
Related episodes from other podcasts
The Joe Rogan Experience
Apr 16
#2484 - David Cross
The Full Ratchet
Feb 19
Investor Stories 460: The Math Behind Venture Returns, When Sizing Matters More Than Selection, and Growth Over Discounts (Wallach, Okike, Banks)
99% Invisible
Dec 30
Beyond the 99% Invisible City
The Jordan Harbinger Show
Nov 18
1242: Christopher Whitcomb | A Life Among Spies Part One
The Indicator
Oct 10
Scam compounds, sewing patterns and stolen dimes
Explore Related Topics
This podcast is featured in Best Cybersecurity Podcasts (2026) — ranked and reviewed with AI summaries.
Read this week's Health & Longevity Podcast Insights — cross-podcast analysis updated weekly.
You're clearly into The Bike Shed.
Every Monday, we deliver AI summaries of the latest episodes from The Bike Shed and 192+ other podcasts. Free for one show.
Start My Monday DigestNo credit card · Unsubscribe anytime