Skip to main content
The Bike Shed

481: Dev Horror Stories

43 min episode · 2 min read

Episode

43 min

Read time

2 min

Topics

Career Growth, Health & Wellness, Software Development

AI-Generated Summary

Key Takeaways

  • Database Recovery Strategy: When production data is lost, combine recent backups with transaction logs to rebuild state. After dropping a production database, recreating every transaction from logs since the last backup restored all data despite the backup being 24 hours old.
  • Case-Insensitive Search Performance: Using Postgres ILIKE for case-insensitive matching bypasses existing indexes, causing severe slowdowns at scale. Switching to citext column type maintains case-insensitive searching while preserving index performance, solving import bottlenecks when processing 10,000 records simultaneously.
  • Multi-Column Index Ordering: Index column order must match query WHERE clause column order exactly in Postgres. An index on name and location only works efficiently if queries use WHERE name AND location, not WHERE location AND name, making query planning critical.
  • Time Precision Bugs: Timestamp precision mismatches between Postgres and Ruby create flaky tests when subsecond values are converted to strings. Bugs appeared only when decimal portions were divisible by eight due to octal interpretation, requiring careful handling of time data transformations.

What It Covers

Sally and Joelle share developer horror stories from their careers, covering production database disasters, data corruption bugs, time zone issues, debugging challenges, and the scariest things encountered when joining new projects.

Key Questions Answered

  • Database Recovery Strategy: When production data is lost, combine recent backups with transaction logs to rebuild state. After dropping a production database, recreating every transaction from logs since the last backup restored all data despite the backup being 24 hours old.
  • Case-Insensitive Search Performance: Using Postgres ILIKE for case-insensitive matching bypasses existing indexes, causing severe slowdowns at scale. Switching to citext column type maintains case-insensitive searching while preserving index performance, solving import bottlenecks when processing 10,000 records simultaneously.
  • Multi-Column Index Ordering: Index column order must match query WHERE clause column order exactly in Postgres. An index on name and location only works efficiently if queries use WHERE name AND location, not WHERE location AND name, making query planning critical.
  • Time Precision Bugs: Timestamp precision mismatches between Postgres and Ruby create flaky tests when subsecond values are converted to strings. Bugs appeared only when decimal portions were divisible by eight due to octal interpretation, requiring careful handling of time data transformations.

Notable Moment

A developer discovered their time tracking application would double-pay an employee due to a bug triggered by the rare combination of a salary raise occurring during the same pay period as daylight savings time, caught only by accounting review.

Know someone who'd find this useful?

Episode Transcript

Auto scaling is a simple concept. Automatically add resources when needed and automatically shut them down to avoid paying for excess capacity. How hard could that be? If you've set up auto scaling yourself, you know it's not that easy, especially using the native auto scalers on platforms like AWS and Heroku. That's why you need Judo Scale. Judo Scale is auto scaling as a service, and they make auto scaling simple and easy, as it should be. You can use Judo Scale on AWS, Heroku, Render, fly.io, and more. It's free for low traffic apps and unlimited plans start at $25 per month. Autoscale on easy mode at judoscale.com. Hello, and welcome to another episode of the Bike Shed, a weekly podcast from your friends at Thoughtbot about developing great software. I'm Sally Hall. And I'm Joelle Kinville. And together, we're here to share a bit of what we've learned along the way. So, Joelle, what's new in your world? So this is one of those things that's new ish. Recently, both you and I were at TopBot's Summit, which is our sort of off-site event where we get all of us who work remotely and spend a week doing things in person. And this was held in Amsterdam, a city that is famous for its canals. So after some, then I had an extra day to myself to explore the city, and I visited the Canal Museum in Amsterdam. It's one of those places that it's not like a big fancy museum. It's just sort of, one of the classic, like, canal row houses. And then you go in. And the way that they've set up their displays is it's a lot of sort of interactive dioramas. So we'll have a diorama of the city and they'll have projectors, and they can sort of project video on top of that or highlight certain areas. So it's a very sort of, like, multimedia show kind of thing. I've seen a couple of these in a few different museums, And particularly for smaller museums, I just absolutely love this, type of display. I love a small niche museum, and that sounds really fun. What was the your favorite thing you learned about canals? Thing that was really interesting to me is how a lot of the canals were sort of developed in the early modern period. So after Amsterdam started getting a lot of wealth, they're sort of cramped inside the old medieval city center, and they're like, hey, we've got all swamps around us. What if we kinda drained them and built these, like, luxurious neighborhoods that have canals going through them? So it's really a sort of a planned expansion of the city where they got more land, built the canals, and then built these row houses that were meant to be for the wealthy along them. And in a weird sort of way, to me, that parallels the development of Boston's Back Bay neighborhood Okay. Which is …

Get the full transcript (7,403 words) + summary by email — free

One-time email with the complete transcript and AI summary of this episode. No account needed.

One email, no spam. We’ll also show you what SignalCast does.

Browse all The Bike Shed transcripts →

You just read a 3-minute summary of a 40-minute episode.

Get The Bike Shed summarized like this every Monday — plus up to 2 more podcasts, free.

Pick Your Podcasts — Free

Keep Reading

Books, tools, and gear mentioned in this episode

SignalCast may earn commission on purchases via these links.

Tools

  • by PostgreSQL

    Using Postgres ILIKE for case-insensitive matching bypasses existing indexes, causing severe slowdowns at scale.
  • Postgres citextRecommended

    by PostgreSQL

    Switching to citext column type maintains case-insensitive searching while preserving index performance, solving import bottlenecks when processing 10,000 records simultaneously.
  • by Scout

    SPONSORS: Scout Monitoring (scoutapm.com)
  • by Judo Scale

    SPONSORS: Judo Scale (judoscale.com)

More from The Bike Shed

We summarize every new episode. Want them in your inbox?

Similar Episodes

Related episodes from other podcasts

Explore Related Topics

This podcast is featured in Best Cybersecurity Podcasts (2026) — ranked and reviewed with AI summaries.

Read this week's Health & Longevity Podcast Insights — cross-podcast analysis updated weekly.

You're clearly into The Bike Shed.

Every Monday, we deliver AI summaries of the latest episodes from The Bike Shed and 192+ other podcasts. Free for one show.

Start My Monday Digest

No credit card · Unsubscribe anytime