Skip to main content
The Bike Shed

481: Dev Horror Stories

43 min episode · 2 min read

Episode

43 min

Read time

2 min

AI-Generated Summary

Key Takeaways

  • Database Recovery Strategy: When production data is lost, combine recent backups with transaction logs to rebuild state. After dropping a production database, recreating every transaction from logs since the last backup restored all data despite the backup being 24 hours old.
  • Case-Insensitive Search Performance: Using Postgres ILIKE for case-insensitive matching bypasses existing indexes, causing severe slowdowns at scale. Switching to citext column type maintains case-insensitive searching while preserving index performance, solving import bottlenecks when processing 10,000 records simultaneously.
  • Multi-Column Index Ordering: Index column order must match query WHERE clause column order exactly in Postgres. An index on name and location only works efficiently if queries use WHERE name AND location, not WHERE location AND name, making query planning critical.
  • Time Precision Bugs: Timestamp precision mismatches between Postgres and Ruby create flaky tests when subsecond values are converted to strings. Bugs appeared only when decimal portions were divisible by eight due to octal interpretation, requiring careful handling of time data transformations.

What It Covers

Sally and Joelle share developer horror stories from their careers, covering production database disasters, data corruption bugs, time zone issues, debugging challenges, and the scariest things encountered when joining new projects.

Key Questions Answered

  • Database Recovery Strategy: When production data is lost, combine recent backups with transaction logs to rebuild state. After dropping a production database, recreating every transaction from logs since the last backup restored all data despite the backup being 24 hours old.
  • Case-Insensitive Search Performance: Using Postgres ILIKE for case-insensitive matching bypasses existing indexes, causing severe slowdowns at scale. Switching to citext column type maintains case-insensitive searching while preserving index performance, solving import bottlenecks when processing 10,000 records simultaneously.
  • Multi-Column Index Ordering: Index column order must match query WHERE clause column order exactly in Postgres. An index on name and location only works efficiently if queries use WHERE name AND location, not WHERE location AND name, making query planning critical.
  • Time Precision Bugs: Timestamp precision mismatches between Postgres and Ruby create flaky tests when subsecond values are converted to strings. Bugs appeared only when decimal portions were divisible by eight due to octal interpretation, requiring careful handling of time data transformations.

Notable Moment

A developer discovered their time tracking application would double-pay an employee due to a bug triggered by the rare combination of a salary raise occurring during the same pay period as daylight savings time, caught only by accounting review.

Know someone who'd find this useful?

You just read a 3-minute summary of a 40-minute episode.

Get The Bike Shed summarized like this every Monday — plus up to 2 more podcasts, free.

Pick Your Podcasts — Free

Keep Reading

More from The Bike Shed

We summarize every new episode. Want them in your inbox?

Similar Episodes

Related episodes from other podcasts

This podcast is featured in Best Cybersecurity Podcasts (2026) — ranked and reviewed with AI summaries.

You're clearly into The Bike Shed.

Every Monday, we deliver AI summaries of the latest episodes from The Bike Shed and 192+ other podcasts. Free for up to 3 shows.

Start My Monday Digest

No credit card · Unsubscribe anytime