Why AI coding claims don't add up (News)
Episode
8 min
Read time
2 min
Topics
Productivity, Artificial Intelligence, Software Development
AI-Generated Summary
Key Takeaways
- ✓Productivity measurement gap: Developers in one study believed AI made them 20% faster but were actually 19% slower, revealing unreliable self-assessment of AI tool effectiveness in coding workflows.
- ✓Statistical significance requires time: Six weeks of productivity testing proved insufficient to determine AI impact; four additional months of data collection needed to establish meaningful conclusions about speed changes.
- ✓Market evidence contradiction: If AI tools truly doubled developer productivity, markets should show floods of new indie apps, games, and software across all platforms, yet no such explosion exists.
What It Covers
Developer Mike Judge tests AI coding tools for six weeks, finds no statistically significant productivity gains, challenging industry claims of massive developer speed improvements.
Key Questions Answered
- •Productivity measurement gap: Developers in one study believed AI made them 20% faster but were actually 19% slower, revealing unreliable self-assessment of AI tool effectiveness in coding workflows.
- •Statistical significance requires time: Six weeks of productivity testing proved insufficient to determine AI impact; four additional months of data collection needed to establish meaningful conclusions about speed changes.
- •Market evidence contradiction: If AI tools truly doubled developer productivity, markets should show floods of new indie apps, games, and software across all platforms, yet no such explosion exists.
Notable Moment
A developer who believed AI accelerated his coding by 25 percent discovered through rigorous self-testing that he could not prove any meaningful speed increase at all.
Episode Transcript
What up, nerds? Sheboy. I'm Jared, and this is Change Log News for the week of Monday, 09/08/2025. A shorter than usual episode this week and no new pods because Adam and I are flying to California to hang with the Oxide team during their annual internal conference. We hope to have some sweet content for you on the backside of this trip. Fingers crossed. I'll get a chance to shout in their data center. What I'm going to do is not recommended. This is not supported. Do not try and disappoint. Okay. Let's get into this week's news. Why AI coding claims don't add up. Mike Judge, no, not that Mike Judge, has been coding for twenty five plus years. He was an early adopter of AI coding and a fan until a couple months ago. Now he's mad, furious even. Quote, I read the meter study and suddenly got serious doubts. In that study, the authors discovered that developers were unreliable narrators of their own productivity. They thought AI was making them 20% faster, but it was actually making them 19% slower. This shocked me because I had just told someone the week before that I thought AI was only making me about 25% faster, and I was bummed. It wasn't a higher number. I was only off by 5% from the developer's own incorrect estimates, end quote. That study left him unsettled, so he put himself to the test. For six weeks, he tested his own productivity with and without AI. What he found was really disappointing. Quote, I discovered that the data isn't statistically significant at any meaningful level. That I would need to record new data points for another four months just to prove if AI was speeding me up or slowing me down at all. I can say definitively that I'm not seeing any massive increase in speed, as in two x, using AI coding tools, end quote. That got Mike thinking, is he the only one or are we all delusional? To answer that, he asked a simple question. Quote, if so many developers are so extraordinarily productive using these tools, where is the flood of shovelware? We should be seeing apps of all shapes and sizes, video games, new websites, mobile apps, software as a service apps, we should be drowning in choice. We should be in the middle of an indie software revolution. We should be seeing 10,000 Tetris clones on Steam, end quote. That question sent him further down the rabbit hole. Click through for receipts. Cactoid wants to be the ultimate RSVP platform. I have been prompting the open source community to kill Meetup for years. It looks like the folks behind Cactoid are taking a crack at it. Quote, like the cactus, great events bloom under any condition when managed with care. Cactoid helps you streamline RSVPs, simplify coordination, and keep every detail efficient so your gatherings are resilient, vibrant, and unforgettable, end quote. They've started off simply, …
Get the full transcript (1,380 words) + summary by email — free
One-time email with the complete transcript and AI summary of this episode. No account needed.
One email, no spam. We’ll also show you what SignalCast does.
You just read a 3-minute summary of a 5-minute episode.
Get The Changelog summarized like this every Monday — plus up to 2 more podcasts, free.
Pick Your Podcasts — FreeKeep Reading
More from The Changelog
Forking Cal.com to closed source (Interview)
Sep 3 · 114 min
The Bootstrapped Founder
393: AI is a Threat to SaaS Multiples
Jun 6
More from The Changelog
Postgres at PlanetScale (Interview)
Aug 25 · 102 min
Deep Questions with Cal Newport
Has AI Conquered Coding? (It’s Not So Simple…) | AI Reality Check
May 21
Books, tools, and gear mentioned in this episode
SignalCast may earn commission on purchases via these links.
Tools
“SPONSORS: Augment Code (https://augmentco.com)”
More from The Changelog
We summarize every new episode. Want them in your inbox?
Forking Cal.com to closed source (Interview)
Postgres at PlanetScale (Interview)
Canary tokens and digital tripwires (Interview)
From open source hits to OpenAI (Interview)
MCP on Code Mode (Interview)
Similar Episodes
Related episodes from other podcasts
The Bootstrapped Founder
Jun 6
393: AI is a Threat to SaaS Multiples
Deep Questions with Cal Newport
May 21
Has AI Conquered Coding? (It’s Not So Simple…) | AI Reality Check
The Daily (NYT)
Apr 14
The Workers Letting A.I. Do Their Jobs
The Vergecast
Feb 24
How Claude Code Claude Codes
Accidental Tech Podcast
Feb 12
678: Mentoring a Box of Numbers
Explore Related Topics
This podcast is featured in Best Cybersecurity Podcasts (2026) — ranked and reviewed with AI summaries.
Read this week's AI & Machine Learning Podcast Insights — cross-podcast analysis updated weekly.
You're clearly into The Changelog.
Every Monday, we deliver AI summaries of the latest episodes from The Changelog and 192+ other podcasts. Free for one show.
Start My Monday DigestNo credit card · Unsubscribe anytime