Ep. 370: Deep Work in the Age of AI

In recent weeks, there's been a lot of turmoil surrounding AI technology. It increasingly seems now like those grand promises of superintelligence or AI systems automating most of the economy, these grand promises are probably not going to come true. But what about the more practical promise? The one that AI tools are gonna make knowledge workers more productive, especially if you do something like computer programming for which AI is well suited. That's still true. Right? Well, the answer turns out to be complicated. A recent study about AI productivity yield a completely unexpected result. It created a a glitch in the matrix, so to speak, that leads to some deeper truths about this technology and its potential role in our work. Today, I wanna explore all of this. If you're at all interested in how AI might impact your job in the near future, this is an episode you won't wanna miss. As always, I'm Cal Newport, and this is Deep Questions. Today's episode, a glitch in the AI matrix. Alright. So I previewed there was a study that caught a lot of people off guard. I wanna load this on the screen here for those who are watching instead of just listening. This study came out in July 2025. It was produced by a nonprofit called METR. That's often pronounced meter. It's a nonprofit that does evaluation of AI and its capabilities. They do high quality studies. They don't have any particular bias. AI companies often cite their results when they find them to be, useful or impressive. So this is sort of a neutral body that does these reports. Here's the title of their July report that caused a bit of a surprise. Measuring the impact of early twenty twenty five AI on experienced open source developer productivity. So what did they do? I'm actually gonna read from the methodology section because I think it's important to understand, like, exactly what they were testing here. So let's read here together. This is from the paper. To directly measure the real world impact of AI tools on software development, we recruited 16 experienced developers from large open source repositories that they've contributed to for multiple years. Developers provide list of real issues that would be valuable to the repository, bug fixes, features, and refactors that would normally be part of their regular work. Then we randomly assign each issue to either allow or disallow use of AI while working on that particular issue. When AI is allowed, developers can use any tool they choose, primarily CursorPro with Claude 3.5 or 3.7 SONNET, which were the frontier models at the time of the study. When disallowed, they work without generative AI assistance. Developers complete these tasks, which average about two hours each while recording their screens, then self report total implementation time they needed. Alright. So that's the setup. It's it's a very elegant experiment. Here's developers working on the normal stuff they develop and issue by issue. …