The "confident idiot" problem (News)

What up, nerds? Sheboy. I'm Jared, and this is Change Log News for the week of Monday, 12/08/2025. We are quickly approaching last call for state of the log voice mails. We record the show in a week, and we have to give BMC time to make the remixes. So if you're thinking about sending one in, and you should, now is the best time. Submit yours today at changelog.fm/sotl. Okay. Let's get into this week's news. The confident idiot problem or why AI needs hard rules, not vibe checks. If you've been following the how do we actually use AI in production conversation stream, you've probably heard people propose a strategy where one LLM checks another LLM's results. But will that work? Quote, we are told to ask g p t four o to grade g p t 3.5. We are told to fix the vibes, but this creates a dangerous circular dependency. If the underlying models suffer from sycophancy, which is agreeing with the user, or hallucination, a judge model often hallucinates a passing grade. We are trying to fix probability with more probability. That is a losing game, end quote. One possible way of dealing with these confident idiots we've introduced into our software stacks the last few years is to stop treating agents like magic boxes and start treating them like software. Hence, the Steer SDK was created. Quote, Steer is an open source Python library that intercepts agent failures, such as hallucinations, bad JSON, PII leaks, etcetera, and allows you to inject fixes via a local dashboard without changing your code, end quote. Another way of dealing with these confident idiots in our software stacks is remove them, but that might not be possible anymore. BUN is joining Anthropic. The company behind BUN, which is the open source runtime for Cloud Code, is joining Anthropic. We discussed the big acquisition slash acqui hire on last week's Friends episode, but at the time, I hadn't quite considered this move and how contrary it is to Anthropic's party line that AI agents are replacing software engineers. From Anthropic's announcement, quote, we've been a close partner of Bun for many months. Our collaboration has been central to the rapid execution of the Cloud Code team, and it directly drove the recent launch of Cloud Code's native installer. We know the Bun team is building from the same vantage point that we do at Anthropic with a focus on rethinking the developer experience and building innovative, useful products, end quote. BUN is open source. Why not just fork it and have a cloud code powered engineer make all the necessary changes and upgrades to the runtime that Anthropic needs? Perhaps because there's no getting there from here, at least not yet. Jared Sumner and the BUN team's expertise is what's so valuable still even to Anthropic. Claude can't recreate classic Space Jam site. Jonah Glover tried to recreate everyone's favorite 1996 website by giving Claude Code, which is running …