Sonnet 5 review: I ran 64 generations to find out if it's worth it

We've got a new model, people, and it's from Anthropic. Now is it Mythos? No. Is it Fable? No. But it is Claude SONNET five. Anthropic is claiming it's the most agentic SONNET model yet, and we will get opus level tasks at Sonnet level prices. Now I've been testing a lot of models, and I'm starting to get bored of doing the vibe check. What I wanna start developing is a set of benchmarks we can regularly test these new models against that you'll care about. So today, I'm going to be introducing the How I AI Bench, a set of AI and Claro graded benchmarks that are gonna tell us if this model and any model is good at writing PRDs, solving bugs, and one shotting designs. I'm gonna show you exactly how I built this benchmark using Claude Code, and we're gonna see on a blind test what comes out on top. Let's get to it. This episode is brought to you by Runway, a new kind of creative platform that has everything you need to generate any image, video, or piece of content you want, all in one place. With Runway, it's now possible to go from initial idea to a finished deliverable in a matter of minutes. From turning low fidelity product shots into campaign ready imagery all the way through putting together big brand films, Runway can help your team scale your creative ambitions while keeping your budgets and timelines from doing the same. Runway brings together the world's most advanced AI models, which is why enterprises like Microsoft, Robinhood, Amazon, and Adobe, along with studios like Lionsgate and Legendary, all use Runway to ship real work every day. Try it yourself at runwayml.com/howiai, promo code how iai. Quickly, before we get to our evals, let's just talk about the headlines of Sonnet five, this new model. Anthropic is pitching it as close to the performance of Opus four eight, but much less expensive. So as you can see here, it's not quite at this 69% on Agenca coating, SuiteBench Pro, or the 82% on terminal bench 2.1, but it's not that far behind. And I suspect that most of us are not going to notice the difference. It's also supposed to be really good at computer work and knowledge work, and so this should be an everyday model that people reach for. In my episode with Felix from Anthropic, he says that we're all abusing opus, and we should definitely be using the sonnet models more, and we are going to put SONNET five to the test against that proposition. Now what do they say that SONNET five is really good at? Well, it's really good at agentic tool use. So you're gonna get slightly longer running tool runs, longer running sessions than you would with SONNET four six at a lower cost than doing the same comparable task with Opus. So you're gonna see here, you know, SONNET four six, a lower …