⚡️The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals & Human Data

Okay. Hi. We're here in the OPI Studio with Mia and Olivia from the Frontier Evals team or however you want to introduce yourself. May maybe Mia want to introduce, name what you do at OpenAI, and we can get it started. Sure. Hi. I'm Olivia. I'm on the Frontier Evals team. Are you sure? And Pete? Hi. I'm Mia. I am a VP of research at OpenAI, and my teams are the codex team, the simulator team, and the alignment team. And we work a lot with Olivia's team on Frontera. Yeah. Very exciting. And as by my understanding, you were part of the original team that worked on Cventorify as well. Yeah. Olivia's team, the Frontera Evarts team, and the human data team collaborated on creating three benchmarks. So you've you've seen the evolution of coding benchmarks over time. And I I think it was roundabout to the mid to late twenty twenty four we first published three branch verified. Things have evolved a lot since then. What's the blog post that you have worked on a year that we're releasing today? Like, what what is the sort of content what's the main thesis that you're pushing out? So the main thesis is that Supervised has been one of the North Star coding benchmarks that the field has looked at to measure coding progress. But recently we've seen that progress has kind of stalled. And this, we realized that this is because the eval is effectively saturated and also highly contaminated. So at this point, we think that it's not really measuring coding performance improvements well anymore, and we think that the field should move away from this towards other benchmarks. Like Supermunch Pro? Like Supermunch Pro. Yeah. Amazing. Yeah. I'd one of the jokes I always have is, like, there's a group chat with all the labs, and everyone just takes turns to increment, like, 0.1 on trucks. And then it's like, okay. Well, you have the best coding model, I guess, because you're 0.1% higher, but it's not super convincing at this point. No. Yeah. So cool. I think the let's let's sort of reset on, like, what was your original work that you guys did for CVS verified, which I think was pretty substantial. Like, it was, like, a very significant investment from OpenAI, which, like, people still don't appreciate. And then what were the dissatisfactions that we that we found over time? Right? So that like, what what was Speechbench Verified or should should that people should know about? SuiteBench verified was a kind of a cleanup of original bench academic benchmark from a lab at Princeton called SuiteBench. And the agent is basically given a code base and a task that was sourced from a real world repository and GitHub issue, and and was asked to solve the task and is graded on whether some tests pass. And at the time, this was it quickly became a popular benchmark because at the time, …