[State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang
Episode
17 min
Read time
2 min
Topics
Productivity, Investing, Fundraising & VC
AI-Generated Summary
Key Takeaways
- ✓SWE-bench Extensions: The benchmark expanded beyond its Django-focused Python origins to include multilingual support across nine languages (JavaScript, Rust, Java, C, Ruby) and 40 repositories, plus multimodal capabilities. Independent teams created variants like SWE-bench Pro without original author involvement, showing benchmark adoption.
- ✓Code Clash Framework: This new evaluation method replaces unit tests with programming tournaments where two or more language models maintain separate codebases, iteratively improving them each round before competing in arenas. Models must demonstrate long-horizon development skills with consequential, dependent changes rather than isolated task completion.
- ✓Benchmark Diversification: New domain-specific benchmarks emerged including SWE-ficiency for code optimization without behavior changes, Psy-code for scientific computing, SEC-bench for security, and SRE-bench for operations. Each targets specific coding domains beyond general software engineering, enabling more targeted model evaluation and development.
- ✓Academic Data Limitations: Academic researchers lack access to valuable user interaction data that companies like Cognition and Cursor collect naturally through product usage. Building compelling products or creating realistic user simulators both present significant challenges, limiting academic progress on human-AI collaboration research compared to industry.
What It Covers
John Yang discusses the evolution of SWE-bench coding benchmarks since its October 2022 launch, including multilingual extensions across nine languages, the new Code Clash tournament framework for long-horizon development evaluation, and emerging challenges in coding evaluation methodology.
Key Questions Answered
- •SWE-bench Extensions: The benchmark expanded beyond its Django-focused Python origins to include multilingual support across nine languages (JavaScript, Rust, Java, C, Ruby) and 40 repositories, plus multimodal capabilities. Independent teams created variants like SWE-bench Pro without original author involvement, showing benchmark adoption.
- •Code Clash Framework: This new evaluation method replaces unit tests with programming tournaments where two or more language models maintain separate codebases, iteratively improving them each round before competing in arenas. Models must demonstrate long-horizon development skills with consequential, dependent changes rather than isolated task completion.
- •Benchmark Diversification: New domain-specific benchmarks emerged including SWE-ficiency for code optimization without behavior changes, Psy-code for scientific computing, SEC-bench for security, and SRE-bench for operations. Each targets specific coding domains beyond general software engineering, enabling more targeted model evaluation and development.
- •Academic Data Limitations: Academic researchers lack access to valuable user interaction data that companies like Cognition and Cursor collect naturally through product usage. Building compelling products or creating realistic user simulators both present significant challenges, limiting academic progress on human-AI collaboration research compared to industry.
Notable Moment
Yang reveals that when Cognition released Devon with strong SWE-bench results, he received only two weeks advance notice via email. The release sparked an industry arms race in coding benchmarks, transforming SWE-bench from a little-used academic project into a central evaluation standard.
Episode Transcript
We're here at NeurIPS with John Yang of SpeedBench and many other things, but welcome. Thanks so much for having me. Yeah. Really happy to be here. Last year, I talked to Ophir and, I think Carlos as well, one one of your co authors. Yeah. How's SpeedBench doing? Like, just just generally, the project is, like, one and a half years old? Yeah. Yeah. So? I think one and a half years old in terms of when it was actually useful. Yeah. And we put it out October 2223, and then people didn't really touch it too much. And then, of course, like, Cognition came on the scene, and Devon was an amazing release. And I think after that, it kinda kicked off the arms race. Did they tell you beforehand, or they just showed up? It was you know, I got an email about, like, two weeks ago. I think it was from I think it was from Walden. He was like, hey. You know, we have a good number on it. I was like, wow. Congrats. You know? Thanks for using it. And then the release was, like, mind blowing. I was like, wow. These guys did an excellent job. Yeah. Amazing. And then Sweet Bunch verified was, like, maybe last year's That's right. Yeah. Like Catch us up this year. Like, you have, other languages. You've there there's, like, a whole bunch of varieties of SuiteBench now. Yeah. So what should people know? Yeah. For sure. I think there's a couple extensions that are happened. One is, like, more Suitebenches, sweep bench pro, sweep bench live, Oh, sweep bench pro, was that with you guys? Because it looks independent. It's like different authors. It's completely independent. Yeah. So they just called themselves sweep bench pro without your blessing? Yeah. I think, I think we're we're we're we're okay with it. When we came out, we were like, oh, cool. Interesting. Would've been, you know, fun to be part of it. But, you know, I mean, congrats to them. That's a great benchmark. Yeah. Right. Yeah. But, yeah, multimodal? Yeah. We did multimodal and multilingual, and I think, like, those have multilingual seems to be the one. Is it, like, JavaScripts? What else? Yeah. Yep. Yep. Multilingual is, like, it's, like, nine languages Yeah. Across, like, 40 repos. But, yeah, you got them, like, JavaScript, Rust, Java, c, you know, Ruby. Yeah. Yeah. You got them. Yeah. And then CorsiBeach itself, a lot of people, like, they they talk about the the Django focus. Yes. Django focus. Is there is there is there, like, I don't know. How do you how do you move past Django? Yeah. For sure. I mean, it it's cool to see, a lot of the newer benchmarks, like, really try to diversify the repos. Like, in the two follow ups we did with multimodal and multilingual, we made it a point to do that. So So I think But you can also …
Get the full transcript (3,594 words) + summary by email — free
One-time email with the complete transcript and AI summary of this episode. No account needed.
One email, no spam. We’ll also show you what SignalCast does.
You just read a 3-minute summary of a 14-minute episode.
Get Latent Space summarized like this every Monday — plus up to 2 more podcasts, free.
Pick Your Podcasts — FreeKeep Reading
More from Latent Space
🔬“We have foundation models for language, not for physics” — Anima Anandkumar, Bren Professor of Computing
Aug 26 · 83 min
The AI Breakdown
Why AI Needs Better Benchmarks
Mar 26
More from Latent Space
Simulation: the new Scaling Law — Joon Sung Park, Simile AI
Aug 21 · 69 min
Cognitive Revolution
What AI Means for Students & Teachers: My Keynote from the Michigan Virtual AI Summit
Nov 25
Books, tools, and gear mentioned in this episode
SignalCast may earn commission on purchases via these links.
Tools
“New domain-specific benchmarks emerged including SWE-ficiency for code optimization without behavior changes, Psy-code for scientific computing, SEC-bench for security, and SRE-bench for operations.”
“New domain-specific benchmarks emerged including SWE-ficiency for code optimization without behavior changes, Psy-code for scientific computing, SEC-bench for security, and SRE-bench for operations.”
“New domain-specific benchmarks emerged including SWE-ficiency for code optimization without behavior changes, Psy-code for scientific computing, SEC-bench for security, and SRE-bench for operations.”
“New domain-specific benchmarks emerged including SWE-ficiency for code optimization without behavior changes, Psy-code for scientific computing, SEC-bench for security, and SRE-bench for operations.”
by Cognition
“Yang reveals that when Cognition released Devon with strong SWE-bench results, he received only two weeks advance notice via email.”
“Code Clash Framework: This new evaluation method replaces unit tests with programming tournaments where two or more language models maintain separate codebases, iteratively improving them each round before competing in arenas.”
“John Yang discusses the evolution of SWE-bench coding benchmarks since its October 2022 launch, including multilingual extensions across nine languages, the new Code Clash tournament framework for long-horizon development evaluation, and emerging challenges in coding evaluation methodology.”
“Independent teams created variants like SWE-bench Pro without original author involvement, showing benchmark adoption.”
company
“Academic researchers lack access to valuable user interaction data that companies like Cognition and Cursor collect naturally through product usage.”
“Academic researchers lack access to valuable user interaction data that companies like Cognition and Cursor collect naturally through product usage.”
More from Latent Space
We summarize every new episode. Want them in your inbox?
🔬“We have foundation models for language, not for physics” — Anima Anandkumar, Bren Professor of Computing
Simulation: the new Scaling Law — Joon Sung Park, Simile AI
🔬The BioAI Phase Shift - Matthew McPartlon & Neil Patil, Chai Discovery
The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten
Codex from 0 to 10M Users: Building ChatGPT Work — Akshay Nathan, OpenAI
Similar Episodes
Related episodes from other podcasts
The AI Breakdown
Mar 26
Why AI Needs Better Benchmarks
Cognitive Revolution
Nov 25
What AI Means for Students & Teachers: My Keynote from the Michigan Virtual AI Summit
How I AI
Jun 24
GLM 5.2: why I’m replacing Opus in Claude Code with this new model
The AI Breakdown
Jun 22
Why AI Users Are Raving About GLM 5.2
How I AI
Jun 15
How Braintrust uses AI agents, evals, and CI to ship better software | Ankur Goyal
Explore Related Topics
This podcast is featured in Best AI Podcasts (2026) — ranked and reviewed with AI summaries.
Read this week's Investing & Markets Podcast Insights — cross-podcast analysis updated weekly.
You're clearly into Latent Space.
Every Monday, we deliver AI summaries of the latest episodes from Latent Space and 192+ other podcasts. Free for one show.
Start My Monday DigestNo credit card · Unsubscribe anytime