[State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang

What It Covers

John Yang discusses SWE-bench evolution since its October 2023 launch, including multilingual extensions across nine languages, the new Code Clash benchmark for long-horizon development, and emerging evaluation approaches for autonomous coding agents.

Key Questions Answered

• •SWE-bench Extensions: Multilingual version covers nine programming languages including JavaScript, Rust, Java, C, and Ruby across 40 repositories, addressing criticism about the original benchmark's Django focus and expanding evaluation beyond Python-centric tasks.
• •Code Clash Framework: New benchmark evaluates long-horizon development by having models maintain separate codebases that compete in programming tournaments across multiple rounds, testing iterative improvement and consequential changes rather than isolated task completion typical of unit test approaches.
• •Impossible Tasks as Cheating Detection: Benchmarks should intentionally include impossible or underspecified tasks as flags to detect when models or teams are gaming the evaluation system, with any scores above certain thresholds indicating potential benchmark contamination or cheating.
• •User Simulator Limitations: Current approaches like Tau-bench and Vending-bench sample single paths and lack realism, creating need for better human-AI interaction data either through compelling products that generate real usage patterns or sophisticated simulators beyond simple prompting.