[State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang

“New domain-specific benchmarks emerged including SWE-ficiency for code optimization without behavior changes, Psy-code for scientific computing, SEC-bench for security, and SRE-bench for operations.”

“New domain-specific benchmarks emerged including SWE-ficiency for code optimization without behavior changes, Psy-code for scientific computing, SEC-bench for security, and SRE-bench for operations.”

“New domain-specific benchmarks emerged including SWE-ficiency for code optimization without behavior changes, Psy-code for scientific computing, SEC-bench for security, and SRE-bench for operations.”

“New domain-specific benchmarks emerged including SWE-ficiency for code optimization without behavior changes, Psy-code for scientific computing, SEC-bench for security, and SRE-bench for operations.”

by Cognition

“Yang reveals that when Cognition released Devon with strong SWE-bench results, he received only two weeks advance notice via email.”

“Code Clash Framework: This new evaluation method replaces unit tests with programming tournaments where two or more language models maintain separate codebases, iteratively improving them each round before competing in arenas.”

“John Yang discusses the evolution of SWE-bench coding benchmarks since its October 2022 launch, including multilingual extensions across nine languages, the new Code Clash tournament framework for long-horizon development evaluation, and emerging challenges in coding evaluation methodology.”

“Independent teams created variants like SWE-bench Pro without original author involvement, showing benchmark adoption.”

“Academic researchers lack access to valuable user interaction data that companies like Cognition and Cursor collect naturally through product usage.”

“Academic researchers lack access to valuable user interaction data that companies like Cognition and Cursor collect naturally through product usage.”