⚡️The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals & Human Data

What It Covers

OpenAI's Mia Glaese and Olivia Watkins explain why SWE-Bench Verified, the dominant coding benchmark since mid-2024, is now saturated and contaminated, why the field should migrate to SWE-Bench Pro, and what next-generation agentic coding evaluations need to measure.

Key Questions Answered

• •Benchmark Saturation Signal: When frontier models cluster around 80%+ on SWE-Bench Verified and gains shrink to 0.1% increments, the benchmark stops measuring coding capability and starts measuring noise. Teams should treat any benchmark where top models score above 80% as a signal to find harder, less saturated alternatives rather than continuing to optimize against it.
• •Contamination Detection Method: OpenAI deployed a contamination auditor agent that presents target models with open-ended questions about task descriptions and patches, probing for memorized ground-truth solutions or task IDs. This method surfaced contamination across GPT, Claude Opus 4.5, and Gemini Flash — including a case where a model's chain-of-thought cited a historical repository argument never mentioned in the problem spec.
• •Test Fairness Audit Finding: A deep-dive human review of problems no frontier model could solve found that over half contained flawed tests — most commonly, tests requiring a specific function or argument name never specified in the problem description. Benchmark creators should audit failures by comparing model solutions against gold patches with domain-expert reviewers, not just checking pass/fail rates.
• •SWE-Bench Pro Advantages: SWE-Bench Pro, produced by Scale, addresses verified's weaknesses: tasks are estimated at one-to-four hours and four-plus hours for expert engineers, covers more repositories and multiple programming languages, and the contamination auditor found only marginal familiarity with one or two source repositories across all tested models — a substantially cleaner signal than its predecessor.
• •Next Benchmark Priorities: Olivia Watkins identifies three gaps the field should fill: tasks requiring top engineers months or teams weeks to complete with rubric-validated grading, end-to-end product creation benchmarks, and real-world usage metrics tracking how much AI is actually deployed in production workflows — moving beyond synthetic pass/fail rates toward measurable economic and labor impact.