"In July 2026 OpenAI estimated that approximately 30% of SWE-bench Pro’s public tasks contained significant evaluation defects and withdrew its recommendation of the benchmark."
OpenAI's audit identified overly strict tests, underspecified prompts, misleading requirements, and insufficient test coverage. This subsequent assessment shows that replacing one flawed benchmark with a harder one does not automatically solve construct-validity problems.
View Source