"In September 2026 Scale AI released SWE-bench Pro V2, removing 89 invalid tasks from its public evaluation and introducing additional verification safeguards."
The September 22 revision reduced the public benchmark from 731 to 642 tasks and strengthened grading procedures. It demonstrates how benchmark scores depend on continuing curation, valid tests, and reliable execution environments.
View Source