"Scale AI introduced SWE-bench Pro in September 2025, and leading models initially scored approximately 23% on its public evaluation."
Scale AI reported initial public-split scores of 23.3% for GPT-5 and 23.1% for Claude Opus 4.1. The benchmark introduced more demanding software-engineering tasks and demonstrated how evaluation design can substantially change measured model performance.
View Source