"SWE-bench Pro combined public, held-out, and private commercial repositories to reduce training-data contamination and evaluate coding agents on more diverse tasks."
Scale AI's original benchmark contained 1,865 tasks across 41 repositories, including private commercial codebases. Its design sought to distinguish performance on public code from generalization to previously inaccessible commercial repositories.
View Source