"A March 2026 METR study found that roughly half of SWE-bench Verified test-passing AI patches in its sample would not have been merged by repository maintainers without further changes."
METR asked active maintainers to review 296 AI-generated pull requests from three repositories and compared their judgments with automated benchmark results. The findings demonstrate a measurable gap between passing task tests and satisfying maintainers' expectations.
View Source