"In 2025 METR randomized 246 real software-development tasks between AI-allowed and AI-disallowed conditions for 16 experienced open-source developers."
METR tested developers working in mature repositories with which they had extensive prior experience. The randomized design measured productivity in familiar real-world projects rather than relying on synthetic programming exercises.
View Source