Evidence

Format Online paper / PDF

"An empirical study of agent-generated files found that human developers performed approximately 83.2% of their subsequent modifications."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER BenchMaxxing

Source title To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study

Source authors Shota Sawada; Tatsuya Shirai; Yutaro Kashiwa; Ken'ichi Yamaguchi; Hiroshi Iwata; Hajimu Iida

Source year 2026

Confidence high

Importance high

Format Online paper / PDF

"On a SpecBench SQL database task, an agent scored 100% on visible tests but only 35% on held-out tests after implementing features without shared abstractions."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER BenchMaxxing

Source title SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents

Source authors Bingchen Zhao; Dhruv Srikanth; Yuxiang Wu; Zhengyao Jiang

Source year 2026

Confidence high

Importance high

Format Online paper / PDF

"In SpecBench, the 90th-percentile reward-hacking gap increased by approximately 27 percentage points for every tenfold increase in reference implementation size."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER BenchMaxxing

Source title SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents

Source authors Bingchen Zhao; Dhruv Srikanth; Yuxiang Wu; Zhengyao Jiang

Source year 2026

Confidence high

Importance high

Format Online paper / PDF

"SpecBench measures the gap between coding agents’ performance on visible feature-level validation tests and held-out tests of interacting system features."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER BenchMaxxing

Source title SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents

Source authors Bingchen Zhao; Dhruv Srikanth; Yuxiang Wu; Zhengyao Jiang

Source year 2026

Confidence high

Importance high

Format Web Page

"In September 2026 Scale AI released SWE-bench Pro V2, removing 89 invalid tasks from its public evaluation and introducing additional verification safeguards."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER BenchMaxxing

Source title SWE-Bench Pro V2

Source authors Scale AI

Source year 2026

Confidence high

Importance high

Format Web Page

"In July 2026 OpenAI estimated that approximately 30% of SWE-bench Pro’s public tasks contained significant evaluation defects and withdrew its recommendation of the benchmark."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER BenchMaxxing

Source title Separating signal from noise in coding evaluations

Source authors OpenAI

Source year 2026

Confidence high

Importance high

Format Web Page

"SWE-bench Pro combined public, held-out, and private commercial repositories to reduce training-data contamination and evaluate coding agents on more diverse tasks."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER BenchMaxxing

Source title SWE-Bench Pro: Raising the Bar for Agentic Coding

Source authors The Scale Research Team

Source year 2025

Confidence high

Importance high

Format Web Page

"Scale AI introduced SWE-bench Pro in September 2025, and leading models initially scored approximately 23% on its public evaluation."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER BenchMaxxing

Source title SWE-Bench Pro: Raising the Bar for Agentic Coding

Source authors The Scale Research Team

Source year 2025

Confidence high

Importance high

Format Web Page

"A March 2026 METR study found that roughly half of SWE-bench Verified test-passing AI patches in its sample would not have been merged by repository maintainers without further changes."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER BenchMaxxing

Source title Many SWE-bench-Passing PRs Would Not Be Merged into Main

Source authors METR

Source year 2026

Confidence high

Importance high

Format Web Page

"In February 2026 OpenAI stopped reporting SWE-bench Verified because test defects and training-data contamination made it unreliable for measuring further frontier progress."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER BenchMaxxing

Source title Why SWE-bench Verified no longer measures frontier coding capabilities

Source authors OpenAI

Source year 2026

Confidence high

Importance high