Evidence

Format Online paper / PDF

"Meta’s Galactica was presented as a broad interface for scientific knowledge and tasks, including literature synthesis, mathematical reasoning, scientific writing, and biological information."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER BenchMaxxing

Source title Galactica: A Large Language Model for Science

Source authors Ross Taylor; Marcin Kardas; Guillem Cucurull; Thomas Scialom; Anthony Hartshorn; Elvis Saravia; Andrew Poulton; Viktor Kerkez; Robert Stojnic

Source year 2022

Confidence high

Importance mid

Format Web Page

"Timnit Gebru argues that building AGI, understood as an undefined general-purpose system, cannot be subjected to ordinary engineering safety tests in the way well-scoped applications can."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER BenchMaxxing

Source title Eugenics and the Promise of Utopia through Artificial General Intelligence

Source authors Timnit Gebru

Source year 2023

Confidence high

Importance high

Format Online paper / PDF

"Influential AI benchmarks are often framed as general measures of progress despite serious construct-validity problems."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER BenchMaxxing

Source title AI and the Everything in the Whole Wide World Benchmark

Source authors Inioluwa Deborah Raji; Emily M. Bender; Amandalynne Paullada; Emily Denton; Alex Hanna

Source year 2021

Confidence high

Importance high

Format Online paper / PDF

"Computer benchmarking has a pedigree reaching back to the early 1960s, when standardized performance comparisons helped guide hardware decisions."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER BenchMaxxing

Source title Estimating computer performance

Source authors J. A. Gosden

Source year 1963

Confidence high

Importance mid

Format Video / YouTube Embed

"Dex Horthy described maintainability problems arising when teams relied on coding agents without consistently reading and reviewing the code they produced."

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER The Final Form of Software Development

Source title Why Software Factories Fail

Source authors Dex Horthy

Source year 2026

Confidence high

Importance mid

Format Web Page

"Kenton Varda reported that AI-written change descriptions repeated visible code details while omitting the higher-level framing needed for review."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER The Final Form of Software Development

Source title A quote from Kenton Varda

Source authors Kenton Varda; collected by Simon Willison

Source year 2026

Confidence high

Importance mid

Format Web Page

"In Kun Chen’s DeepSWE experiment, preventing an agent from writing its own tests reduced time and token use without a statistically significant decline in task success."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER The Final Form of Software Development

Source title OK, it's official... AI-written unit tests and integration tests are empirically proven to be unhelpful.

Source authors Kun Chen

Source year 2026

Confidence high

Importance mid

Format Online paper / PDF

"Agent-written tests did not significantly change final task-resolution outcomes when researchers increased or decreased their use in a controlled software-engineering benchmark."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER The Final Form of Software Development

Source title Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents

Source authors Zhi Chen; Zhensu Sun; Yuling Shi; Chao Peng; Xiaodong Gu; David Lo; Lingxiao Jiang

Source year 2026

Confidence high

Importance high

Format Web Page

"Armin Ronacher asked an AI coding agent to build a rate limiter that appeared to work but omitted jitter and made poor storage decisions."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER The Final Form of Software Development

Source title 90%

Source authors Armin Ronacher

Source year 2025

Confidence high

Importance high

Format Podcast / Spotify embed

"PHP_CodeSniffer’s automated checking and fixing capabilities depend on specialized open-source maintenance work that benefits a much wider software ecosystem."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER The Final Form of Software Development

Source title Underfunded PHP Tools: A Global Business Threat

Source authors Anne Bovelett; Juliette Reinders Folmer

Source year 2026

Confidence high

Importance mid