Evidence

Format Online paper / PDF

"LLM-based code reviewers can mistakenly classify correct implementations as violating natural-language requirements."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER Judging the Plausibility Machine

Source title Are LLMs reliable code reviewers? systematic overcorrection in requirement conformance judgement

Source authors Haolin Jin; Huaming Chen

Source year 2026

Confidence high

Importance high

Format Web Page

"No general algorithm can decide every non-trivial semantic property of arbitrary programs."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER Judging the Plausibility Machine

Source title Recursive Functions

Source authors Walter Dean

Source year 2020

Confidence high

Importance high

Format Web Page

"In Dex Horthy’s three-problem SlopCodeBench experiment, Opus 5 passed four of seventeen strict checkpoints, while the other two evaluated models passed one each."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER Judging the Plausibility Machine

Source title Benchmarking Opus 5 on SlopCodeBench

Source authors Dexter Horthy

Source year 2026

Confidence high

Importance high

Format Tweet

"Testing whether coding agents can keep changing a codebase without breaking existing behavior offers a useful measure of their long-term engineering ability."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER Judging the Plausibility Machine

Source title Benchmarking the New Frontier

Source authors Dexter Horthy

Source year 2026

Confidence mid

Importance high

Format Online paper / PDF

"In the original SlopCodeBench evaluation of fifteen coding agents across thirty-six problems and 196 checkpoints, no evaluated agent fully solved any problem end to end."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER Judging the Plausibility Machine

Source title SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks

Source authors Gabriel Orlanski; Devjeet Roy; Alexander Yun; Changho Shin; Alex Gu; Albert Ge; Dyah Adila; Nicholas Roberts; Frederic Sala; Aws Albarghouthi

Source year 2026

Confidence high

Importance high

Format Online paper / PDF

"Software agents can pass individual correctness checks while producing code that becomes harder to maintain as requirements evolve."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER Judging the Plausibility Machine

Source title SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks

Source authors Gabriel Orlanski; Devjeet Roy; Alexander Yun; Changho Shin; Alex Gu; Albert Ge; Dyah Adila; Nicholas Roberts; Frederic Sala; Aws Albarghouthi

Source year 2026

Confidence high

Importance high

Format Online paper / PDF

"Mutation testing measures a test suite’s ability to catch deliberately introduced defects rather than establishing complete program correctness."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER Judging the Plausibility Machine

Source title Practical Mutation Testing at Scale: A view from Google

Source authors Goran Petrović; Marko Ivanković; Gordon Fraser; René Just

Source year 2021

Confidence high

Importance high

Format Online paper / PDF

"Mutation testing deliberately plants small artificial faults into code and checks whether the test suite detects them."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER Judging the Plausibility Machine

Source title Practical Mutation Testing at Scale: A view from Google

Source authors Goran Petrović; Marko Ivanković; Gordon Fraser; René Just

Source year 2021

Confidence high

Importance high

Format Online paper / PDF

"Dynamically generated coding evaluations can reduce dependence on fixed benchmark examples that language models may have memorized."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER Judging the Plausibility Machine

Source title DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation

Source authors Wenhao Hu; Jinhao Duan; Chunchen Wei; Li Zhang; Yue Zhang; Kaidi Xu

Source year 2025

Confidence high

Importance mid

Format Online paper / PDF

"Property-based testing checks general rules that should hold across many automatically generated inputs rather than relying only on manually selected examples."

View Source

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER Judging the Plausibility Machine

Source title QuickCheck: A lightweight tool for random testing of Haskell programs

Source authors Koen Claessen; John Hughes

Source year 2000

Confidence high

Importance high