"Language models can generate plausible but false statements because learning the statistical distribution of language is not the same as verifying facts."
OpenAI explains how next-token pretraining can learn highly regular linguistic patterns while struggling with arbitrary, low-frequency facts. This supports the book's distinction between synthesising fluent prose and producing statements that have been independently grounded in reality.
View Source