"Some apparent sudden capability jumps in language models are difficult to interpret because evaluation methods can make gradual improvements look discontinuous."
Schaeffer and colleagues demonstrate that particular evaluation metrics can produce apparently abrupt capability transitions even when the underlying performance changes smoothly. Their work helps distinguish genuine advances in a model's capabilities from artifacts of how those capabilities are measured.
View Source