"Language models learn unevenly from common and rare linguistic patterns, with infrequently encountered tokens receiving less effective generalization."
Diehl Martinez and colleagues document frequency bias in language-model pre-training and show how infrequently encountered tokens can be disadvantaged under standard maximum-likelihood training. Their research supports the section's distinction between frequently repeated formulations and the comparatively poorly represented linguistic long tail.
View Source