"One of the problems surrounding deeply ingrained language biases is that language is lopsided in a specific, Zipfian way."
Piantadosi reviews the highly unequal frequency distribution of natural language, in which a relatively small number of words appear extraordinarily often and many others occupy a long tail. This provides the statistical foundation for the manuscript's argument about uneven representation in the written material from which language models learn.
View Source