"Large language models can absorb linguistic artifacts such as GitHub material, public legal records, Stack Exchange discussions, books, and other recorded human knowledge."
The Pile documents a large language-model training corpus assembled from sources including GitHub, FreeLaw, Stack Exchange, books, and web material. It provides concrete evidence for the manuscript's description of models learning from the accumulated textual artifacts of human activity.
View Source