"Language models can have more difficulty retrieving relevant information from the middle of a long input than from its beginning or end."
Liu and colleagues evaluated language models on long-context question answering and retrieval. They documented substantial position-dependent performance differences, showing how important information can become harder for models to use when buried inside a large context.
View Source