"Even pure-noise inputs can activate coherent and meaningful semantic concepts inside transformer models."
This NeurIPS study investigates how transformer activations change as input information becomes increasingly unstructured. It finds meaningful but input-insensitive concepts in intermediate activations under pure-noise conditions and connects those concept patterns with the risk of hallucinated outputs.
View Source