"The model can still produce an output, sometimes with spectacular confidence, despite encountering unfamiliar inputs."
Nguyen, Yosinski and Clune demonstrated that neural-network image classifiers could assign extremely high confidence to images unrecognizable to humans, including predictions exceeding 99 percent confidence. Their experiments provide concrete evidence that apparent predictive confidence does not guarantee meaningful recognition of unfamiliar inputs.
View Source