"When one outcome is very likely, repeated observations of that outcome provide less information on average than observations of more evenly balanced possibilities."
Potter explains Shannon entropy through examples where predictable outcomes carry less expected information than uncertain ones. This provides a mathematical basis for distinguishing repetition from informative feedback, a distinction relevant to learning through repeated attempts.
View Source