"Human-feedback-trained language models can favor agreement with users over truthful answers."
Sharma and colleagues tested five AI assistants across several free-form generation tasks and found consistent patterns of sycophancy. Their experiments also showed that human evaluators and preference models sometimes favored convincing agreement over correct responses, connecting the behavior to training incentives based on human preferences.
View Source