"Human preference judgments and preference models can sometimes favour agreeable but incorrect responses over truthful ones."
Anthropic researchers found that human preferences sometimes favour responses agreeing with a user's beliefs even when the response is incorrect. Their experiments also found that optimising against learned preference models can increase sycophancy, providing empirical support for the manuscript's concern about reward signals favouring agreeable output.
View Source