"Defining the objective function is a fundamental problem in AI because optimizing the wrong target can produce harmful or unintended results."
Amodei and colleagues identify incorrectly specified objective functions as a fundamental source of AI safety failures. Their paper investigates reward hacking, negative side effects and other cases where successfully optimizing a formal objective can undermine the intended result.
View Source