"In standard RLHF, a separate reward model learns human preferences from ranked outputs and supplies an optimisation signal to the language model."
The InstructGPT paper documents collecting human rankings, fitting a reward model to those preferences and using reinforcement learning to optimise the language model against the learned reward. This establishes the technical foundation for the book's discussion of human judgments being converted into a computational proxy.
View Source