"The generator-discriminator gap helps explain why evaluating candidate outputs can be easier than producing a good answer from scratch in some tasks."
Karpathy's detailed explanation of the model-training pipeline discusses reinforcement learning from human feedback and the relationship between generating candidate answers and evaluating them. This provides context for how human judgments can guide optimisation without human evaluators having to author every candidate response themselves.