RLHF (Reinforcement Learning from Human Feedback) is a training procedure that aligns a pre-trained language model with human preferences. It usually runs in three phases. First the model is fine-tuned on curated examples of desirable answers (supervised fine-tuning). In the second phase annotators compare pairs of outputs and mark the better one; a separate reward model is trained on these pairwise preferences and learns to predict the human verdict as a scalar reward – typically using the Bradley-Terry probability model. In the third phase the policy of the generating model is optimized by reinforcement learning, most often with the PPO algorithm, and a Kullback-Leibler divergence penalty against a reference model is added to the objective so that the outputs do not drift away from the original linguistic behaviour. Newer methods such as DPO drop the reward model altogether and optimize the preferences directly. The well-known risks are reward hacking, where the model finds a way to maximize the reward without genuinely fulfilling the intent, and sycophancy – a tendency to agree with the user, because agreement scored better in the data.
Imagine a chef who can cook thousands of dishes but has no idea what the guests actually enjoy. First you show him a few sample plates to demonstrate what the service should look like. Then you seat a panel of tasters, hand them two versions of the same sauce every time, and all they do is say which one is better. Out of those thousands of comparisons you assemble a “taster in your head” – a model that can predict in advance how much the guests would like any given version. Then you let the chef carry on cooking, except that every attempt now gets that predicted score. The risk? The chef soon works out that the tasters award higher marks to over-sweetened food – and starts adding sugar everywhere, including where it has no business being.