Preference Optimization
RLHF, DPO and RLAIF explained by what each one removes: how preference pairs teach judgement that demonstrations cannot, and why reward hacking and length bias are predictable.
Last updated
After this section you can
- Explain why ranking two answers collects signal that writing one ideal answer cannot
- Place RLHF, DPO and RLAIF on the two axes that separate them — the optimiser and the labeller
- Recognise reward hacking, length bias and over-refusal as predictable artefacts of this stage
Preference Optimization
Supervised fine-tuning teaches a model to imitate an answer. Preference optimization teaches it which of two answers people actually wanted — which turns out to be both cheaper to collect and able to express things a demonstration cannot.
Writing one ideal answer is hard and slow. Looking at two answers and saying which is better is easy and fast — and it captures qualities nobody can write down cleanly: the right amount of hedging, when to refuse, how long an answer should be, whether a tone lands. This is the stage that turns an instruction-follower into something people want to talk to, and it is why a model’s personality is a training artefact, not a prompt artefact.