Subjects › AGI Agents (6-week course)
W5 · Reward models, DPO & reward hacking
Bradley–Terry preferences, learned reward models, DPO, best-of-n, and when higher reward stops meaning better.
The idea
A reward model learns a scalar score so that P(A preferred over B) = σ(r_A − r_B) — the Bradley–Terry model. RLHF optimises against it; DPO skips the explicit reward model and trains the policy directly on preference pairs, with a β that controls how far it may move from the reference model. Optimising hard against any learned reward invites reward hacking — Goodhart's law, and the 'wrong model' failure from Week 2. In your repo: Week05_Alignment_Safety/reward_model.py, data/preferences.jsonl; Exercises 5.1 (dpo_loss) and 5.4 (safe_best_of_n).
Lesson
No lesson built yet. The worked solutions below are the material it would be assembled from.
Questions (5)
| Question | Type | Difficulty | Drift | Accuracy |
|---|---|---|---|---|
| A reward model scores response A at r_A = 1.5 and response B at r_B = 0.5. Under the Bradley–Terry model used … | numeric | Easy -0.93 | +0.27 | 0% (1) |
| Exercise 5.4. `safe_best_of_n` receives four candidates with these reward scores and safety verdicts: A 3.1 (u… | mcq | Easy -0.80 | — | — (0) |
| Exercise 5.1. For one preference pair the sequence log-probabilities are: policy chosen −10, policy rejected −… | numeric | Hard 0.50 | — | — (0) |
| `reward_model.py` ends with a candidate that is nothing but hedge words ('I am not sure, maybe, it might, roug… | mcq | Hard 1.10 | — | — (0) |
| The `dpo_loss` docstring asks: 'DPO has no reward model. Where did the reward go?' Answer in two or three sent… | short | Very hard 1.60 | — | — (0) |