Part VI · Decisions | from pretrained model to assistant, human preference data, Bradley–Terry reward models, RLHF with PPO and a KL penalty, the closed-form KL-regularised optimum, direct preference optimisation, GRPO and RL with verifiable rewards, reward hacking and Goodhart's law | reading: Christiano et al. (2017); Ouyang et al. (2022); Bai et al. (2022); Rafailov et al. (2023); Shao et al. (2024); DeepSeek-AI (2025); Gao, Schulman & Hilton (2023) | policy gradients, PPO and offline RL are in Lecture 22a
A language model trained to predict the next token (Lecture 12) can continue any text, but it does not reliably answer questions, follow instructions or decline harmful requests. Since 2022, the step that turns such a model into an assistant has been reinforcement learning: the model is a policy whose actions are tokens, a full response is an episode, and the reward comes from human preferences or from an automatic check. This page shows the pieces at a scale that runs on a laptop and is honest about what that scale can and cannot show.
By the end you should be able to fit a Bradley–Terry reward model; derive the closed-form optimum of KL-regularised reward maximisation and from it the DPO loss; describe GRPO and why it needs no critic; and recognise reward hacking in data.
The recipe of InstructGPT (Ouyang et al., 2022), building on Christiano et al. (2017) and Stiennon et al. (2020), has three steps:
Comparisons are used because people are far more consistent at saying which of two answers is better than at giving either a score out of ten.
Each response \(y\) to a prompt \(x\) has a latent reward \(r(x,y)\). The probability that \(y_w\) is preferred to \(y_l\) is
$$P(y_w\succ y_l\mid x)=\sigma\big(r(x,y_w)-r(x,y_l)\big),\qquad\sigma(z)=\frac1{1+e^{-z}}.$$A reward model \(r_\phi\) is fitted to labelled pairs by maximising the log-likelihood \(\sum\log\sigma\big(r_\phi(x,y_w)-r_\phi(x,y_l)\big)\): logistic regression on reward differences. Only differences are identified; adding a constant to every reward changes nothing.
Bai et al. (2022) released about 170,000 human comparisons collected while training Anthropic's assistants: in each, a crowdworker talked to a model and chose the better of two candidate replies. We use the "helpful-base" subset (43,835 training and 2,354 test comparisons) from the GitHub repository anthropics/hh-rlhf (MIT licence), keeping the pairs whose two conversations agree up to the final reply. The data contain upsetting content in places, as collected. The first reward model uses TF-IDF features of the reply (Lecture 13), which see words but not meaning; Section 2 replaces them by a sentence encoder that also reads the conversation. The policy experiments use a two-layer transformer trained in the lecture and a real small chat model, SmolLM2-135M-Instruct (Allal et al., 2025).
# A Bradley-Terry reward model trained on Anthropic's HH-RLHF helpfulness preferences import gzip, json, re, numpy as np, time, warnings; warnings.filterwarnings('ignore') from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.linear_model import LogisticRegression def load(path): out = [] for line in gzip.open(path): d = json.loads(line); p, c = d['chosen'].rsplit('\n\nAssistant:', 1); p2, r = d['rejected'].rsplit('\n\nAssistant:', 1) if p == p2: out.append((p, c.strip(), r.strip())) # keep pairs that share the conversation and differ in the last reply return out tr, te = load('data/hh/helpful-base_train.jsonl.gz'), load('data/hh/helpful-base_test.jsonl.gz') print(f"preference pairs: {len(tr):,} for training, {len(te):,} for testing (helpful-base; same conversation, two candidate final replies)") nlen = lambda s: np.log1p(len(s.split())) print(f"length alone: the chosen reply is the longer one in {np.mean([len(c.split()) > len(r.split()) for _, c, r in te]):.1%} of test pairs") vec = TfidfVectorizer(ngram_range=(1, 2), min_df=3, max_features=50_000, sublinear_tf=True).fit([c for _, c, _ in tr] + [r for _, _, r in tr]) def features(pairs): C = vec.transform([c for _, c, _ in pairs]); R = vec.transform([r for _, _, r in pairs]) L = np.array([[nlen(c) - nlen(r)] for _, c, r in pairs]) from scipy.sparse import hstack, csr_matrix return hstack([C - R, csr_matrix(L)]).tocsr() # Bradley-Terry: P(chosen > rejected) = sigmoid(r(chosen) - r(rejected)) with a linear reward r(y) = w . phi(y). # Fitting it is logistic regression on feature differences, with both orderings so the classes are balanced. Xd = features(tr); Xt = features(te) from scipy.sparse import vstack bt = LogisticRegression(C=1.0, max_iter=2000, fit_intercept=False).fit(vstack([Xd, -Xd]), np.r_[np.ones(len(tr)), np.zeros(len(tr))]) acc = np.mean(Xt @ bt.coef_[0] > 0); print(f"Bradley-Terry reward model (TF-IDF words and pairs + log length): test accuracy {acc:.1%}") w_len = bt.coef_[0][-1]; vocab = np.array(vec.get_feature_names_out()); wv = bt.coef_[0][:-1] print(f"weight on log length {w_len:+.2f}; most rewarded terms: {', '.join(vocab[np.argsort(-wv)[:8]])}") print(f"most penalised terms: {', '.join(vocab[np.argsort(wv)[:8]])}") reward = lambda texts: vec.transform(texts) @ wv + w_len * np.array([nlen(t) for t in texts])
preference pairs: 43,707 for training, 2,345 for testing (helpful-base; same conversation, two candidate final replies) length alone: the chosen reply is the longer one in 59.0% of test pairs Bradley-Terry reward model (TF-IDF words and pairs + log length): test accuracy 62.4% weight on log length +0.66; most rewarded terms: welcome, goodbye, tomatoes, chinese, flies, routine, pleasure, question but most penalised terms: https, human, http, not, question, humans, here, are quite
Human preferences are hard to predict from words alone. Trained on 43,707 pairs, the reward model picks the reply the crowdworker chose in 62.4% of 2,345 test pairs; always picking the longer reply already gets 59.0%. The model does use length (weight +0.66 on log length), and its most rewarded terms are polite closings such as "welcome" and "goodbye", while the most penalised include "https" and "human": links, and replies that run on into a new "Human:" turn, a quirk of how some replies in the data were cut. Real reward models are large transformers fine-tuned on the whole conversation and do considerably better; even so, crowdworkers often disagree with each other, which caps the accuracy any model can reach.
The RL step maximises the reward model's score, but not without limit: the policy \(\pi_\theta\) is penalised for drifting away from a reference policy \(\pi_{\text{ref}}\), usually the fine-tuned model. The objective, for each prompt \(x\), is
$$\max_\pi\;\mathbb E_{y\sim\pi(\cdot\mid x)}\big[r(x,y)\big]-\beta\,\mathrm{KL}\big(\pi(\cdot\mid x)\,\Vert\,\pi_{\text{ref}}(\cdot\mid x)\big).$$The penalty keeps the language fluent (the reference model knows how to write), limits how far the policy can exploit the reward model's errors, and keeps the output diverse. InstructGPT optimised it with PPO, adding the penalty to the reward of each token.
For \(\beta>0\), the maximiser of the objective above is
$$\pi^*(y\mid x)=\frac1{Z(x)}\,\pi_{\text{ref}}(y\mid x)\exp\!\Big(\frac{r(x,y)}\beta\Big),\qquad Z(x)=\sum_y\pi_{\text{ref}}(y\mid x)\exp\!\Big(\frac{r(x,y)}\beta\Big),$$and the maximum value is \(\beta\log Z(x)\).
Fix \(x\) and drop it from the notation. Divide the objective by \(\beta\) and combine the terms:
$$\frac1\beta\,\mathbb E_\pi[r]-\mathrm{KL}(\pi\,\Vert\,\pi_{\text{ref}})=\sum_y\pi(y)\log\frac{\pi_{\text{ref}}(y)e^{r(y)/\beta}}{\pi(y)}=\log Z-\mathrm{KL}(\pi\,\Vert\,\pi^*).$$The KL divergence is non-negative and zero only at \(\pi=\pi^*\). This is the same argument as Theorem 22.6, with the uniform distribution replaced by \(\pi_{\text{ref}}\). \(\square\)
The optimum reweights the reference model's probabilities by \(\exp(r/\beta)\). A large \(\beta\) leaves \(\pi_{\text{ref}}\) almost unchanged; a small one concentrates on the highest-reward responses. Responses that the reference model never produces stay at probability zero, whatever their reward.
Theorem 22.8 can also be read backwards. Any policy \(\pi\) defines an implicit reward for which it is optimal, \(r(x,y)=\beta\log\frac{\pi(y\mid x)}{\pi_{\text{ref}}(y\mid x)}+\beta\log Z(x)\). Rafailov et al. (2023) substituted this into the Bradley–Terry model, where the awkward \(Z(x)\) cancels.
Under the Bradley–Terry model with reward \(r\), and with \(\pi^*\) the KL-regularised optimum of Theorem 22.8,
$$P(y_w\succ y_l\mid x)=\sigma\Big(\beta\log\frac{\pi^*(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)}-\beta\log\frac{\pi^*(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\Big).$$So the policy can be fitted to the preference data directly, by minimising the DPO loss
$$\mathcal L_{\mathrm{DPO}}(\theta)=-\mathbb E_{(x,y_w,y_l)}\Big[\log\sigma\Big(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\Big)\Big],$$with no reward model and no sampling.
Take logarithms in Theorem 22.8 and solve for the reward:
$$r(x,y)=\beta\log\frac{\pi^*(y\mid x)}{\pi_{\text{ref}}(y\mid x)}+\beta\log Z(x).$$In the Bradley–Terry model only the difference \(r(x,y_w)-r(x,y_l)\) appears, and both responses share the prompt \(x\), so the \(\beta\log Z(x)\) terms cancel. Replacing the unknown \(\pi^*\) by \(\pi_\theta\) and maximising the likelihood of the observed preferences gives \(\mathcal L_{\mathrm{DPO}}\). \(\square\)
DPO turns RLHF into a classification problem: increase the log-probability ratio of preferred responses relative to rejected ones. It is simpler and more stable than PPO, and became a common choice for open models. The catch is that it learns only from the fixed comparison data; PPO-style methods generate new responses and get feedback on them, which can matter when the policy moves away from the data.
# KL-regularised optimisation against the reward model: the closed form pi* ~ pi_ref exp(r / beta), and how it gets hacked g = np.random.default_rng(0); P = 200; K = 8 own = [te[i] for i in range(P)] # 200 test conversations others = [[te[j] for j in row] for row in g.choice(np.arange(P, len(te)), (P, K - 2))] # plus 6 replies taken from other conversations cands = [[c, r] + [o[1 + int(g.integers(2))] for o in others[i]] for i, (_, c, r) in enumerate(own)] Rw = np.array([reward(cs) for cs in cands]); Rw = (Rw - Rw.mean()) / Rw.std() # standardised reward of every candidate pref = np.full(K, 1 / K) # reference policy: uniform over the 8 candidates print(f"{P} conversations x {K} candidate replies (the chosen and rejected reply of that conversation, and 6 replies from other conversations)") front = [] print("beta E[reward] KL(pi* || pi_ref) probability on this conversation's own two replies on the chosen reply") for beta in [10, 1, 0.3, 0.1, 0.03]: logits = np.log(pref) + Rw / beta; pi = np.exp(logits - logits.max(1, keepdims=True)); pi /= pi.sum(1, keepdims=True) kl = np.sum(pi * np.log(pi / pref), 1).mean(); front.append((beta, np.sum(pi * Rw, 1).mean(), kl, pi[:, :2].sum(1).mean())) print(f"{beta:<6} {np.sum(pi * Rw, 1).mean():+8.2f} {kl:12.2f} {pi[:, :2].sum(1).mean():37.1%} {pi[:, 0].mean():.1%}") # Direct preference optimisation: fit the policy itself to the preference pairs. With pi_theta(y) proportional to pi_ref(y) exp(s_theta(y)), # the DPO loss -log sigmoid(beta * [log pi/pi_ref (y_w) - log pi/pi_ref (y_l)]) depends only on s_theta(y_w) - s_theta(y_l): it is # Bradley-Terry with implicit reward beta * s_theta. Train it by gradient descent from s = 0 and compare with the reward model. beta = 0.5; theta = np.zeros(Xd.shape[1]); lr_ = 0.5 for it in range(300): z = beta * (Xd @ theta); grad = -(Xd.T @ (1 / (1 + np.exp(z)))) * beta / Xd.shape[0] + 1e-5 * theta theta -= lr_ / beta * grad * 20 acc_dpo = np.mean(Xt @ theta > 0); sp = np.corrcoef(Xt @ theta, Xt @ bt.coef_[0])[0, 1] print(f"\nDPO (beta = {beta}) on the same pairs: test accuracy of its implicit reward {acc_dpo:.1%} (reward model {acc:.1%}); " f"correlation of the two rewards' predictions on test pairs {sp:.2f}")
200 conversations x 8 candidate replies (the chosen and rejected reply of that conversation, and 6 replies from other conversations) beta E[reward] KL(pi* || pi_ref) probability on this conversation's own two replies on the chosen reply 10 +0.09 0.00 25.2% 12.9% 1 +0.70 0.32 27.0% 15.7% 0.3 +1.21 1.23 28.5% 17.8% 0.1 +1.32 1.81 28.2% 17.4% 0.03 +1.33 2.00 27.8% 17.2% DPO (beta = 0.5) on the same pairs: test accuracy of its implicit reward 63.4% (reward model 62.4%); correlation of the two rewards' predictions on test pairs 0.77
As \(\beta\) falls from 10 to 0.03, the optimal policy's expected reward rises from +0.09 to +1.33 standard deviations and its KL divergence from the reference from 0.00 to 2.00 nats (the maximum, \(\log 8\approx2.08\), would put all mass on one reply). But the probability it gives to the conversation's own two replies barely moves (25.2% to 27.8%, against 25% for the uniform reference): roughly three times in four it picks a reply written for a different conversation. The reward model never saw the conversation, so a long, polite reply to someone else's question scores as well as a relevant one, and optimisation exploits that. DPO fitted directly to the same pairs reaches 63.4% with its implicit reward (the reward model: 62.4%). With a linear model and a uniform reference the two methods fit the same Bradley–Terry model, as Theorem 22.9 predicts; their predictions correlate at 0.77 rather than 1 only because they are fitted differently (a stronger penalty for the reward model, a fixed number of gradient steps for DPO).
The obvious repair is a reward model that reads the conversation. The next box encodes the conversation and each reply with a pretrained sentence encoder, all-MiniLM-L6-v2 (Reimers and Gurevych, 2019; a 22-million-parameter transformer trained to map sentences with similar meanings to nearby unit vectors), and gives the reward model the reply's embedding, its elementwise product with the conversation's embedding (whose sum is their cosine similarity) and the log length.
# A reward model that reads the conversation: sentence embeddings (all-MiniLM-L6-v2) of the prompt and the reply from sentence_transformers import SentenceTransformer enc = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2', device='cpu'); enc.max_seq_length = 128 emb = lambda texts: enc.encode(texts, batch_size=64, normalize_embeddings=True) # unit vectors in R^384 sub = [tr[i] for i in np.random.default_rng(1).choice(len(tr), 12_000, replace=False)] # 12,000 training pairs (encoding is the slow part) t0 = time.time() Ps, Cs, Rs = emb([p[-500:] for p, _, _ in sub]), emb([c for _, c, _ in sub]), emb([r for _, _, r in sub]) Pt, Ct, Rt = emb([p[-500:] for p, _, _ in te]), emb([c for _, c, _ in te]), emb([r for _, _, r in te]) phi = lambda P, E, L: np.hstack([E, P * E, L[:, None]]) # reply, reply x prompt (its sum is the relevance), log length lens = lambda texts: np.array([nlen(t) for t in texts]) Lc, Lr, Ltc, Ltr = lens([c for _, c, _ in sub]), lens([r for _, _, r in sub]), lens([c for _, c, _ in te]), lens([r for _, _, r in te]) D = phi(Ps, Cs, Lc) - phi(Ps, Rs, Lr); Dt = phi(Pt, Ct, Ltc) - phi(Pt, Rt, Ltr) fit = lambda D: LogisticRegression(C=1.0, max_iter=3000, fit_intercept=False).fit(np.vstack([D, -D]), np.r_[np.ones(len(D)), np.zeros(len(D))]).coef_[0] we = fit(D) # the same pairs plus 12,000 made-up ones: the conversation's own chosen reply beats a reply written for another conversation perm = np.random.default_rng(2).permutation(len(sub)) wa = fit(np.vstack([D, phi(Ps, Cs, Lc) - phi(Ps, Cs[perm], Lc[perm])])) tf_sub = LogisticRegression(C=1.0, max_iter=2000, fit_intercept=False).fit(vstack([features(sub), -features(sub)]), np.r_[np.ones(len(sub)), np.zeros(len(sub))]) print(f"test accuracy, all trained on the same 12,000 human pairs: TF-IDF {np.mean(Xt @ tf_sub.coef_[0] > 0):.1%}; " f"embeddings {np.mean(Dt @ we > 0):.1%}; embeddings + made-up off-topic pairs {np.mean(Dt @ wa > 0):.1%} (encoding: {time.time() - t0:.0f} s)") # the same KL-regularised optimisation as before, against each reward model; the candidates are those of the previous box E8 = [emb(cs) for cs in cands]; L8 = [lens(cs) for cs in cands] def std_rewards(w): R = np.array([phi(np.repeat(Pt[i][None], len(cs), 0), E8[i], L8[i]) @ w for i, cs in enumerate(cands)]); return (R - R.mean()) / R.std() Re, Ra = std_rewards(we), std_rewards(wa) sm = lambda R, beta: (lambda z: z / z.sum(1, keepdims=True))(np.exp(R / beta - (R / beta).max(1, keepdims=True))) print("probability that the optimal policy picks one of the conversation's own two replies (uniform reference: 25%)") print("beta TF-IDF reward embedding reward embedding reward + off-topic pairs") for beta in [10, 1, 0.3, 0.1, 0.03]: print(f"{beta:<6} {sm(Rw, beta)[:, :2].sum(1).mean():12.1%} {sm(Re, beta)[:, :2].sum(1).mean():16.1%} {sm(Ra, beta)[:, :2].sum(1).mean():22.1%}")
test accuracy, all trained on the same 12,000 human pairs: TF-IDF 61.3%; embeddings 62.3%; embeddings + made-up off-topic pairs 60.2% (encoding: 812 s) probability that the optimal policy picks one of the conversation's own two replies (uniform reference: 25%) beta TF-IDF reward embedding reward embedding reward + off-topic pairs 10 25.2% 25.5% 28.6% 1 27.0% 28.8% 65.6% 0.3 28.5% 30.3% 94.0% 0.1 28.2% 30.0% 97.0% 0.03 27.8% 29.3% 97.1%
Reading the conversation is not enough. Trained on the same 12,000 human pairs, the embedding reward model is about as accurate as TF-IDF (62.3% against 61.3%), and the optimal policy still picks a reply written for another conversation most of the time (only 29.3% of its probability lands on the conversation's own replies at \(\beta=0.03\)). The reason is in the data: in every human pair both replies answer the same conversation, so relevance never varies between chosen and rejected, and the model has no reason to learn that it matters. A reward model learns what separates the pairs it is shown, nothing more. Add 12,000 made-up pairs in which the conversation's own reply beats one taken from another conversation, and the optimum puts 97.1% of its probability on the conversation's own replies, at a cost of 2.1 points of accuracy on the human pairs (60.2%). This is one reason practitioners add synthetic and adversarially collected comparisons to reward-model training; each such fix closes one blind spot without revealing the next.
Theorem 22.9 also works with a real language model in place of the linear one: the policy's own log-probabilities define the implicit reward, and no reward model is trained at all. The next box applies DPO to SmolLM2-135M-Instruct, a 135-million-parameter chat model, on short HH-RLHF pairs. To fit a CPU budget it trains only the top 10 of the model's 30 transformer blocks, for two passes over 320 pairs.
# DPO on a real (small) language model: SmolLM2-135M-Instruct, fine-tuned on HH-RLHF helpfulness pairs import torch, copy; torch.manual_seed(0) from transformers import AutoModelForCausalLM, AutoTokenizer from transformers.utils import logging as hf_logging; hf_logging.set_verbosity_error() name = 'HuggingFaceTB/SmolLM2-135M-Instruct' tok = AutoTokenizer.from_pretrained(name); ref = AutoModelForCausalLM.from_pretrained(name).eval() def messages(p): # "\n\nHuman: ...\n\nAssistant: ..." -> chat messages turns = re.split(r'\n\n(Human|Assistant): ', '\n\n' + p.strip())[1:] return [{'role': 'user' if r == 'Human' else 'assistant', 'content': c.strip()} for r, c in zip(turns[::2], turns[1::2])] def encode(p, y): a = tok(tok.apply_chat_template(messages(p), add_generation_prompt=True, tokenize=False), add_special_tokens=False).input_ids return a, tok(y + '<|im_end|>', add_special_tokens=False).input_ids def short(pairs, n, seed, L=128): # pairs whose prompt + longer reply fit in L tokens g = np.random.default_rng(seed); out = [] for i in g.permutation(len(pairs)): p, c, r = pairs[i]; (a, yc), (_, yr) = encode(p, c), encode(p, r) if len(a) + max(len(yc), len(yr)) <= L and c != r: out.append((a, yc, yr)) if len(out) == n: return out def logp(model, prompts, replies): # sum of log-probabilities of each reply given its prompt seqs = [a + y for a, y in zip(prompts, replies)]; T = max(map(len, seqs)) ids = torch.tensor([s + [tok.eos_token_id] * (T - len(s)) for s in seqs]); mask = torch.zeros(len(seqs), T - 1) for i, (a, y) in enumerate(zip(prompts, replies)): mask[i, len(a) - 1:len(a) - 1 + len(y)] = 1 lp = torch.log_softmax(model(ids, attention_mask=(torch.arange(T)[None] < torch.tensor([[len(s)] for s in seqs])).long()).logits[:, :-1], -1) return (lp.gather(-1, ids[:, 1:, None])[..., 0] * mask).sum(1) train, test = short(tr, 320, 0), short(te, 200, 1) with torch.no_grad(): ref_tr = [(logp(ref, [a], [yc]).item(), logp(ref, [a], [yr]).item()) for a, yc, yr in train] ref_te = torch.tensor([(logp(ref, [a], [yc]).item(), logp(ref, [a], [yr]).item()) for a, yc, yr in test]) pol = copy.deepcopy(ref).train() for n_, p_ in pol.named_parameters(): # train the top 10 of 30 blocks; freezing the rest halves the cost p_.requires_grad = any(f'layers.{k}.' in n_ for k in range(20, 30)) or n_.startswith('model.norm') opt = torch.optim.AdamW([p_ for p_ in pol.parameters() if p_.requires_grad], lr=1e-4) beta, B, t0 = 0.1, 4, time.time(); hist = [] for step in range(2 * len(train) // B): # two passes over the pairs idx = [i % len(train) for i in range(step * B, (step + 1) * B)]; bt_ = [train[i] for i in idx] lc = logp(pol, [a for a, _, _ in bt_], [yc for _, yc, _ in bt_]); lr_ = logp(pol, [a for a, _, _ in bt_], [yr for _, _, yr in bt_]) rc = torch.tensor([ref_tr[i][0] for i in idx]); rr = torch.tensor([ref_tr[i][1] for i in idx]) h = beta * ((lc - rc) - (lr_ - rr)); loss = -torch.nn.functional.logsigmoid(h).mean() loss.backward(); opt.step(); opt.zero_grad(); hist.append(((h > 0).float().mean().item(), loss.item())) pol.eval() with torch.no_grad(): pt = torch.stack([torch.cat([logp(pol, [a], [yc]), logp(pol, [a], [yr])]) for a, yc, yr in test]) imp = beta * (pt - ref_te); acc_imp = (imp[:, 0] > imp[:, 1]).float().mean().item() acc_ref = (ref_te[:, 0] > ref_te[:, 1]).float().mean().item() print(f"{len(train)} training pairs (2 passes, batch {B}, beta = {beta}), {len(test)} test pairs, all at most 128 tokens; {time.time() - t0:.0f} s") print(f"training pairs ranked correctly by the implicit reward: first 20 steps {np.mean([a for a, _ in hist[:20]]):.0%}, last 20 steps {np.mean([a for a, _ in hist[-20:]]):.0%}") print(f"test pairs: implicit reward of DPO prefers the chosen reply in {acc_imp:.1%}; " f"the reference model's own likelihood prefers it in {acc_ref:.1%}") # what changed in the replies: greedy answers to 30 held-out prompts, before and after prompts = [encode(p, '')[0] for p, _, _ in te[1000:1100] if len(encode(p, '')[0]) <= 96][:30] def reply(model, a): out = model.generate(torch.tensor([a]), max_new_tokens=80, do_sample=False, pad_token_id=tok.eos_token_id)[0, len(a):] return tok.decode(out, skip_special_tokens=True).strip() with torch.no_grad(): before = [reply(ref, a) for a in prompts]; after = [reply(pol, a) for a in prompts] wc = lambda xs: np.mean([len(x.split()) for x in xs]) print(f"greedy replies to {len(prompts)} held-out prompts: mean length {wc(before):.0f} words before DPO, {wc(after):.0f} after; " f"{np.mean([b != a for b, a in zip(before, after)]):.0%} of replies changed")
320 training pairs (2 passes, batch 4, beta = 0.1), 200 test pairs, all at most 128 tokens; 2364 s training pairs ranked correctly by the implicit reward: first 20 steps 49%, last 20 steps 91% test pairs: implicit reward of DPO prefers the chosen reply in 59.0%; the reference model's own likelihood prefers it in 49.0% greedy replies to 29 held-out prompts: mean length 56 words before DPO, 52 after; 97% of replies changed
DPO learns its training pairs: the implicit reward ranks 49% of them correctly during the first 20 steps and 91% during the last 20, which fall in the second pass, so part of that is memory. On 200 held-out pairs it prefers the human-chosen reply 59.0% of the time, where the reference model's own likelihood is at chance (49.0%). That is not far below the linear reward models of Section 1 (about 62%) with a hundredth of their training pairs, and with no reward model at all. Its greedy replies to 29 new prompts change in 97% of cases, but their mean length hardly moves (56 to 52 words, with replies cut off at 80 tokens). Full-scale preference training often makes replies longer, because human raters favour longer ones (Singhal et al., 2023); 320 short pairs and two passes are too little to show that.
For some tasks the reward needs no human at all: a maths answer can be checked, a program can be run against tests. RL against such verifiable rewards trains "reasoning" models that think step by step before answering.
For each prompt \(x\), sample a group of \(G\) responses \(y_1,\dots,y_G\) from the current policy and score them, \(r_1,\dots,r_G\). The advantage of each response is its reward standardised within the group:
$$\hat A_i=\frac{r_i-\operatorname{mean}(r_1,\dots,r_G)}{\operatorname{std}(r_1,\dots,r_G)} .$$Every token of \(y_i\) gets advantage \(\hat A_i\) in a PPO-style clipped objective, and a KL penalty towards \(\pi_{\text{ref}}\) is added. No value network is needed: the group mean is the baseline (Theorem 22.2).
Dropping the critic matters at the scale of language models, where a value network as large as the policy doubles the memory. The group baseline also fits verifiable rewards well: if a prompt is too hard (all answers wrong) or too easy (all right), every advantage is zero and the prompt is skipped. DeepSeek-R1 (DeepSeek-AI, 2025) applied GRPO with rule-based rewards for correct answers and correct formatting to a large pretrained model and reported long chains of thought, self-checking and backtracking emerging from RL alone, in the variant R1-Zero trained without supervised examples of reasoning.
# RL with a verifiable reward: GRPO on a tiny transformer that adds two-digit numbers import jax, jax.numpy as jnp V_, L_ = 12, 9 # tokens 0-9, '+' = 10, '=' = 11; sequence "a1 a2 + b1 b2 = s1 s2 s3" def encode(a, b): return [a // 10, a % 10, 10, b // 10, b % 10, 11] + [int(c) for c in f"{a + b:03d}"] pairs_all = [(a, b) for a in range(100) for b in range(100)]; g = np.random.default_rng(1); g.shuffle(pairs_all) sft_pairs, rl_pairs, test_pairs = pairs_all[:200], pairs_all[200:8000], pairs_all[8000:] # labelled / unlabelled prompts / test def tinit(key, d=64, layers=2): ks = jax.random.split(key, 4 + 6 * layers); n = lambda k, s: jax.random.normal(k, s) * 0.02 P = {'emb': n(ks[0], (V_, d)), 'pos': n(ks[1], (L_, d)), 'out': n(ks[2], (d, V_)), 'layers': []} for l in range(layers): k = ks[4 + 6 * l: 10 + 6 * l] P['layers'].append({'qkv': n(k[0], (d, 3 * d)), 'o': n(k[1], (d, d)), 'f1': n(k[2], (d, 4 * d)), 'f2': n(k[3], (4 * d, d)), 'g1': jnp.ones(d), 'g2': jnp.ones(d)}) return P def tfwd(P, x, heads=4): # causal transformer (Lecture 12): logits for the next token at every position T = x.shape[1]; h = P['emb'][x] + P['pos'][:T]; mask = jnp.tril(jnp.ones((T, T), bool)); d = h.shape[-1] ln = lambda z, gm: gm * (z - z.mean(-1, keepdims=True)) / jnp.sqrt(z.var(-1, keepdims=True) + 1e-5) for lp in P['layers']: q, k, v = jnp.split(ln(h, lp['g1']) @ lp['qkv'], 3, -1); sh = lambda z: z.reshape(z.shape[0], T, heads, d // heads).transpose(0, 2, 1, 3) att = jnp.where(mask, sh(q) @ sh(k).transpose(0, 1, 3, 2) / np.sqrt(d // heads), -1e9) h = h + (jax.nn.softmax(att, -1) @ sh(v)).transpose(0, 2, 1, 3).reshape(h.shape) @ lp['o'] h = h + jax.nn.gelu(ln(h, lp['g2']) @ lp['f1']) @ lp['f2'] return h @ P['out'] def answer_logp(P, seqs): # log-probability of each of the three answer tokens lp = jax.nn.log_softmax(tfwd(P, seqs[:, :-1]), -1)[:, 5:8]; return jnp.take_along_axis(lp, seqs[:, 6:9, None], -1)[..., 0] def jadam(p, gr, st, lr): m, v, t = st; t += 1 m = jax.tree.map(lambda m, g: 0.9 * m + 0.1 * g, m, gr); v = jax.tree.map(lambda v, g: 0.999 * v + 0.001 * g * g, v, gr) return jax.tree.map(lambda p, m, v: p - lr * (m / (1 - 0.9 ** t)) / (jnp.sqrt(v / (1 - 0.999 ** t)) + 1e-8), p, m, v), (m, v, t) sft_step = jax.jit(lambda P, st, seqs: jadam(P, jax.grad(lambda P: -answer_logp(P, seqs).mean())(P), st, 1e-3)) @jax.jit def sample(P, prompts, key): # draw the three answer digits one at a time seq = jnp.concatenate([prompts, jnp.zeros((len(prompts), 3), prompts.dtype)], 1) for t in range(6, 9): key, k = jax.random.split(key); seq = seq.at[:, t].set(jax.random.categorical(k, tfwd(P, seq[:, :t])[:, -1])) return seq @jax.jit def greedy(P, prompts): seq = jnp.concatenate([prompts, jnp.zeros((len(prompts), 3), prompts.dtype)], 1) for t in range(6, 9): seq = seq.at[:, t].set(tfwd(P, seq[:, :t])[:, -1].argmax(-1)) return seq def accuracy(P, prs): X = jnp.array([encode(a, b) for a, b in prs]); return float(jnp.mean(jnp.all(greedy(P, X[:, :6]) == X, 1))) def grpo_loss(P, Pref, seqs, adv, beta): lp = answer_logp(P, seqs); lr_ = jax.lax.stop_gradient(answer_logp(Pref, seqs)) kl = jnp.exp(lr_ - lp) - (lr_ - lp) - 1 # per-token estimate of KL(pi || pi_ref), always >= 0 return -jnp.mean(adv[:, None] * lp) + beta * jnp.mean(kl) grpo_step = jax.jit(lambda P, Pref, st, seqs, adv, beta: jadam(P, jax.grad(grpo_loss)(P, Pref, seqs, adv, beta), st, 3e-4)) def target_digits(prs): return np.array([[int(c) for c in f"{a + b:03d}"] for a, b in prs]) def grpo(P0, steps, reward_kind, beta=0.02, n_prompts=32, G=8, seed=0): P, st, key, gg = P0, (jax.tree.map(jnp.zeros_like, P0), jax.tree.map(jnp.zeros_like, P0), 0), jax.random.PRNGKey(seed), np.random.default_rng(seed) log = [] for it in range(steps): prs = [rl_pairs[i] for i in gg.integers(0, len(rl_pairs), n_prompts)]; prompts = np.repeat(np.array([encode(a, b)[:6] for a, b in prs]), G, 0) key, k = jax.random.split(key); seqs = sample(P, jnp.array(prompts), k); out = np.asarray(seqs[:, 6:]); tgt = np.repeat(target_digits(prs), G, 0) r = np.all(out == tgt, 1).astype(float) if reward_kind == 'exact' else (out[:, 2] == tgt[:, 2]).astype(float) # the verifier rg = r.reshape(n_prompts, G); adv = ((rg - rg.mean(1, keepdims=True)) / (rg.std(1, keepdims=True) + 1e-6)).ravel() # group-relative advantage P, st = grpo_step(P, P0, st, seqs, jnp.array(adv), beta); log.append(r.mean()) return P, np.array(log) t0 = time.time(); P = tinit(jax.random.PRNGKey(0)); st = (jax.tree.map(jnp.zeros_like, P), jax.tree.map(jnp.zeros_like, P), 0); Xs = np.array([encode(a, b) for a, b in sft_pairs]) for it in range(300): P, st = sft_step(P, st, jnp.array(Xs[g.integers(0, len(Xs), 128)])) P_sft = P; acc0 = accuracy(P_sft, test_pairs) print(f"supervised fine-tuning on {len(sft_pairs):,} worked examples (300 steps): exact accuracy on {len(test_pairs):,} unseen problems {acc0:.1%}") P_rl, log_rl = grpo(P_sft, 800, "exact") print(f"GRPO, reward = 1 if the whole answer is right: accuracy {accuracy(P_rl, test_pairs):.1%} after 800 steps " f"(32 unlabelled prompts x 8 samples per step; mean training reward {log_rl[:20].mean():.2f} -> {log_rl[-20:].mean():.2f})") P_hk, log_hk = grpo(P_sft, 800, "last digit") lastdig = lambda P: float(np.mean(np.asarray(greedy(P, jnp.array([encode(a, b)[:6] for a, b in test_pairs])))[:, 8] == target_digits(test_pairs)[:, 2])) print(f"GRPO, reward = 1 if only the last digit is right: training reward {log_hk[:20].mean():.2f} -> {log_hk[-20:].mean():.2f}; " f"last digit right on {lastdig(P_hk):.1%} of test problems (before: {lastdig(P_sft):.1%}); whole answer right {accuracy(P_hk, test_pairs):.1%}") print(f"time {time.time() - t0:.0f} s")
supervised fine-tuning on 200 worked examples (300 steps): exact accuracy on 2,000 unseen problems 57.6% GRPO, reward = 1 if the whole answer is right: accuracy 61.4% after 800 steps (32 unlabelled prompts x 8 samples per step; mean training reward 0.36 -> 0.51) GRPO, reward = 1 if only the last digit is right: training reward 0.71 -> 0.99; last digit right on 100.0% of test problems (before: 82.0%); whole answer right 23.6% time 533 s
After fine-tuning on 200 worked examples, the model adds correctly on 57.6% of unseen problems. GRPO then trains on prompts alone, with no answers given, only a checker: accuracy rises to 61.4% and the training reward from 0.36 to 0.51. The gain is modest because, as Exercise 4 explains, prompts on which all eight samples fail carry no signal. The hackable verifier rewards a correct last digit only. Its reward climbs from 0.71 to 0.99, and the last digit becomes right on 100.0% of test problems (from 82.0%), but whole answers are now right only 23.6% of the time, far below where training started. The policy learned exactly what it was paid for.
The same experiment on a real language model is closer to how reasoning models are trained: a pretrained chat model, a natural-language prompt, sampled answers in free text, and a program that checks them. The next box runs GRPO on SmolLM2-135M-Instruct with both verifiers. Each step samples eight answers to each of eight prompts (at temperature 0.7), and as in the DPO box only the top 10 blocks are trained.
# GRPO on the real model: SmolLM2-135M-Instruct learns two-digit addition from a checker, with an honest and a hackable verifier tok.padding_side = 'left' def ask(a, b): return tok.apply_chat_template([{'role': 'user', 'content': f'What is {a} + {b}? Reply with just the number.'}], add_generation_prompt=True, tokenize=False) last_int = lambda s: (lambda xs: int(xs[-1]) if xs else None)(re.findall(r'\d+', s)) honest = lambda s, a, b: float(last_int(s) == a + b) hackable = lambda s, a, b: float(last_int(s) is not None and last_int(s) % 10 == (a + b) % 10) # checks the last digit only held = np.random.default_rng(99).integers(10, 100, (128, 2)) def evaluate(model): enc_ = tok([ask(a, b) for a, b in held], return_tensors='pt', padding=True, add_special_tokens=False) with torch.no_grad(): out = model.generate(**enc_, max_new_tokens=12, do_sample=False, pad_token_id=tok.eos_token_id) txt = tok.batch_decode(out[:, enc_.input_ids.shape[1]:], skip_special_tokens=True) return (np.mean([honest(t, a, b) for t, (a, b) in zip(txt, held)]), np.mean([hackable(t, a, b) for t, (a, b) in zip(txt, held)]), np.mean([len(t.split()) for t in txt]), txt[:3]) def grpo(verifier, steps=30, n_prompts=8, G=8, beta=0.04, lr=3e-5, seed=0): g = np.random.default_rng(seed); torch.manual_seed(seed); pol = copy.deepcopy(ref).train(); log = [] for n_, p_ in pol.named_parameters(): p_.requires_grad = any(f'layers.{k}.' in n_ for k in range(20, 30)) or n_.startswith('model.norm') opt = torch.optim.AdamW([p_ for p_ in pol.parameters() if p_.requires_grad], lr=lr) for step in range(steps): prob = g.integers(10, 100, (n_prompts, 2)); enc_ = tok([ask(a, b) for a, b in prob for _ in range(G)], return_tensors='pt', padding=True, add_special_tokens=False) with torch.no_grad(): seq = pol.generate(**enc_, max_new_tokens=12, do_sample=True, temperature=0.7, pad_token_id=tok.eos_token_id) T0 = enc_.input_ids.shape[1]; comp = seq[:, T0:]; txt = tok.batch_decode(comp, skip_special_tokens=True) r = torch.tensor([verifier(t, a, b) for t, (a, b) in zip(txt, np.repeat(prob, G, 0))]).view(n_prompts, G) adv = ((r - r.mean(1, keepdim=True)) / (r.std(1, keepdim=True) + 1e-4)).view(-1) # group-relative advantage (Definition 22.5) cmask = torch.cumprod((comp != tok.eos_token_id).long(), 1); cmask = torch.cat([torch.ones_like(cmask[:, :1]), cmask[:, :-1]], 1).float() # up to the first end token att = torch.cat([enc_.attention_mask, torch.ones_like(comp)], 1) def tok_logp(model): lp = torch.log_softmax(model(seq, attention_mask=att).logits[:, T0 - 1:-1], -1); return lp.gather(-1, comp[..., None])[..., 0] with torch.no_grad(): lref = tok_logp(ref) lpol = tok_logp(pol); d = lref - lpol; kl = torch.exp(d) - d - 1 # the k3 estimate of KL(pi || pi_ref), per token per_tok = -(adv[:, None] * torch.exp(lpol - lpol.detach())) + beta * kl # on-policy: the ratio is 1, its gradient is the score loss = ((per_tok * cmask).sum(1) / cmask.sum(1)).mean(); loss.backward(); opt.step(); opt.zero_grad() log.append((r.mean().item(), ((kl * cmask).sum(1) / cmask.sum(1)).mean().item())) return pol.eval(), log t0 = time.time(); a0, h0, w0, ex0 = evaluate(ref) print(f"before GRPO (greedy, 128 held-out problems): whole answer right {a0:.1%}, last digit right {h0:.1%}, {w0:.1f} words per reply") print(f" e.g. {ex0}") for name, ver in [('honest verifier (whole answer)', honest), ('hackable verifier (last digit)', hackable)]: pol_v, log = grpo(ver); a1, h1, w1, ex1 = evaluate(pol_v) print(f"GRPO, {name}: training reward {np.mean([x for x, _ in log[:5]]):.2f} -> {np.mean([x for x, _ in log[-5:]]):.2f} over 30 steps; " f"held out: whole answer {a1:.1%}, last digit {h1:.1%}, {w1:.1f} words per reply; KL {log[-1][1]:.3f}") print(f" e.g. {ex1}") print(f"({time.time() - t0:.0f} s)")
before GRPO (greedy, 128 held-out problems): whole answer right 58.6%, last digit right 86.7%, 5.0 words per reply e.g. ['96 + 55 = 151', '78 + 60 = 138', '26 + 56 is 102.'] GRPO, honest verifier (whole answer): training reward 0.39 -> 0.51 over 30 steps; held out: whole answer 71.1%, last digit 93.8%, 5.0 words per reply; KL 0.008 e.g. ['96 + 55 = 151', '78 + 60 = 148.', '26 + 56 = 82.'] GRPO, hackable verifier (last digit): training reward 0.56 -> 0.75 over 30 steps; held out: whole answer 68.8%, last digit 92.2%, 5.0 words per reply; KL 0.009 e.g. ['96 + 55 = 151', '78 + 60 = 138.', '26 + 56 = 82'] (4575 s)
Before GRPO the chat model gets 58.6% of held-out sums right with greedy decoding (it often restates the problem, "96 + 55 = 151", and the checker reads the last number). Thirty GRPO steps with the honest verifier move the training reward from 0.39 to 0.51 and held-out accuracy to 71.1%, while the policy stays close to the reference (KL 0.008 nats per token). The hackable verifier, unlike in the tiny transformer, is not hacked: its reward rises from 0.56 to 0.75, and whole-answer accuracy rises with it, to 68.8%. For a model that already adds, the easiest way to get the last digit right is to get the sum right; the tiny transformer, trained from scratch on 200 examples, found a cheaper route. Reward hacking needs an exploit that is easier for the policy than the task, and enough optimisation to find it: thirty steps on a pretrained model may not be enough, and a run of a few hundred steps is the right test.
A reward model is a statistical model of human judgement, fitted on a few thousand or a few hundred thousand comparisons, and it is wrong in places. Optimisation finds those places. This is Goodhart's law (Goodhart, 1975), in Marilyn Strathern's phrasing: "when a measure becomes a target, it ceases to be a good measure." Both experiments above show it.
Real systems show the same patterns at scale. RLHF-trained models write longer answers than their reward actually justifies (Singhal et al., 2023), agree with the user's stated opinion (sycophancy; Sharma et al., 2024), and, with code rewards, can learn to special-case the tests. Gao, Schulman and Hilton (2023) measured the effect directly: as a policy is optimised harder against a proxy reward model, the score of a separate "gold" reward model first rises and then falls, with the turning point predictable from the KL divergence travelled. Defences include the KL penalty, larger and more varied preference data, ensembles of reward models, rewards that check what matters, and evaluation by people on fresh prompts.
Christiano et al. (2017) showed that an agent could learn Atari games and simulated robot tasks from a few hundred (robot tasks) to a few thousand (Atari) human comparisons of short video clips. Stiennon et al. (2020) applied the same idea to summarisation, and OpenAI's InstructGPT (Ouyang et al., 2022) used it to make GPT-3 follow instructions: labellers preferred the outputs of a 1.3-billion-parameter InstructGPT model to those of the 175-billion-parameter GPT-3. ChatGPT followed in November 2022. Anthropic's Constitutional AI (Bai et al., 2022) replaced many human labels with AI feedback guided by written principles (RLAIF). DPO (2023) removed the explicit reward model, and DeepSeek-R1 (January 2025) showed that RL with simple verifiable rewards can produce strong step-by-step reasoning, a recipe quickly adopted across the field.
Every widely used chatbot is trained with some form of preference or reward optimisation after pretraining. The same tools now reach beyond chat: verifiable-reward RL trains models for mathematics, programming and formal proof, where answers can be checked automatically, and RL from AI feedback scales supervision where human labels are scarce. These methods also raise the questions of this lecture in sharper form: whose preferences are learned, how a model trained to please differs from one trained to be correct, and how to detect reward hacking in systems too capable to check by hand. Lecture 23 returns to how to read the claims made about such systems.
Only differences \(r(x,y_w)-r(x,y_l)\) between responses to the same prompt enter the model, and \(c(x)\) cancels in them. So a reward model's scores are meaningful only relative to other responses to the same prompt: a score of 2 on one prompt and 1 on another says nothing about which response is better. This is also why \(Z(x)\) cancels in Theorem 22.9.
\(\pi^*\propto(\tfrac12e,\tfrac12)\), so \(\pi^*=(e/(e+1),1/(e+1))\approx(0.731,0.269)\). Then
$$\mathrm{KL}=0.731\log\frac{0.731}{0.5}+0.269\log\frac{0.269}{0.5}\approx0.278-0.167=0.111\ \text{nats}.$$The expected reward rises from 0.5 to 0.731 at a KL cost of about 0.11 nats.
Write \(h=\beta(\log\frac{\pi_\theta(y_w)}{\pi_{\text{ref}}(y_w)}-\log\frac{\pi_\theta(y_l)}{\pi_{\text{ref}}(y_l)})\). The loss is \(-\log\sigma(h)\), whose derivative with respect to \(h\) is \(-(1-\sigma(h))=-\sigma(-h)\). So
$$\frac{\partial\mathcal L}{\partial\log\pi_\theta(y_w)}=-\beta\,\sigma(-h).$$The gradient is largest when \(h\) is very negative, that is, when the model currently prefers the rejected response more than the reference does. Pairs the model already gets right contribute little: DPO is a weighted, self-correcting classifier.
All advantages are zero, so the prompt contributes nothing. For a prompt the model never solves, no learning signal arrives at all: verifiable-reward RL can only sharpen abilities the model already shows occasionally. Remedies include a curriculum from easy to hard prompts, larger groups (more chances of one success), partial-credit rewards, and supervised examples for problems out of reach. Our addition model, which still fails every sample on some prompts, learns slowly partly for this reason.
(a) Add features of the prompt–reply pair, such as the TF-IDF similarity between the last human turn and the reply, or interaction features of prompt and reply words. (b) Use a model that encodes the whole conversation, such as a fine-tuned transformer (the standard approach). Either way, replies written for other conversations would score lower, and the optimised policy would put more probability on the conversation's own replies; how much depends on how well the new features capture relevance.
A larger β holds the policy closer to the fine-tuned model: learning from the honest verifier slows down, and so does the collapse under the hackable one. At large β the hacked policy keeps more of its real accuracy but gains less on the proxy. The KL penalty is a brake, not a cure: it limits how far the policy can follow a flawed reward, but it does not tell it which direction is flawed.
Each project uses this lecture's methods and public data or small open models, and has a clear result.
Data. HH-RLHF helpful-base comparisons: Bai et al. (2022), github.com/anthropics/hh-rlhf (MIT licence). The arithmetic problems are generated in the lecture. Pretrained models from the Hugging Face Hub: all-MiniLM-L6-v2 (sentence-transformers/all-MiniLM-L6-v2, Apache 2.0) and SmolLM2-135M-Instruct (HuggingFaceTB/SmolLM2-135M-Instruct, Apache 2.0). The DPO and GRPO runs on SmolLM2 are sized for two CPU cores (a few hundred pairs, thirty GRPO steps, the top third of the network trained), so they show the mechanisms at work on a real chat model, not results of a full-scale training run.
numbers in the text not printed by a code box are computed in verify_L22b.py, figures and widget data in figs_L22b.py; code-box outputs are pasted from actual runs (numpy, scikit-learn and JAX on two CPU cores)