Machine Learning · Lecture 22b

Lecture 22b
Reinforcement Learning IV: RL for Language Models

Part VI · Decisions  |  from pretrained model to assistant, human preference data, Bradley–Terry reward models, RLHF with PPO and a KL penalty, the closed-form KL-regularised optimum, direct preference optimisation, GRPO and RL with verifiable rewards, reward hacking and Goodhart's law  |  reading: Christiano et al. (2017); Ouyang et al. (2022); Bai et al. (2022); Rafailov et al. (2023); Shao et al. (2024); DeepSeek-AI (2025); Gao, Schulman & Hilton (2023)  |  policy gradients, PPO and offline RL are in Lecture 22a

Cartoon: the essay markerA teacher holds a stack of essays and says she gives marks for length. A student hands in an enormous scroll and says it is not very good, but it is very long. Longer essays tend to bebetter, so I mark length. It isn't very good,but it is very long.
When a measure becomes a target, it ceases to be a good measure: optimising a learned reward finds its blind spots
In this lecture
  1. From pretrained model to assistant; reward models from preferences
  2. RLHF, the KL-regularised optimum and DPO
  3. RL with verifiable rewards: GRPO and reasoning models
  4. Reward hacking and Goodhart's law
  5. Summary, exercises, projects, reading

A language model trained to predict the next token (Lecture 12) can continue any text, but it does not reliably answer questions, follow instructions or decline harmful requests. Since 2022, the step that turns such a model into an assistant has been reinforcement learning: the model is a policy whose actions are tokens, a full response is an episode, and the reward comes from human preferences or from an automatic check. This page shows the pieces at a scale that runs on a laptop and is honest about what that scale can and cannot show.

By the end you should be able to fit a Bradley–Terry reward model; derive the closed-form optimum of KL-regularised reward maximisation and from it the DPO loss; describe GRPO and why it needs no critic; and recognise reward hacking in data.

1 — From pretrained model to assistant; reward models from preferences

The recipe of InstructGPT (Ouyang et al., 2022), building on Christiano et al. (2017) and Stiennon et al. (2020), has three steps:

  1. supervised fine-tuning: train on demonstrations of good responses written by people;
  2. reward modelling: show people pairs of responses to the same prompt, ask which is better, and train a model to predict their choices;
  3. reinforcement learning: optimise the language model against the reward model with PPO (Lecture 22a), keeping it close to the fine-tuned model.

Comparisons are used because people are far more consistent at saying which of two answers is better than at giving either a score out of ten.

definition 22.4 — the Bradley–Terry model

Each response \(y\) to a prompt \(x\) has a latent reward \(r(x,y)\). The probability that \(y_w\) is preferred to \(y_l\) is

$$P(y_w\succ y_l\mid x)=\sigma\big(r(x,y_w)-r(x,y_l)\big),\qquad\sigma(z)=\frac1{1+e^{-z}}.$$

A reward model \(r_\phi\) is fitted to labelled pairs by maximising the log-likelihood \(\sum\log\sigma\big(r_\phi(x,y_w)-r_\phi(x,y_l)\big)\): logistic regression on reward differences. Only differences are identified; adding a constant to every reward changes nothing.

From reward difference to preference

The Bradley–Terry model turns a difference in reward into a probability of preference. A difference of 0 is a coin toss; a difference of 3 makes the better response win about 95% of the time.

data — Anthropic HH-RLHF (helpfulness)

Bai et al. (2022) released about 170,000 human comparisons collected while training Anthropic's assistants: in each, a crowdworker talked to a model and chose the better of two candidate replies. We use the "helpful-base" subset (43,835 training and 2,354 test comparisons) from the GitHub repository anthropics/hh-rlhf (MIT licence), keeping the pairs whose two conversations agree up to the final reply. The data contain upsetting content in places, as collected. The first reward model uses TF-IDF features of the reply (Lecture 13), which see words but not meaning; Section 2 replaces them by a sentence encoder that also reads the conversation. The policy experiments use a two-layer transformer trained in the lecture and a real small chat model, SmolLM2-135M-Instruct (Allal et al., 2025).

code — a Bradley–Terry reward model on human preference pairs
# A Bradley-Terry reward model trained on Anthropic's HH-RLHF helpfulness preferences
import gzip, json, re, numpy as np, time, warnings; warnings.filterwarnings('ignore')
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
def load(path):
    out = []
    for line in gzip.open(path):
        d = json.loads(line); p, c = d['chosen'].rsplit('\n\nAssistant:', 1); p2, r = d['rejected'].rsplit('\n\nAssistant:', 1)
        if p == p2: out.append((p, c.strip(), r.strip()))                    # keep pairs that share the conversation and differ in the last reply
    return out
tr, te = load('data/hh/helpful-base_train.jsonl.gz'), load('data/hh/helpful-base_test.jsonl.gz')
print(f"preference pairs: {len(tr):,} for training, {len(te):,} for testing (helpful-base; same conversation, two candidate final replies)")
nlen = lambda s: np.log1p(len(s.split()))
print(f"length alone: the chosen reply is the longer one in {np.mean([len(c.split()) > len(r.split()) for _, c, r in te]):.1%} of test pairs")
vec = TfidfVectorizer(ngram_range=(1, 2), min_df=3, max_features=50_000, sublinear_tf=True).fit([c for _, c, _ in tr] + [r for _, _, r in tr])
def features(pairs):
    C = vec.transform([c for _, c, _ in pairs]); R = vec.transform([r for _, _, r in pairs])
    L = np.array([[nlen(c) - nlen(r)] for _, c, r in pairs])
    from scipy.sparse import hstack, csr_matrix
    return hstack([C - R, csr_matrix(L)]).tocsr()
# Bradley-Terry: P(chosen > rejected) = sigmoid(r(chosen) - r(rejected)) with a linear reward r(y) = w . phi(y).
# Fitting it is logistic regression on feature differences, with both orderings so the classes are balanced.
Xd = features(tr); Xt = features(te)
from scipy.sparse import vstack
bt = LogisticRegression(C=1.0, max_iter=2000, fit_intercept=False).fit(vstack([Xd, -Xd]), np.r_[np.ones(len(tr)), np.zeros(len(tr))])
acc = np.mean(Xt @ bt.coef_[0] > 0); print(f"Bradley-Terry reward model (TF-IDF words and pairs + log length): test accuracy {acc:.1%}")
w_len = bt.coef_[0][-1]; vocab = np.array(vec.get_feature_names_out()); wv = bt.coef_[0][:-1]
print(f"weight on log length {w_len:+.2f}; most rewarded terms: {', '.join(vocab[np.argsort(-wv)[:8]])}")
print(f"most penalised terms: {', '.join(vocab[np.argsort(wv)[:8]])}")
reward = lambda texts: vec.transform(texts) @ wv + w_len * np.array([nlen(t) for t in texts])
preference pairs: 43,707 for training, 2,345 for testing (helpful-base; same conversation, two candidate final replies)
length alone: the chosen reply is the longer one in 59.0% of test pairs
Bradley-Terry reward model (TF-IDF words and pairs + log length): test accuracy 62.4%
weight on log length +0.66; most rewarded terms: welcome, goodbye, tomatoes, chinese, flies, routine, pleasure, question but
most penalised terms: https, human, http, not, question, humans, here, are quite

Human preferences are hard to predict from words alone. Trained on 43,707 pairs, the reward model picks the reply the crowdworker chose in 62.4% of 2,345 test pairs; always picking the longer reply already gets 59.0%. The model does use length (weight +0.66 on log length), and its most rewarded terms are polite closings such as "welcome" and "goodbye", while the most penalised include "https" and "human": links, and replies that run on into a new "Human:" turn, a quirk of how some replies in the data were cut. Real reward models are large transformers fine-tuned on the whole conversation and do considerably better; even so, crowdworkers often disagree with each other, which caps the accuracy any model can reach.

2 — RLHF, the KL-regularised optimum and DPO

The RL step maximises the reward model's score, but not without limit: the policy \(\pi_\theta\) is penalised for drifting away from a reference policy \(\pi_{\text{ref}}\), usually the fine-tuned model. The objective, for each prompt \(x\), is

$$\max_\pi\;\mathbb E_{y\sim\pi(\cdot\mid x)}\big[r(x,y)\big]-\beta\,\mathrm{KL}\big(\pi(\cdot\mid x)\,\Vert\,\pi_{\text{ref}}(\cdot\mid x)\big).$$

The penalty keeps the language fluent (the reference model knows how to write), limits how far the policy can exploit the reward model's errors, and keeps the output diverse. InstructGPT optimised it with PPO, adding the penalty to the reward of each token.

theorem 22.8 — the KL-regularised optimum in closed form

For \(\beta>0\), the maximiser of the objective above is

$$\pi^*(y\mid x)=\frac1{Z(x)}\,\pi_{\text{ref}}(y\mid x)\exp\!\Big(\frac{r(x,y)}\beta\Big),\qquad Z(x)=\sum_y\pi_{\text{ref}}(y\mid x)\exp\!\Big(\frac{r(x,y)}\beta\Big),$$

and the maximum value is \(\beta\log Z(x)\).

Fix \(x\) and drop it from the notation. Divide the objective by \(\beta\) and combine the terms:

$$\frac1\beta\,\mathbb E_\pi[r]-\mathrm{KL}(\pi\,\Vert\,\pi_{\text{ref}})=\sum_y\pi(y)\log\frac{\pi_{\text{ref}}(y)e^{r(y)/\beta}}{\pi(y)}=\log Z-\mathrm{KL}(\pi\,\Vert\,\pi^*).$$

The KL divergence is non-negative and zero only at \(\pi=\pi^*\). This is the same argument as Theorem 22.6, with the uniform distribution replaced by \(\pi_{\text{ref}}\). \(\square\)

The optimum reweights the reference model's probabilities by \(\exp(r/\beta)\). A large \(\beta\) leaves \(\pi_{\text{ref}}\) almost unchanged; a small one concentrates on the highest-reward responses. Responses that the reference model never produces stay at probability zero, whatever their reward.

The KL-regularised policy

Five possible responses, with a reference policy (grey) and rewards you can set. The optimum (red) is the reference reweighted by \(\exp(r/\beta)\). Lower \(\beta\) to let the reward dominate; give the "off-topic but long" response a high reward, as a flawed reward model might, and watch the policy follow it.

Theorem 22.8 can also be read backwards. Any policy \(\pi\) defines an implicit reward for which it is optimal, \(r(x,y)=\beta\log\frac{\pi(y\mid x)}{\pi_{\text{ref}}(y\mid x)}+\beta\log Z(x)\). Rafailov et al. (2023) substituted this into the Bradley–Terry model, where the awkward \(Z(x)\) cancels.

theorem 22.9 — direct preference optimisation (Rafailov et al., 2023)

Under the Bradley–Terry model with reward \(r\), and with \(\pi^*\) the KL-regularised optimum of Theorem 22.8,

$$P(y_w\succ y_l\mid x)=\sigma\Big(\beta\log\frac{\pi^*(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)}-\beta\log\frac{\pi^*(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\Big).$$

So the policy can be fitted to the preference data directly, by minimising the DPO loss

$$\mathcal L_{\mathrm{DPO}}(\theta)=-\mathbb E_{(x,y_w,y_l)}\Big[\log\sigma\Big(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\Big)\Big],$$

with no reward model and no sampling.

Take logarithms in Theorem 22.8 and solve for the reward:

$$r(x,y)=\beta\log\frac{\pi^*(y\mid x)}{\pi_{\text{ref}}(y\mid x)}+\beta\log Z(x).$$

In the Bradley–Terry model only the difference \(r(x,y_w)-r(x,y_l)\) appears, and both responses share the prompt \(x\), so the \(\beta\log Z(x)\) terms cancel. Replacing the unknown \(\pi^*\) by \(\pi_\theta\) and maximising the likelihood of the observed preferences gives \(\mathcal L_{\mathrm{DPO}}\). \(\square\)

DPO turns RLHF into a classification problem: increase the log-probability ratio of preferred responses relative to rejected ones. It is simpler and more stable than PPO, and became a common choice for open models. The catch is that it learns only from the fixed comparison data; PPO-style methods generate new responses and get feedback on them, which can matter when the policy moves away from the data.

code — the KL-regularised optimum against the reward model; DPO on the same pairs
# KL-regularised optimisation against the reward model: the closed form pi* ~ pi_ref exp(r / beta), and how it gets hacked
g = np.random.default_rng(0); P = 200; K = 8
own = [te[i] for i in range(P)]                                              # 200 test conversations
others = [[te[j] for j in row] for row in g.choice(np.arange(P, len(te)), (P, K - 2))]        # plus 6 replies taken from other conversations
cands = [[c, r] + [o[1 + int(g.integers(2))] for o in others[i]] for i, (_, c, r) in enumerate(own)]
Rw = np.array([reward(cs) for cs in cands]); Rw = (Rw - Rw.mean()) / Rw.std()   # standardised reward of every candidate
pref = np.full(K, 1 / K)                                                     # reference policy: uniform over the 8 candidates
print(f"{P} conversations x {K} candidate replies (the chosen and rejected reply of that conversation, and 6 replies from other conversations)")
front = []
print("beta    E[reward]   KL(pi* || pi_ref)   probability on this conversation's own two replies   on the chosen reply")
for beta in [10, 1, 0.3, 0.1, 0.03]:
    logits = np.log(pref) + Rw / beta; pi = np.exp(logits - logits.max(1, keepdims=True)); pi /= pi.sum(1, keepdims=True)
    kl = np.sum(pi * np.log(pi / pref), 1).mean(); front.append((beta, np.sum(pi * Rw, 1).mean(), kl, pi[:, :2].sum(1).mean()))
    print(f"{beta:<6}  {np.sum(pi * Rw, 1).mean():+8.2f}    {kl:12.2f}         {pi[:, :2].sum(1).mean():37.1%}            {pi[:, 0].mean():.1%}")
# Direct preference optimisation: fit the policy itself to the preference pairs. With pi_theta(y) proportional to pi_ref(y) exp(s_theta(y)),
# the DPO loss -log sigmoid(beta * [log pi/pi_ref (y_w) - log pi/pi_ref (y_l)]) depends only on s_theta(y_w) - s_theta(y_l): it is
# Bradley-Terry with implicit reward beta * s_theta. Train it by gradient descent from s = 0 and compare with the reward model.
beta = 0.5; theta = np.zeros(Xd.shape[1]); lr_ = 0.5
for it in range(300):
    z = beta * (Xd @ theta); grad = -(Xd.T @ (1 / (1 + np.exp(z)))) * beta / Xd.shape[0] + 1e-5 * theta
    theta -= lr_ / beta * grad * 20
acc_dpo = np.mean(Xt @ theta > 0); sp = np.corrcoef(Xt @ theta, Xt @ bt.coef_[0])[0, 1]
print(f"\nDPO (beta = {beta}) on the same pairs: test accuracy of its implicit reward {acc_dpo:.1%} (reward model {acc:.1%}); "
      f"correlation of the two rewards' predictions on test pairs {sp:.2f}")
200 conversations x 8 candidate replies (the chosen and rejected reply of that conversation, and 6 replies from other conversations)
beta    E[reward]   KL(pi* || pi_ref)   probability on this conversation's own two replies   on the chosen reply
10         +0.09            0.00                                         25.2%            12.9%
1          +0.70            0.32                                         27.0%            15.7%
0.3        +1.21            1.23                                         28.5%            17.8%
0.1        +1.32            1.81                                         28.2%            17.4%
0.03       +1.33            2.00                                         27.8%            17.2%

DPO (beta = 0.5) on the same pairs: test accuracy of its implicit reward 63.4% (reward model 62.4%); correlation of the two rewards' predictions on test pairs 0.77

As \(\beta\) falls from 10 to 0.03, the optimal policy's expected reward rises from +0.09 to +1.33 standard deviations and its KL divergence from the reference from 0.00 to 2.00 nats (the maximum, \(\log 8\approx2.08\), would put all mass on one reply). But the probability it gives to the conversation's own two replies barely moves (25.2% to 27.8%, against 25% for the uniform reference): roughly three times in four it picks a reply written for a different conversation. The reward model never saw the conversation, so a long, polite reply to someone else's question scores as well as a relevant one, and optimisation exploits that. DPO fitted directly to the same pairs reaches 63.4% with its implicit reward (the reward model: 62.4%). With a linear model and a uniform reference the two methods fit the same Bradley–Terry model, as Theorem 22.9 predicts; their predictions correlate at 0.77 rather than 1 only because they are fitted differently (a stronger penalty for the reward model, a fixed number of gradient steps for DPO).

The obvious repair is a reward model that reads the conversation. The next box encodes the conversation and each reply with a pretrained sentence encoder, all-MiniLM-L6-v2 (Reimers and Gurevych, 2019; a 22-million-parameter transformer trained to map sentences with similar meanings to nearby unit vectors), and gives the reward model the reply's embedding, its elementwise product with the conversation's embedding (whose sum is their cosine similarity) and the log length.

code — a reward model that reads the conversation, and what it takes to make it care
# A reward model that reads the conversation: sentence embeddings (all-MiniLM-L6-v2) of the prompt and the reply
from sentence_transformers import SentenceTransformer
enc = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2', device='cpu'); enc.max_seq_length = 128
emb = lambda texts: enc.encode(texts, batch_size=64, normalize_embeddings=True)           # unit vectors in R^384
sub = [tr[i] for i in np.random.default_rng(1).choice(len(tr), 12_000, replace=False)]    # 12,000 training pairs (encoding is the slow part)
t0 = time.time()
Ps, Cs, Rs = emb([p[-500:] for p, _, _ in sub]), emb([c for _, c, _ in sub]), emb([r for _, _, r in sub])
Pt, Ct, Rt = emb([p[-500:] for p, _, _ in te]), emb([c for _, c, _ in te]), emb([r for _, _, r in te])
phi = lambda P, E, L: np.hstack([E, P * E, L[:, None]])        # reply, reply x prompt (its sum is the relevance), log length
lens = lambda texts: np.array([nlen(t) for t in texts])
Lc, Lr, Ltc, Ltr = lens([c for _, c, _ in sub]), lens([r for _, _, r in sub]), lens([c for _, c, _ in te]), lens([r for _, _, r in te])
D = phi(Ps, Cs, Lc) - phi(Ps, Rs, Lr); Dt = phi(Pt, Ct, Ltc) - phi(Pt, Rt, Ltr)
fit = lambda D: LogisticRegression(C=1.0, max_iter=3000, fit_intercept=False).fit(np.vstack([D, -D]), np.r_[np.ones(len(D)), np.zeros(len(D))]).coef_[0]
we = fit(D)
# the same pairs plus 12,000 made-up ones: the conversation's own chosen reply beats a reply written for another conversation
perm = np.random.default_rng(2).permutation(len(sub))
wa = fit(np.vstack([D, phi(Ps, Cs, Lc) - phi(Ps, Cs[perm], Lc[perm])]))
tf_sub = LogisticRegression(C=1.0, max_iter=2000, fit_intercept=False).fit(vstack([features(sub), -features(sub)]), np.r_[np.ones(len(sub)), np.zeros(len(sub))])
print(f"test accuracy, all trained on the same 12,000 human pairs: TF-IDF {np.mean(Xt @ tf_sub.coef_[0] > 0):.1%}; "
      f"embeddings {np.mean(Dt @ we > 0):.1%}; embeddings + made-up off-topic pairs {np.mean(Dt @ wa > 0):.1%}  (encoding: {time.time() - t0:.0f} s)")
# the same KL-regularised optimisation as before, against each reward model; the candidates are those of the previous box
E8 = [emb(cs) for cs in cands]; L8 = [lens(cs) for cs in cands]
def std_rewards(w):
    R = np.array([phi(np.repeat(Pt[i][None], len(cs), 0), E8[i], L8[i]) @ w for i, cs in enumerate(cands)]); return (R - R.mean()) / R.std()
Re, Ra = std_rewards(we), std_rewards(wa)
sm = lambda R, beta: (lambda z: z / z.sum(1, keepdims=True))(np.exp(R / beta - (R / beta).max(1, keepdims=True)))
print("probability that the optimal policy picks one of the conversation's own two replies (uniform reference: 25%)")
print("beta     TF-IDF reward   embedding reward   embedding reward + off-topic pairs")
for beta in [10, 1, 0.3, 0.1, 0.03]:
    print(f"{beta:<6} {sm(Rw, beta)[:, :2].sum(1).mean():12.1%} {sm(Re, beta)[:, :2].sum(1).mean():16.1%} {sm(Ra, beta)[:, :2].sum(1).mean():22.1%}")
test accuracy, all trained on the same 12,000 human pairs: TF-IDF 61.3%; embeddings 62.3%; embeddings + made-up off-topic pairs 60.2%  (encoding: 812 s)
probability that the optimal policy picks one of the conversation's own two replies (uniform reference: 25%)
beta     TF-IDF reward   embedding reward   embedding reward + off-topic pairs
10            25.2%            25.5%                  28.6%
1             27.0%            28.8%                  65.6%
0.3           28.5%            30.3%                  94.0%
0.1           28.2%            30.0%                  97.0%
0.03          27.8%            29.3%                  97.1%

Reading the conversation is not enough. Trained on the same 12,000 human pairs, the embedding reward model is about as accurate as TF-IDF (62.3% against 61.3%), and the optimal policy still picks a reply written for another conversation most of the time (only 29.3% of its probability lands on the conversation's own replies at \(\beta=0.03\)). The reason is in the data: in every human pair both replies answer the same conversation, so relevance never varies between chosen and rejected, and the model has no reason to learn that it matters. A reward model learns what separates the pairs it is shown, nothing more. Add 12,000 made-up pairs in which the conversation's own reply beats one taken from another conversation, and the optimum puts 97.1% of its probability on the conversation's own replies, at a cost of 2.1 points of accuracy on the human pairs (60.2%). This is one reason practitioners add synthetic and adversarially collected comparisons to reward-model training; each such fix closes one blind spot without revealing the next.

Theorem 22.9 also works with a real language model in place of the linear one: the policy's own log-probabilities define the implicit reward, and no reward model is trained at all. The next box applies DPO to SmolLM2-135M-Instruct, a 135-million-parameter chat model, on short HH-RLHF pairs. To fit a CPU budget it trains only the top 10 of the model's 30 transformer blocks, for two passes over 320 pairs.

code — DPO on a real small language model
# DPO on a real (small) language model: SmolLM2-135M-Instruct, fine-tuned on HH-RLHF helpfulness pairs
import torch, copy; torch.manual_seed(0)
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.utils import logging as hf_logging; hf_logging.set_verbosity_error()
name = 'HuggingFaceTB/SmolLM2-135M-Instruct'
tok = AutoTokenizer.from_pretrained(name); ref = AutoModelForCausalLM.from_pretrained(name).eval()
def messages(p):                                             # "\n\nHuman: ...\n\nAssistant: ..." -> chat messages
    turns = re.split(r'\n\n(Human|Assistant): ', '\n\n' + p.strip())[1:]
    return [{'role': 'user' if r == 'Human' else 'assistant', 'content': c.strip()} for r, c in zip(turns[::2], turns[1::2])]
def encode(p, y):
    a = tok(tok.apply_chat_template(messages(p), add_generation_prompt=True, tokenize=False), add_special_tokens=False).input_ids
    return a, tok(y + '<|im_end|>', add_special_tokens=False).input_ids
def short(pairs, n, seed, L=128):                            # pairs whose prompt + longer reply fit in L tokens
    g = np.random.default_rng(seed); out = []
    for i in g.permutation(len(pairs)):
        p, c, r = pairs[i]; (a, yc), (_, yr) = encode(p, c), encode(p, r)
        if len(a) + max(len(yc), len(yr)) <= L and c != r: out.append((a, yc, yr))
        if len(out) == n: return out
def logp(model, prompts, replies):                           # sum of log-probabilities of each reply given its prompt
    seqs = [a + y for a, y in zip(prompts, replies)]; T = max(map(len, seqs))
    ids = torch.tensor([s + [tok.eos_token_id] * (T - len(s)) for s in seqs]); mask = torch.zeros(len(seqs), T - 1)
    for i, (a, y) in enumerate(zip(prompts, replies)): mask[i, len(a) - 1:len(a) - 1 + len(y)] = 1
    lp = torch.log_softmax(model(ids, attention_mask=(torch.arange(T)[None] < torch.tensor([[len(s)] for s in seqs])).long()).logits[:, :-1], -1)
    return (lp.gather(-1, ids[:, 1:, None])[..., 0] * mask).sum(1)
train, test = short(tr, 320, 0), short(te, 200, 1)
with torch.no_grad():
    ref_tr = [(logp(ref, [a], [yc]).item(), logp(ref, [a], [yr]).item()) for a, yc, yr in train]
    ref_te = torch.tensor([(logp(ref, [a], [yc]).item(), logp(ref, [a], [yr]).item()) for a, yc, yr in test])
pol = copy.deepcopy(ref).train()
for n_, p_ in pol.named_parameters():                        # train the top 10 of 30 blocks; freezing the rest halves the cost
    p_.requires_grad = any(f'layers.{k}.' in n_ for k in range(20, 30)) or n_.startswith('model.norm')
opt = torch.optim.AdamW([p_ for p_ in pol.parameters() if p_.requires_grad], lr=1e-4)
beta, B, t0 = 0.1, 4, time.time(); hist = []
for step in range(2 * len(train) // B):                      # two passes over the pairs
    idx = [i % len(train) for i in range(step * B, (step + 1) * B)]; bt_ = [train[i] for i in idx]
    lc = logp(pol, [a for a, _, _ in bt_], [yc for _, yc, _ in bt_]); lr_ = logp(pol, [a for a, _, _ in bt_], [yr for _, _, yr in bt_])
    rc = torch.tensor([ref_tr[i][0] for i in idx]); rr = torch.tensor([ref_tr[i][1] for i in idx])
    h = beta * ((lc - rc) - (lr_ - rr)); loss = -torch.nn.functional.logsigmoid(h).mean()
    loss.backward(); opt.step(); opt.zero_grad(); hist.append(((h > 0).float().mean().item(), loss.item()))
pol.eval()
with torch.no_grad():
    pt = torch.stack([torch.cat([logp(pol, [a], [yc]), logp(pol, [a], [yr])]) for a, yc, yr in test])
imp = beta * (pt - ref_te); acc_imp = (imp[:, 0] > imp[:, 1]).float().mean().item()
acc_ref = (ref_te[:, 0] > ref_te[:, 1]).float().mean().item()
print(f"{len(train)} training pairs (2 passes, batch {B}, beta = {beta}), {len(test)} test pairs, all at most 128 tokens; {time.time() - t0:.0f} s")
print(f"training pairs ranked correctly by the implicit reward: first 20 steps {np.mean([a for a, _ in hist[:20]]):.0%}, last 20 steps {np.mean([a for a, _ in hist[-20:]]):.0%}")
print(f"test pairs: implicit reward of DPO prefers the chosen reply in {acc_imp:.1%}; "
      f"the reference model's own likelihood prefers it in {acc_ref:.1%}")
# what changed in the replies: greedy answers to 30 held-out prompts, before and after
prompts = [encode(p, '')[0] for p, _, _ in te[1000:1100] if len(encode(p, '')[0]) <= 96][:30]
def reply(model, a):
    out = model.generate(torch.tensor([a]), max_new_tokens=80, do_sample=False, pad_token_id=tok.eos_token_id)[0, len(a):]
    return tok.decode(out, skip_special_tokens=True).strip()
with torch.no_grad(): before = [reply(ref, a) for a in prompts]; after = [reply(pol, a) for a in prompts]
wc = lambda xs: np.mean([len(x.split()) for x in xs])
print(f"greedy replies to {len(prompts)} held-out prompts: mean length {wc(before):.0f} words before DPO, {wc(after):.0f} after; "
      f"{np.mean([b != a for b, a in zip(before, after)]):.0%} of replies changed")
320 training pairs (2 passes, batch 4, beta = 0.1), 200 test pairs, all at most 128 tokens; 2364 s
training pairs ranked correctly by the implicit reward: first 20 steps 49%, last 20 steps 91%
test pairs: implicit reward of DPO prefers the chosen reply in 59.0%; the reference model's own likelihood prefers it in 49.0%
greedy replies to 29 held-out prompts: mean length 56 words before DPO, 52 after; 97% of replies changed

DPO learns its training pairs: the implicit reward ranks 49% of them correctly during the first 20 steps and 91% during the last 20, which fall in the second pass, so part of that is memory. On 200 held-out pairs it prefers the human-chosen reply 59.0% of the time, where the reference model's own likelihood is at chance (49.0%). That is not far below the linear reward models of Section 1 (about 62%) with a hundredth of their training pairs, and with no reward model at all. Its greedy replies to 29 new prompts change in 97% of cases, but their mean length hardly moves (56 to 52 words, with replies cut off at 80 tokens). Full-scale preference training often makes replies longer, because human raters favour longer ones (Singhal et al., 2023); 320 short pairs and two passes are too little to show that.

3 — RL with verifiable rewards: GRPO and reasoning models

For some tasks the reward needs no human at all: a maths answer can be checked, a program can be run against tests. RL against such verifiable rewards trains "reasoning" models that think step by step before answering.

definition 22.5 — group relative policy optimisation (Shao et al., 2024)

For each prompt \(x\), sample a group of \(G\) responses \(y_1,\dots,y_G\) from the current policy and score them, \(r_1,\dots,r_G\). The advantage of each response is its reward standardised within the group:

$$\hat A_i=\frac{r_i-\operatorname{mean}(r_1,\dots,r_G)}{\operatorname{std}(r_1,\dots,r_G)} .$$

Every token of \(y_i\) gets advantage \(\hat A_i\) in a PPO-style clipped objective, and a KL penalty towards \(\pi_{\text{ref}}\) is added. No value network is needed: the group mean is the baseline (Theorem 22.2).

Dropping the critic matters at the scale of language models, where a value network as large as the policy doubles the memory. The group baseline also fits verifiable rewards well: if a prompt is too hard (all answers wrong) or too easy (all right), every advantage is zero and the prompt is skipped. DeepSeek-R1 (DeepSeek-AI, 2025) applied GRPO with rule-based rewards for correct answers and correct formatting to a large pretrained model and reported long chains of thought, self-checking and backtracking emerging from RL alone, in the variant R1-Zero trained without supervised examples of reasoning.

code — GRPO on a tiny transformer that learns to add, with an honest and a hackable verifier
# RL with a verifiable reward: GRPO on a tiny transformer that adds two-digit numbers
import jax, jax.numpy as jnp
V_, L_ = 12, 9                                                     # tokens 0-9, '+' = 10, '=' = 11; sequence "a1 a2 + b1 b2 = s1 s2 s3"
def encode(a, b): return [a // 10, a % 10, 10, b // 10, b % 10, 11] + [int(c) for c in f"{a + b:03d}"]
pairs_all = [(a, b) for a in range(100) for b in range(100)]; g = np.random.default_rng(1); g.shuffle(pairs_all)
sft_pairs, rl_pairs, test_pairs = pairs_all[:200], pairs_all[200:8000], pairs_all[8000:]          # labelled / unlabelled prompts / test
def tinit(key, d=64, layers=2):
    ks = jax.random.split(key, 4 + 6 * layers); n = lambda k, s: jax.random.normal(k, s) * 0.02
    P = {'emb': n(ks[0], (V_, d)), 'pos': n(ks[1], (L_, d)), 'out': n(ks[2], (d, V_)), 'layers': []}
    for l in range(layers):
        k = ks[4 + 6 * l: 10 + 6 * l]
        P['layers'].append({'qkv': n(k[0], (d, 3 * d)), 'o': n(k[1], (d, d)), 'f1': n(k[2], (d, 4 * d)), 'f2': n(k[3], (4 * d, d)),
                            'g1': jnp.ones(d), 'g2': jnp.ones(d)})
    return P
def tfwd(P, x, heads=4):                                            # causal transformer (Lecture 12): logits for the next token at every position
    T = x.shape[1]; h = P['emb'][x] + P['pos'][:T]; mask = jnp.tril(jnp.ones((T, T), bool)); d = h.shape[-1]
    ln = lambda z, gm: gm * (z - z.mean(-1, keepdims=True)) / jnp.sqrt(z.var(-1, keepdims=True) + 1e-5)
    for lp in P['layers']:
        q, k, v = jnp.split(ln(h, lp['g1']) @ lp['qkv'], 3, -1); sh = lambda z: z.reshape(z.shape[0], T, heads, d // heads).transpose(0, 2, 1, 3)
        att = jnp.where(mask, sh(q) @ sh(k).transpose(0, 1, 3, 2) / np.sqrt(d // heads), -1e9)
        h = h + (jax.nn.softmax(att, -1) @ sh(v)).transpose(0, 2, 1, 3).reshape(h.shape) @ lp['o']
        h = h + jax.nn.gelu(ln(h, lp['g2']) @ lp['f1']) @ lp['f2']
    return h @ P['out']
def answer_logp(P, seqs):                                           # log-probability of each of the three answer tokens
    lp = jax.nn.log_softmax(tfwd(P, seqs[:, :-1]), -1)[:, 5:8]; return jnp.take_along_axis(lp, seqs[:, 6:9, None], -1)[..., 0]
def jadam(p, gr, st, lr):
    m, v, t = st; t += 1
    m = jax.tree.map(lambda m, g: 0.9 * m + 0.1 * g, m, gr); v = jax.tree.map(lambda v, g: 0.999 * v + 0.001 * g * g, v, gr)
    return jax.tree.map(lambda p, m, v: p - lr * (m / (1 - 0.9 ** t)) / (jnp.sqrt(v / (1 - 0.999 ** t)) + 1e-8), p, m, v), (m, v, t)
sft_step = jax.jit(lambda P, st, seqs: jadam(P, jax.grad(lambda P: -answer_logp(P, seqs).mean())(P), st, 1e-3))
@jax.jit
def sample(P, prompts, key):                                        # draw the three answer digits one at a time
    seq = jnp.concatenate([prompts, jnp.zeros((len(prompts), 3), prompts.dtype)], 1)
    for t in range(6, 9):
        key, k = jax.random.split(key); seq = seq.at[:, t].set(jax.random.categorical(k, tfwd(P, seq[:, :t])[:, -1]))
    return seq
@jax.jit
def greedy(P, prompts):
    seq = jnp.concatenate([prompts, jnp.zeros((len(prompts), 3), prompts.dtype)], 1)
    for t in range(6, 9): seq = seq.at[:, t].set(tfwd(P, seq[:, :t])[:, -1].argmax(-1))
    return seq
def accuracy(P, prs):
    X = jnp.array([encode(a, b) for a, b in prs]); return float(jnp.mean(jnp.all(greedy(P, X[:, :6]) == X, 1)))
def grpo_loss(P, Pref, seqs, adv, beta):
    lp = answer_logp(P, seqs); lr_ = jax.lax.stop_gradient(answer_logp(Pref, seqs))
    kl = jnp.exp(lr_ - lp) - (lr_ - lp) - 1                         # per-token estimate of KL(pi || pi_ref), always >= 0
    return -jnp.mean(adv[:, None] * lp) + beta * jnp.mean(kl)
grpo_step = jax.jit(lambda P, Pref, st, seqs, adv, beta: jadam(P, jax.grad(grpo_loss)(P, Pref, seqs, adv, beta), st, 3e-4))
def target_digits(prs): return np.array([[int(c) for c in f"{a + b:03d}"] for a, b in prs])
def grpo(P0, steps, reward_kind, beta=0.02, n_prompts=32, G=8, seed=0):
    P, st, key, gg = P0, (jax.tree.map(jnp.zeros_like, P0), jax.tree.map(jnp.zeros_like, P0), 0), jax.random.PRNGKey(seed), np.random.default_rng(seed)
    log = []
    for it in range(steps):
        prs = [rl_pairs[i] for i in gg.integers(0, len(rl_pairs), n_prompts)]; prompts = np.repeat(np.array([encode(a, b)[:6] for a, b in prs]), G, 0)
        key, k = jax.random.split(key); seqs = sample(P, jnp.array(prompts), k); out = np.asarray(seqs[:, 6:]); tgt = np.repeat(target_digits(prs), G, 0)
        r = np.all(out == tgt, 1).astype(float) if reward_kind == 'exact' else (out[:, 2] == tgt[:, 2]).astype(float)   # the verifier
        rg = r.reshape(n_prompts, G); adv = ((rg - rg.mean(1, keepdims=True)) / (rg.std(1, keepdims=True) + 1e-6)).ravel()   # group-relative advantage
        P, st = grpo_step(P, P0, st, seqs, jnp.array(adv), beta); log.append(r.mean())
    return P, np.array(log)
t0 = time.time(); P = tinit(jax.random.PRNGKey(0)); st = (jax.tree.map(jnp.zeros_like, P), jax.tree.map(jnp.zeros_like, P), 0); Xs = np.array([encode(a, b) for a, b in sft_pairs])
for it in range(300): P, st = sft_step(P, st, jnp.array(Xs[g.integers(0, len(Xs), 128)]))
P_sft = P; acc0 = accuracy(P_sft, test_pairs)
print(f"supervised fine-tuning on {len(sft_pairs):,} worked examples (300 steps): exact accuracy on {len(test_pairs):,} unseen problems {acc0:.1%}")
P_rl, log_rl = grpo(P_sft, 800, "exact")
print(f"GRPO, reward = 1 if the whole answer is right: accuracy {accuracy(P_rl, test_pairs):.1%} after 800 steps "
      f"(32 unlabelled prompts x 8 samples per step; mean training reward {log_rl[:20].mean():.2f} -> {log_rl[-20:].mean():.2f})")
P_hk, log_hk = grpo(P_sft, 800, "last digit")
lastdig = lambda P: float(np.mean(np.asarray(greedy(P, jnp.array([encode(a, b)[:6] for a, b in test_pairs])))[:, 8] == target_digits(test_pairs)[:, 2]))
print(f"GRPO, reward = 1 if only the last digit is right: training reward {log_hk[:20].mean():.2f} -> {log_hk[-20:].mean():.2f}; "
      f"last digit right on {lastdig(P_hk):.1%} of test problems (before: {lastdig(P_sft):.1%}); whole answer right {accuracy(P_hk, test_pairs):.1%}")
print(f"time {time.time() - t0:.0f} s")
supervised fine-tuning on 200 worked examples (300 steps): exact accuracy on 2,000 unseen problems 57.6%
GRPO, reward = 1 if the whole answer is right: accuracy 61.4% after 800 steps (32 unlabelled prompts x 8 samples per step; mean training reward 0.36 -> 0.51)
GRPO, reward = 1 if only the last digit is right: training reward 0.71 -> 0.99; last digit right on 100.0% of test problems (before: 82.0%); whole answer right 23.6%
time 533 s
020040060080000.250.50.751GRPO stepmean reward of the sampleshonest verifier (whole answer right)hackable verifier (last digit right)
Mean reward of the eight samples per prompt during GRPO, smoothed over 25 steps. Both start from the same fine-tuned model; the hackable verifier pays for a correct last digit alone. With only 32 prompts per step the honest reward is noisy, and it dips near the end.

After fine-tuning on 200 worked examples, the model adds correctly on 57.6% of unseen problems. GRPO then trains on prompts alone, with no answers given, only a checker: accuracy rises to 61.4% and the training reward from 0.36 to 0.51. The gain is modest because, as Exercise 4 explains, prompts on which all eight samples fail carry no signal. The hackable verifier rewards a correct last digit only. Its reward climbs from 0.71 to 0.99, and the last digit becomes right on 100.0% of test problems (from 82.0%), but whole answers are now right only 23.6% of the time, far below where training started. The policy learned exactly what it was paid for.

The same experiment on a real language model is closer to how reasoning models are trained: a pretrained chat model, a natural-language prompt, sampled answers in free text, and a program that checks them. The next box runs GRPO on SmolLM2-135M-Instruct with both verifiers. Each step samples eight answers to each of eight prompts (at temperature 0.7), and as in the DPO box only the top 10 blocks are trained.

code — GRPO on a real small language model, with the honest and the hackable verifier
# GRPO on the real model: SmolLM2-135M-Instruct learns two-digit addition from a checker, with an honest and a hackable verifier
tok.padding_side = 'left'
def ask(a, b): return tok.apply_chat_template([{'role': 'user', 'content': f'What is {a} + {b}? Reply with just the number.'}],
                                              add_generation_prompt=True, tokenize=False)
last_int = lambda s: (lambda xs: int(xs[-1]) if xs else None)(re.findall(r'\d+', s))
honest = lambda s, a, b: float(last_int(s) == a + b)
hackable = lambda s, a, b: float(last_int(s) is not None and last_int(s) % 10 == (a + b) % 10)   # checks the last digit only
held = np.random.default_rng(99).integers(10, 100, (128, 2))
def evaluate(model):
    enc_ = tok([ask(a, b) for a, b in held], return_tensors='pt', padding=True, add_special_tokens=False)
    with torch.no_grad(): out = model.generate(**enc_, max_new_tokens=12, do_sample=False, pad_token_id=tok.eos_token_id)
    txt = tok.batch_decode(out[:, enc_.input_ids.shape[1]:], skip_special_tokens=True)
    return (np.mean([honest(t, a, b) for t, (a, b) in zip(txt, held)]), np.mean([hackable(t, a, b) for t, (a, b) in zip(txt, held)]),
            np.mean([len(t.split()) for t in txt]), txt[:3])
def grpo(verifier, steps=30, n_prompts=8, G=8, beta=0.04, lr=3e-5, seed=0):
    g = np.random.default_rng(seed); torch.manual_seed(seed); pol = copy.deepcopy(ref).train(); log = []
    for n_, p_ in pol.named_parameters(): p_.requires_grad = any(f'layers.{k}.' in n_ for k in range(20, 30)) or n_.startswith('model.norm')
    opt = torch.optim.AdamW([p_ for p_ in pol.parameters() if p_.requires_grad], lr=lr)
    for step in range(steps):
        prob = g.integers(10, 100, (n_prompts, 2)); enc_ = tok([ask(a, b) for a, b in prob for _ in range(G)], return_tensors='pt', padding=True, add_special_tokens=False)
        with torch.no_grad(): seq = pol.generate(**enc_, max_new_tokens=12, do_sample=True, temperature=0.7, pad_token_id=tok.eos_token_id)
        T0 = enc_.input_ids.shape[1]; comp = seq[:, T0:]; txt = tok.batch_decode(comp, skip_special_tokens=True)
        r = torch.tensor([verifier(t, a, b) for t, (a, b) in zip(txt, np.repeat(prob, G, 0))]).view(n_prompts, G)
        adv = ((r - r.mean(1, keepdim=True)) / (r.std(1, keepdim=True) + 1e-4)).view(-1)      # group-relative advantage (Definition 22.5)
        cmask = torch.cumprod((comp != tok.eos_token_id).long(), 1); cmask = torch.cat([torch.ones_like(cmask[:, :1]), cmask[:, :-1]], 1).float()   # up to the first end token
        att = torch.cat([enc_.attention_mask, torch.ones_like(comp)], 1)
        def tok_logp(model):
            lp = torch.log_softmax(model(seq, attention_mask=att).logits[:, T0 - 1:-1], -1); return lp.gather(-1, comp[..., None])[..., 0]
        with torch.no_grad(): lref = tok_logp(ref)
        lpol = tok_logp(pol); d = lref - lpol; kl = torch.exp(d) - d - 1                   # the k3 estimate of KL(pi || pi_ref), per token
        per_tok = -(adv[:, None] * torch.exp(lpol - lpol.detach())) + beta * kl            # on-policy: the ratio is 1, its gradient is the score
        loss = ((per_tok * cmask).sum(1) / cmask.sum(1)).mean(); loss.backward(); opt.step(); opt.zero_grad()
        log.append((r.mean().item(), ((kl * cmask).sum(1) / cmask.sum(1)).mean().item()))
    return pol.eval(), log
t0 = time.time(); a0, h0, w0, ex0 = evaluate(ref)
print(f"before GRPO (greedy, 128 held-out problems): whole answer right {a0:.1%}, last digit right {h0:.1%}, {w0:.1f} words per reply")
print(f"  e.g. {ex0}")
for name, ver in [('honest verifier (whole answer)', honest), ('hackable verifier (last digit)', hackable)]:
    pol_v, log = grpo(ver); a1, h1, w1, ex1 = evaluate(pol_v)
    print(f"GRPO, {name}: training reward {np.mean([x for x, _ in log[:5]]):.2f} -> {np.mean([x for x, _ in log[-5:]]):.2f} over 30 steps; "
          f"held out: whole answer {a1:.1%}, last digit {h1:.1%}, {w1:.1f} words per reply; KL {log[-1][1]:.3f}")
    print(f"  e.g. {ex1}")
print(f"({time.time() - t0:.0f} s)")
before GRPO (greedy, 128 held-out problems): whole answer right 58.6%, last digit right 86.7%, 5.0 words per reply
  e.g. ['96 + 55 = 151', '78 + 60 = 138', '26 + 56 is 102.']
GRPO, honest verifier (whole answer): training reward 0.39 -> 0.51 over 30 steps; held out: whole answer 71.1%, last digit 93.8%, 5.0 words per reply; KL 0.008
  e.g. ['96 + 55 = 151', '78 + 60 = 148.', '26 + 56 = 82.']
GRPO, hackable verifier (last digit): training reward 0.56 -> 0.75 over 30 steps; held out: whole answer 68.8%, last digit 92.2%, 5.0 words per reply; KL 0.009
  e.g. ['96 + 55 = 151', '78 + 60 = 138.', '26 + 56 = 82']
(4575 s)

Before GRPO the chat model gets 58.6% of held-out sums right with greedy decoding (it often restates the problem, "96 + 55 = 151", and the checker reads the last number). Thirty GRPO steps with the honest verifier move the training reward from 0.39 to 0.51 and held-out accuracy to 71.1%, while the policy stays close to the reference (KL 0.008 nats per token). The hackable verifier, unlike in the tiny transformer, is not hacked: its reward rises from 0.56 to 0.75, and whole-answer accuracy rises with it, to 68.8%. For a model that already adds, the easiest way to get the last digit right is to get the sum right; the tiny transformer, trained from scratch on 200 examples, found a cheaper route. Reward hacking needs an exploit that is easier for the policy than the task, and enough optimisation to find it: thirty steps on a pretrained model may not be enough, and a run of a few hundred steps is the right test.

4 — Reward hacking and Goodhart's law

A reward model is a statistical model of human judgement, fitted on a few thousand or a few hundred thousand comparisons, and it is wrong in places. Optimisation finds those places. This is Goodhart's law (Goodhart, 1975), in Marilyn Strathern's phrasing: "when a measure becomes a target, it ceases to be a good measure." Both experiments above show it.

Real systems show the same patterns at scale. RLHF-trained models write longer answers than their reward actually justifies (Singhal et al., 2023), agree with the user's stated opinion (sycophancy; Sharma et al., 2024), and, with code rewards, can learn to special-case the tests. Gao, Schulman and Hilton (2023) measured the effect directly: as a policy is optimised harder against a proxy reward model, the score of a separate "gold" reward model first rises and then falls, with the turning point predictable from the KL divergence travelled. Defences include the KL penalty, larger and more varied preference data, ensembles of reward models, rewards that check what matters, and evaluation by people on fresh prompts.

history — from Atari preferences to reasoning models

Christiano et al. (2017) showed that an agent could learn Atari games and simulated robot tasks from a few hundred (robot tasks) to a few thousand (Atari) human comparisons of short video clips. Stiennon et al. (2020) applied the same idea to summarisation, and OpenAI's InstructGPT (Ouyang et al., 2022) used it to make GPT-3 follow instructions: labellers preferred the outputs of a 1.3-billion-parameter InstructGPT model to those of the 175-billion-parameter GPT-3. ChatGPT followed in November 2022. Anthropic's Constitutional AI (Bai et al., 2022) replaced many human labels with AI feedback guided by written principles (RLAIF). DPO (2023) removed the explicit reward model, and DeepSeek-R1 (January 2025) showed that RL with simple verifiable rewards can produce strong step-by-step reasoning, a recipe quickly adopted across the field.

big picture — RL behind every chatbot

Every widely used chatbot is trained with some form of preference or reward optimisation after pretraining. The same tools now reach beyond chat: verifiable-reward RL trains models for mathematics, programming and formal proof, where answers can be checked automatically, and RL from AI feedback scales supervision where human labels are scarce. These methods also raise the questions of this lecture in sharper form: whose preferences are learned, how a model trained to please differs from one trained to be correct, and how to detect reward hacking in systems too capable to check by hand. Lecture 23 returns to how to read the claims made about such systems.

5 — Summary, exercises, projects, reading

summary
exercises
  1. Show that the Bradley–Terry likelihood is unchanged if every reward \(r(x,y)\) is replaced by \(r(x,y)+c(x)\). What does this imply for comparing reward-model scores across different prompts?

    Only differences \(r(x,y_w)-r(x,y_l)\) between responses to the same prompt enter the model, and \(c(x)\) cancels in them. So a reward model's scores are meaningful only relative to other responses to the same prompt: a score of 2 on one prompt and 1 on another says nothing about which response is better. This is also why \(Z(x)\) cancels in Theorem 22.9.

  2. In Theorem 22.8, compute \(\mathrm{KL}(\pi^*\Vert\pi_{\text{ref}})\) for two responses with \(\pi_{\text{ref}}=(\tfrac12,\tfrac12)\) and rewards \(r=(1,0)\), at \(\beta=1\).

    \(\pi^*\propto(\tfrac12e,\tfrac12)\), so \(\pi^*=(e/(e+1),1/(e+1))\approx(0.731,0.269)\). Then

    $$\mathrm{KL}=0.731\log\frac{0.731}{0.5}+0.269\log\frac{0.269}{0.5}\approx0.278-0.167=0.111\ \text{nats}.$$

    The expected reward rises from 0.5 to 0.731 at a KL cost of about 0.11 nats.

  3. Compute the gradient of the DPO loss for one pair with respect to \(\log\pi_\theta(y_w\mid x)\). When is it large, and what does that mean?

    Write \(h=\beta(\log\frac{\pi_\theta(y_w)}{\pi_{\text{ref}}(y_w)}-\log\frac{\pi_\theta(y_l)}{\pi_{\text{ref}}(y_l)})\). The loss is \(-\log\sigma(h)\), whose derivative with respect to \(h\) is \(-(1-\sigma(h))=-\sigma(-h)\). So

    $$\frac{\partial\mathcal L}{\partial\log\pi_\theta(y_w)}=-\beta\,\sigma(-h).$$

    The gradient is largest when \(h\) is very negative, that is, when the model currently prefers the rejected response more than the reference does. Pairs the model already gets right contribute little: DPO is a weighted, self-correcting classifier.

  4. In GRPO, if all \(G\) responses to a prompt get the same reward, what is the update from that prompt (ignoring the KL term)? Why might this be a problem for very hard prompts, and what could one do?

    All advantages are zero, so the prompt contributes nothing. For a prompt the model never solves, no learning signal arrives at all: verifiable-reward RL can only sharpen abilities the model already shows occasionally. Remedies include a curriculum from easy to hard prompts, larger groups (more chances of one success), partial-credit rewards, and supervised examples for problems out of reach. Our addition model, which still fails every sample on some prompts, learns slowly partly for this reason.

  5. Our reward model scores replies without the conversation. Suggest two ways to make it see the prompt, and predict their effect on the experiment of Section 2.

    (a) Add features of the prompt–reply pair, such as the TF-IDF similarity between the last human turn and the reply, or interaction features of prompt and reply words. (b) Use a model that encodes the whole conversation, such as a fine-tuned transformer (the standard approach). Either way, replies written for other conversations would score lower, and the optimised policy would put more probability on the conversation's own replies; how much depends on how well the new features capture relevance.

  6. (Code.) In the GRPO experiment, raise the KL weight β from 0.02 to 0.2 and 1, for both verifiers. How do the speed of learning and the damage done by the hackable verifier change?

    A larger β holds the policy closer to the fine-tuned model: learning from the honest verifier slows down, and so does the collapse under the hackable one. At large β the hacked policy keeps more of its real accuracy but gains less on the proxy. The KL penalty is a brake, not a cure: it limits how far the policy can follow a flawed reward, but it does not tell it which direction is flawed.

student projects

Each project uses this lecture's methods and public data or small open models, and has a clear result.

1. DPO on a class-built preference set

question
Can a small open language model be steered towards clearer explanations of maths concepts with a few hundred preference pairs written by the class, and does it learn to be longer rather than better?
data
Pairs of explanations of the same concept (for example, of the chain rule or of a p-value), generated by the model and ranked by students, with a held-out set ranked blind.
method
DPO with several values of β; length-controlled evaluation (compare responses of matched length); a second group of students as independent judges.
outcome
Win rates with confidence intervals, a measurement of length bias, and examples of reward hacking if any appear.

2. How stable is GRPO?

question
On a toy verifiable task, how do the group size \(G\), the KL weight β and the difficulty of the prompts affect the speed and stability of GRPO?
data
Multi-digit arithmetic, sorting short lists, or simple algebra problems generated with known answers.
method
This lecture's tiny transformer; grids over \(G\) and β, several seeds each; learning curves, the fraction of prompts with zero advantage, and the KL travelled.
outcome
Practical guidance on the settings, and an account of when GRPO fails to learn.

Reading and sources

Data. HH-RLHF helpful-base comparisons: Bai et al. (2022), github.com/anthropics/hh-rlhf (MIT licence). The arithmetic problems are generated in the lecture. Pretrained models from the Hugging Face Hub: all-MiniLM-L6-v2 (sentence-transformers/all-MiniLM-L6-v2, Apache 2.0) and SmolLM2-135M-Instruct (HuggingFaceTB/SmolLM2-135M-Instruct, Apache 2.0). The DPO and GRPO runs on SmolLM2 are sized for two CPU cores (a few hundred pairs, thirty GRPO steps, the top third of the network trained), so they show the mechanisms at work on a real chat model, not results of a full-scale training run.

numbers in the text not printed by a code box are computed in verify_L22b.py, figures and widget data in figs_L22b.py; code-box outputs are pasted from actual runs (numpy, scikit-learn and JAX on two CPU cores)