Machine Learning · Lecture 23

Lecture 23
Reading the Frontier

Part VII · Onward  |  the anatomy of an ML paper, a notation decoder for the whole course, reading experiments critically (baselines, seeds, leakage, benchmark saturation), three guided dissections, reproducing a figure (grokking), the 2026 map of the field  |  reading: Lipton & Steinhardt (2019); Henderson et al. (2018); Agarwal et al. (2021); Kapoor & Narayanan (2023); Power et al. (2022)

Cartoon: reading the small printA stick figure holds a paper whose headline says "state of the art". A second figure with a magnifying glass looks at a footnote and says: on one seed, with tuned baselines missing. State of the art!Best result ever. On one seed, andthe baselines weren'ttuned.
A paper's claim is only as strong as the experiment behind it: the course ends by turning methods into a reading skill
In this lecture
  1. The anatomy of an ML paper
  2. A notation decoder
  3. Reading experiments critically
  4. Three guided dissections
  5. Reproducing a figure: grokking
  6. The 2026 map, and a map of the course
  7. Exercises, projects, reading

The aim of these notes, from the first lecture, has been that a reader can pick up a current paper from NeurIPS, ICML or ICLR and follow its methods. Twenty-two lectures have supplied the methods. This one supplies the habits: where to look in a paper, how to decode its notation, what to distrust in its experiments, and how to check a result yourself.

1 — The anatomy of an ML paper

Most machine-learning papers follow the same plan, and each part answers one question.

A useful order for a first reading is abstract, figures and tables, conclusion, then the method, and only then the rest. Lipton and Steinhardt (2019) list four common weaknesses in ML writing worth checking for: failing to separate explanation from speculation; failing to identify the source of an empirical gain; "mathiness" (equations that decorate rather than clarify); and misuse of language, such as suggestive names for ordinary techniques.

2 — A notation decoder

Every symbol used in the course, with its meaning and where it first appears. Type in the box to filter the table.

Notation decoder

3 — Reading experiments critically

Four questions catch most weak experiments.

code — how often too few seeds pick the wrong winner; how leakage manufactures accuracy
# Reading experiments critically: how often three seeds pick the wrong winner, and how leakage manufactures accuracy
import numpy as np, time, warnings; warnings.filterwarnings('ignore')
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
rng = np.random.default_rng(0)
print("method B is truly better than A by a gap d (in units of the seed-to-seed standard deviation); compare their means over n seeds")
for d in (0.25, 0.5, 1.0):
    row = []
    for n in (1, 3, 5, 10):
        a = rng.standard_normal((100_000, n)).mean(1); b = (d + rng.standard_normal((100_000, n))).mean(1); row.append(np.mean(a > b))
    print(f"  d = {d:<4}: the worse method wins in {', '.join(f'{x:.0%}' for x in row)} of comparisons with 1, 3, 5, 10 seeds")
# leakage: choose features using all the data, then cross-validate, on data with no signal at all
X = rng.standard_normal((100, 5000)); y = rng.integers(0, 2, 100)                 # 100 samples, 5,000 noise features, random labels
def top_k(Xa, ya, k=20): return np.argsort(-np.abs(np.corrcoef(Xa.T, ya)[-1, :-1]))[:k]
leaky = cross_val_score(LogisticRegression(max_iter=2000), X[:, top_k(X, y)], y, cv=5).mean()
honest = []
for f in range(5):
    te = np.arange(f * 20, (f + 1) * 20); tr = np.setdiff1d(np.arange(100), te); k = top_k(X[tr], y[tr])
    honest.append(LogisticRegression(max_iter=2000).fit(X[tr][:, k], y[tr]).score(X[te][:, k], y[te]))
print(f"\npure noise (100 samples, 5,000 features, random labels): 5-fold accuracy {leaky:.0%} if the 20 'best' features are chosen on all the data first;"
      f" {np.mean(honest):.0%} if they are chosen inside each fold (chance: 50%)")
method B is truly better than A by a gap d (in units of the seed-to-seed standard deviation); compare their means over n seeds
  d = 0.25: the worse method wins in 43%, 38%, 34%, 29% of comparisons with 1, 3, 5, 10 seeds
  d = 0.5 : the worse method wins in 36%, 27%, 22%, 13% of comparisons with 1, 3, 5, 10 seeds
  d = 1.0 : the worse method wins in 24%, 11%, 6%, 1% of comparisons with 1, 3, 5, 10 seeds

pure noise (100 samples, 5,000 features, random labels): 5-fold accuracy 90% if the 20 'best' features are chosen on all the data first; 54% if they are chosen inside each fold (chance: 50%)

Seeds first. When the better method's advantage is half a seed-to-seed standard deviation, a comparison on three seeds each picks the wrong winner 27% of the time, and even ten seeds get it wrong 13% of the time; for an advantage of a full standard deviation, three seeds still mislead 11% of the time. Leakage next. On data with no signal at all, choosing the 20 features most correlated with the labels on the whole data set and then cross-validating reports 90% accuracy; doing the same selection inside each training fold gives 54%, close to chance, as it should. Nothing in the leaky pipeline looks wrong at a glance, which is why this mistake is common in published work.

4 — Three guided dissections

Each dissection applies Section 1 to a well-known paper: the claim, the evidence that carries it, the check that matters most, and what the paper does not show.

dissection 1 (deep learning) — He, Zhang, Ren and Sun, "Deep residual learning for image recognition" (CVPR 2016)

Claim. Very deep networks become trainable if each block learns a residual \(F(x)\) added to its input, \(y=x+F(x)\), rather than a whole new mapping.

Key evidence. On ImageNet, a plain 34-layer network has higher training error than a plain 18-layer one, while the residual 34-layer network beats the residual 18-layer one. Because the comparison is on training error, the problem is optimisation, not overfitting: this "degradation" observation is the paper's central argument (Lecture 11).

The check that matters. Plain and residual networks have the same depth, width and number of parameters, so the shortcut connection is the only difference. The ablation over shortcut types (identity, or projection only where shapes change, or projection everywhere) shows that the identity shortcut alone gives most of the gain.

What it does not show. Why residual networks optimise more easily is argued informally; later work (for example on the loss landscape, and on residual networks as discretised differential equations) supplied explanations. The results are on image recognition (classification, plus detection on PASCAL VOC and COCO); the idea's spread to transformers was established by later papers, not this one.

dissection 2 (causal) — Reisach, Seiler and Weichwald, "Beware of the simulated DAG! Causal discovery benchmarks may be easy to game" (NeurIPS 2021)

Claim. On common simulated benchmarks, continuous structure-learning methods such as NOTEARS succeed largely because variance increases along the causal order, a regularity they call varsortability, rather than because they reason about causal structure.

Key evidence. A trivial baseline that sorts variables by variance and regresses each on its predecessors matches the sophisticated methods, and rescaling the data (which leaves the causal structure unchanged) degrades the continuous methods sharply but not scale-invariant ones such as PC. Lecture 20b reproduced both findings.

The check that matters. The paper measures varsortability for the benchmarks themselves, so the explanation is tested, not just proposed; and it offers the baseline as a tool others can run.

What it does not show. It does not show that real data are never varsortable (in some physical systems variance really does accumulate downstream), nor that continuous methods are useless; it shows that benchmark success needs a stronger control.

dissection 3 (quantum ML) — Tang, "A quantum-inspired classical algorithm for recommendation systems" (STOC 2019)

Claim. The quantum recommendation algorithm of Kerenidis and Prakash (2017), believed to give an exponential speed-up, can be matched by a classical algorithm whose running time is polynomial in the logarithm of the matrix size, if the classical algorithm gets comparable access to the data.

Key evidence. The quantum algorithm assumes the data are stored in a structure (QRAM, or a binary tree of partial sums) that lets it prepare states proportional to rows of the matrix. Tang shows that the same structure gives a classical computer "sample and query" access, enough to sample from a low-rank approximation by length-squared sampling, the randomised linear algebra of Lecture 16a.

The check that matters. Compare quantum and classical algorithms under the same input model. Much of the claimed exponential advantage came from the input assumptions, not the quantum processing.

What it does not show. The classical algorithm's polynomial dependence on rank and precision is much worse than the quantum one's, so the gap became polynomial rather than disappearing. Speed-ups for learning from quantum data, where the input is itself a quantum state, are not affected (Lecture 20a's classical shadows, and Huang et al., 2022).

5 — Reproducing a figure: grokking

Power, Burda, Edwards, Babuschkin and Misra (2022) trained small transformers on tables such as \(a+b\bmod p\) and found that, long after the training accuracy reached 100%, the validation accuracy suddenly jumped from chance to near-perfect: the network "groks" the rule. Their first figure plots both accuracies against training steps on a log scale. Reproducing it is a good exercise: it is cheap, the claim is sharp, and the details (data fraction, weight decay) matter. Following Gromov (2023), we use a two-layer network with a quadratic activation instead of a transformer, which groks within a few thousand steps on a CPU.

code — grokking modular addition: training and test accuracy over time
# Reproducing a figure: "grokking" on modular addition (Power et al., 2022), with a small MLP in the style of Gromov (2023)
import jax, jax.numpy as jnp
p_ = 97; pairs_ = np.array([(a, b) for a in range(p_) for b in range(p_)]); y_all = (pairs_[:, 0] + pairs_[:, 1]) % p_
def onehot_pairs(pr): return np.concatenate([np.eye(p_)[pr[:, 0]], np.eye(p_)[pr[:, 1]]], 1).astype(np.float32)
def grok(frac, wd, steps=3000, seed=0, width=256, lr=1e-3):
    g = np.random.default_rng(seed); idx = g.permutation(len(pairs_)); ntr = int(frac * len(pairs_)); tr, te = idx[:ntr], idx[ntr:]
    Xtr, ytr, Xte, yte = jnp.array(onehot_pairs(pairs_[tr])), jnp.array(y_all[tr]), jnp.array(onehot_pairs(pairs_[te])), jnp.array(y_all[te])
    k1, k2 = jax.random.split(jax.random.PRNGKey(seed)); P = {'W1': jax.random.normal(k1, (2 * p_, width)) / np.sqrt(2 * p_), 'W2': jax.random.normal(k2, (width, p_)) / np.sqrt(width)}
    f = lambda P, X: (X @ P['W1']) ** 2 @ P['W2']                      # quadratic activation, as in Gromov (2023)
    loss = lambda P, X, y: -jnp.mean(jax.nn.log_softmax(f(P, X))[jnp.arange(len(y)), y])
    acc = jax.jit(lambda P, X, y: jnp.mean(f(P, X).argmax(1) == y))
    @jax.jit
    def step(P, m, v, t):
        gr = jax.grad(loss)(P, Xtr, ytr)
        m = jax.tree.map(lambda m, g: 0.9 * m + 0.1 * g, m, gr); v = jax.tree.map(lambda v, g: 0.98 * v + 0.02 * g * g, v, gr)
        P = jax.tree.map(lambda p, m, v: p - lr * ((m / (1 - 0.9 ** t)) / (jnp.sqrt(v / (1 - 0.98 ** t)) + 1e-8) + wd * p), P, m, v)   # AdamW
        return P, m, v
    m = jax.tree.map(jnp.zeros_like, P); v = jax.tree.map(jnp.zeros_like, P); log = []
    for t in range(1, steps + 1):
        P, m, v = step(P, m, v, t)
        if t in (1, 2, 5, 10, 20, 30, 50, 70) or t % 25 == 0: log.append((t, float(acc(P, Xtr, ytr)), float(acc(P, Xte, yte))))
    return np.array(sorted(set(log)))
t0 = time.time(); gk = {}
for frac, wd in [(0.5, 1.0), (0.5, 0.0), (0.3, 1.0)]:
    gk[(frac, wd)] = L = grok(frac, wd)
    t_tr = next((int(t) for t, a, b in L if a >= 0.99), None); t_te = next((int(t) for t, a, b in L if b >= 0.99), None)
    print(f"train on {frac:.0%} of the {len(pairs_):,} sums mod {p_}, weight decay {wd}: train accuracy reaches 99% at step {t_tr}; "
          + (f"test accuracy at step {t_te}" if t_te else "test accuracy never within 3,000 steps") + f"; final test accuracy {L[-1, 2]:.1%}")
print(f"time {time.time() - t0:.0f} s")
train on 50% of the 9,409 sums mod 97, weight decay 1.0: train accuracy reaches 99% at step 325; test accuracy at step 825; final test accuracy 100.0%
train on 50% of the 9,409 sums mod 97, weight decay 0.0: train accuracy reaches 99% at step 300; test accuracy never within 3,000 steps; final test accuracy 79.9%
train on 30% of the 9,409 sums mod 97, weight decay 1.0: train accuracy reaches 99% at step 275; test accuracy never within 3,000 steps; final test accuracy 98.9%
time 588 s

The figure reproduces the phenomenon. With half of the table for training and weight decay 1, training accuracy reaches 99% at step 325, but test accuracy only at step 825: for hundreds of steps the network has memorised the training sums without generalising. Without weight decay it memorises just as fast but never generalises fully (test accuracy 79.9% after 3,000 steps). With less data (30% of the table) generalisation is slower again: test accuracy climbs towards the training accuracy but has not passed 99% by step 3,000 (it ends at 98.9%). Weight decay and the amount of data control grokking, as Power et al. reported; the exact step counts differ from theirs, because the architecture and optimiser do.

1101001,0003,0000%25%50%75%100%optimisation step (log scale)accuracy50% of the table for training, weight decay 150% of the table for training, weight decay 030% of the table for training, weight decay 1training accuracytest accuracy
Training (dashed) and test (solid) accuracy for addition modulo 97, log-scaled steps, in the style of Figure 1 of Power et al. (2022), which shows division modulo 97. Each colour is one configuration of training fraction and weight decay.

Nanda et al. (2023) reverse-engineered grokked networks and found that they compute modular addition with discrete Fourier features (Lecture 18): the weights encode \(\cos(2\pi ka/p)\) and \(\sin(2\pi ka/p)\) for a few frequencies \(k\), and trigonometric identities combine them. They proposed "progress measures" that grow smoothly during the apparently sudden transition: the jump in test accuracy is the end of a gradual process, driven by weight decay slowly removing a memorising solution.

6 — The 2026 map, and a map of the course

Six active areas connect directly to this course.

A map of the course and its landmark papers

Choose a lecture or a theme to see the papers behind it: what each showed, and where it was published. The timeline places them in time; click a dot to jump to its paper.

7 — Exercises, projects, reading

exercises
  1. Pick a recent paper from NeurIPS, ICML or ICLR in an area of this course. Write its claim in the one-sentence form of Section 1, list its baselines and the number of runs behind its main table, and name the single experiment that would most change your mind about the claim.

    A good answer is specific: "Method M improves accuracy on benchmark B by 2 points over baseline Y because of component Z; the main table reports the mean of three seeds without error bars; the most informative missing experiment is an ablation of Z with the baselines tuned with the same budget." If you cannot write the claim in one sentence, the paper may not make one.

  2. Two methods differ by half a seed-to-seed standard deviation. Using the code of Section 3, how many seeds does each need before the worse method wins less than 5% of comparisons?

    The difference of two means of \(n\) seeds has standard deviation \(\sqrt{2/n}\), so the worse method wins with probability \(\Phi(-d\sqrt{n/2})\). For \(d=0.5\) we need \(0.5\sqrt{n/2}\ge1.645\), i.e. \(n\ge2(1.645/0.5)^2\approx21.6\), so 22 seeds each. Most papers use far fewer.

  3. Explain why choosing features on the whole data set before cross-validation inflates accuracy, even when there is no signal. How should the pipeline be organised?

    Among thousands of noise features, some correlate with the labels by chance in this particular sample, including in the samples later used for testing. Selecting them on all the data lets the test folds influence the model. Every step that looks at labels (feature selection, tuning, early stopping) must happen inside each training fold, for example with a scikit-learn Pipeline passed to cross_val_score.

  4. In the grokking experiment, why might weight decay be necessary for generalisation? What happens to the weight norm of the memorising and the generalising solutions?

    Both solutions fit the training set. Memorising the table needs large, unstructured weights; the Fourier solution needs only a few frequencies and has a smaller norm. Weight decay continually shrinks the weights, so it slowly favours the smaller-norm, generalising solution once the training loss is near zero; without it, nothing pushes the network away from memorisation. Tracking the weight norm during training shows it falling as test accuracy rises.

  5. A paper reports a quantum algorithm with an exponential speed-up for a machine-learning task. List three questions to ask before accepting the claim.

    (1) What is the input model? If the data must be loaded into quantum memory, how long does that take, and would the same access help a classical algorithm (dequantisation)? (2) What does the output look like? A quantum state encoding the answer is not the answer; reading it out may cost as much as the speed-up saved. (3) Against which classical algorithm, under which assumptions on rank, condition number and precision, is the comparison made? Many speed-ups shrink when these are fixed.

student projects

Each project produces a public, checkable result.

1. Reproduce a paper for the ML Reproducibility Challenge

question
Does the main result of a recent paper hold up when reimplemented from its description, and which details does it depend on?
data
A paper chosen from the challenge's list, with its data sets and, if available, its code.
method
Reimplement the key experiment; run several seeds; add the ablations and baselines the paper omits; document every deviation.
outcome
A report in the challenge's format; accepted reports are published in Transactions on Machine Learning Research and in ReScience C.

2. A benchmark contamination audit

question
Do items from a popular benchmark appear verbatim or nearly verbatim in public web text that language models are trained on?
data
A public benchmark (for example a question-answering or maths set) and a public web corpus such as a Common Crawl sample.
method
n-gram overlap and near-duplicate search (MinHash); manual checks of matches; comparison of model accuracy on contaminated and clean items, if model access allows.
outcome
An estimate of the contaminated share, with examples and a recommendation for how the benchmark should be used.

Reading and sources

numbers in the text not printed by a code box are computed in verify_L23.py, figures and widget data in figs_L23.py; code-box outputs are pasted from actual runs (numpy, scikit-learn and JAX on two CPU cores)