Part V · Signals, senses, structure | images as tensors, hand-designed against learned features, CNNs through the Fourier lens, augmentation as symmetry, transfer learning, CLIP and zero-shot classification, diffusion models, spectrograms, mel scale and MFCCs, audio classification, speech recognition at scale | reading: He et al. (2016); Dosovitskiy et al. (2021); Radford et al. (2021); Ho, Jain & Abbeel (2020); Radford et al. (2023)
Images and sounds are signals (Lecture 18) with a lot of structure: nearby pixels and nearby moments are related, and what an object or a sound is does not change when it moves, gets louder or is seen in a mirror. This lecture shows how models exploit that structure. Convolutional networks learn the kind of filters Lecture 18 designed by hand; augmentation and pooling build in symmetry; large pretrained models turn images and text into shared embeddings that work with few or no labels; diffusion models generate images by learning to remove noise; and audio models classify sound from its spectrogram.
By the end of this lecture you should be able to compare hand-designed and learned image features; explain convolution and pooling as built-in symmetry and prove that group-averaged features are invariant; use a pretrained encoder with a linear probe and explain when transfer learning helps; explain CLIP's zero-shot classification and its sensitivity to prompts; state the diffusion training objective and its link to denoising score matching; compute spectrograms, mel spectrograms and MFCCs; train an audio classifier; and describe how wav2vec 2.0 and Whisper reach large-scale speech recognition.
A colour image is a tensor: an array of numbers with three indices, height × width × colour channel. A CIFAR-10 image is 32 × 32 × 3, so 3,072 numbers in [0, 1]. Treating those numbers as a flat vector throws away the fact that neighbouring pixels belong together.
Before deep learning, vision relied on hand-designed features. An edge detector such as the Sobel filter is a small convolution kernel (Lecture 18) that responds where brightness changes sharply. The histogram of oriented gradients (HOG; Dalal and Triggs, 2005) divides the image into small cells and records, in each, how much edge there is in each of nine directions: a summary of shape that ignores exact colour and small shifts.
60,000 colour images of 32 × 32 pixels in ten classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck), collected by Alex Krizhevsky, Vinod Nair and Geoffrey Hinton (Krizhevsky, 2009) from the "80 million tiny images" collection. We use the first training batch (10,000 images) and the first 2,000 test images, from the Python version of the official files in the GitHub repository 0Colin/cifar-10-batches-py, because the Toronto server was not reachable.
# Images as tensors; hand-designed features (edges, HOG) against raw pixels on CIFAR-10 import numpy as np, pickle, time from skimage.feature import hog from scipy.ndimage import convolve from sklearn.linear_model import LogisticRegression def cifar(name): d = pickle.load(open(f'data/{name}', 'rb'), encoding='bytes') return d[b'data'].reshape(-1, 3, 32, 32).transpose(0, 2, 3, 1).astype(np.float32) / 255, np.array(d[b'labels']) Xtr, ytr = cifar('data_batch_1'); Xte, yte = cifar('test_batch'); Xte, yte = Xte[:2000], yte[:2000] names = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'] print(f"training images {Xtr.shape} (height, width, colour channels), test images {Xte.shape[0]}; one image = {Xtr[0].size} numbers") grey = lambda X: X @ np.array([0.299, 0.587, 0.114], np.float32) sobel_x = np.array([[1, 0, -1], [2, 0, -2], [1, 0, -1]], np.float32) # responds to vertical edges g = grey(Xte[:1]).squeeze(); gx = convolve(g, sobel_x); gy = convolve(g, sobel_x.T) print(f"Sobel edge magnitude on test image 0 ({names[yte[0]]}): mean {np.hypot(gx, gy).mean():.3f}, " f"strongest 5% of pixels above {np.quantile(np.hypot(gx, gy), 0.95):.3f}") def hogf(X): return np.array([hog(x, orientations=9, pixels_per_cell=(8, 8), cells_per_block=(2, 2)) for x in grey(X)]) t0 = time.time(); Htr, Hte = hogf(Xtr), hogf(Xte) print(f"HOG: {Htr.shape[1]} features per image ({time.time() - t0:.0f} s for {len(Xtr) + len(Xte):,} images)") for nm, A, B in [('raw pixels (3,072)', Xtr.reshape(len(Xtr), -1), Xte.reshape(len(Xte), -1)), (f'HOG ({Htr.shape[1]})', Htr, Hte)]: clf = LogisticRegression(max_iter=2000, C=0.1).fit(A, ytr) print(f"logistic regression on {nm:20s} trained on 10,000 images: test accuracy {clf.score(B, yte):.3f}")
training images (10000, 32, 32, 3) (height, width, colour channels), test images 2000; one image = 3072 numbers Sobel edge magnitude on test image 0 (cat): mean 0.528, strongest 5% of pixels above 1.479 HOG: 324 features per image (4 s for 12,000 images) logistic regression on raw pixels (3,072) trained on 10,000 images: test accuracy 0.355 logistic regression on HOG (324) trained on 10,000 images: test accuracy 0.483
Edges and HOG help a lot. Logistic regression on the 3,072 raw pixel values reaches 0.355; on the 324 HOG numbers it reaches 0.483, with about a tenth as many inputs. Raw pixels make a linear classifier compare colours at fixed positions, so a cat one pixel to the left looks like a different image. HOG records local edge directions and pools them over cells, which is closer to shape and less sensitive to small shifts. Chance is 0.100.
A convolutional layer (Lecture 11) slides the same small set of weights over every position of the image. Lecture 18 explains why this is natural: convolution is exactly the operation that commutes with shifts. A cat shifted three pixels to the right produces feature maps shifted three pixels to the right. This property is called equivariance; pooling over positions then turns it into invariance, an output that does not change at all.
Let a group \(G\) of transformations act on inputs, written \(x\mapsto g\cdot x\): for example horizontal flips, or cyclic shifts of an image.
A feature map \(f\) is invariant under \(G\) if
$$f(g\cdot x)=f(x)\qquad\text{for all }g\in G\text{ and all }x.$$It is equivariant if \(G\) also acts on the outputs and
$$f(g\cdot x)=g\cdot f(x)\qquad\text{for all }g\in G\text{ and all }x.$$Let \(G\) be a finite group acting on inputs, and \(f\) any feature map. The group-averaged feature
$$\bar f(x)=\frac1{|G|}\sum_{h\in G}f(h\cdot x)$$is invariant under \(G\). If \(f\) is already invariant, then \(\bar f=f\).
Fix \(g\in G\). Then
$$\bar f(g\cdot x)=\frac1{|G|}\sum_{h\in G}f\big((hg)\cdot x\big).$$As \(h\) runs over \(G\), so does \(h'=hg\), because multiplication by \(g\) on the right is a bijection of \(G\). Hence the sum equals \(\sum_{h'\in G}f(h'\cdot x)\), and \(\bar f(g\cdot x)=\bar f(x)\).
If \(f\) is invariant, every term equals \(f(x)\), so the average is \(f(x)\). \(\square\)
Global average pooling over all positions is group averaging over shifts, so a CNN that ends with it is (nearly) shift-invariant. For symmetries the architecture does not build in, such as mirror images, there are two options: average the model's predictions over the transformed inputs at test time (Theorem 19.1 directly), or augment the training data with transformed copies so that the network learns to be approximately invariant. Augmentation is cheaper at test time and is the standard choice.
# A small CNN in JAX, trained with and without augmentation; equivariance and the Fourier view of learned filters import jax, jax.numpy as jnp def init(key): k = jax.random.split(key, 4); he = lambda k, s, fan: jax.random.normal(k, s) * jnp.sqrt(2 / fan) return {'c1': he(k[0], (5, 5, 3, 16), 75), 'b1': jnp.zeros(16), 'c2': he(k[1], (3, 3, 16, 32), 144), 'b2': jnp.zeros(32), 'c3': he(k[2], (3, 3, 32, 64), 288), 'b3': jnp.zeros(64), 'W': he(k[3], (64, 10), 64) * 0.5, 'b': jnp.zeros(10)} conv = lambda x, w: jax.lax.conv_general_dilated(x, w, (1, 1), 'SAME', dimension_numbers=('NHWC', 'HWIO', 'NHWC')) pool = lambda x: jax.lax.reduce_window(x, -jnp.inf, jax.lax.max, (1, 2, 2, 1), (1, 2, 2, 1), 'VALID') def net(p, x, feats=False): h1 = jax.nn.relu(conv(x, p['c1']) + p['b1']); h = pool(h1) h = pool(jax.nn.relu(conv(h, p['c2']) + p['b2'])); h = jax.nn.relu(conv(h, p['c3']) + p['b3']) return h1 if feats else h.mean((1, 2)) @ p['W'] + p['b'] # global average pooling, then a linear layer def augment(key, x): # random flip and random shift by up to 4 pixels k1, k2 = jax.random.split(key); x = jnp.where(jax.random.bernoulli(k1), x[:, ::-1], x) i, j = jax.random.randint(k2, (2,), 0, 9); return jax.lax.dynamic_slice(jnp.pad(x, ((4, 4), (4, 4), (0, 0)), mode='reflect'), (i, j, 0), (32, 32, 3)) def loss(p, x, y): z = net(p, x); return jnp.mean(jax.nn.logsumexp(z, 1) - z[jnp.arange(len(y)), y]) def make_step(aug): @jax.jit def step(p, m, v, t, key, x, y): if aug: x = jax.vmap(augment)(jax.random.split(key, len(x)), x) L, g = jax.value_and_grad(loss)(p, x, y) m = jax.tree.map(lambda m, g: 0.9 * m + 0.1 * g, m, g); v = jax.tree.map(lambda v, g: 0.999 * v + 0.001 * g * g, v, g) p = jax.tree.map(lambda p, m, v: p - 2e-3 * (m / (1 - 0.9**t)) / (jnp.sqrt(v / (1 - 0.999**t)) + 1e-8), p, m, v) return p, m, v, L return step predict = jax.jit(lambda p, x: net(p, x).argmax(1)) def acc(p, X, Y): return float(np.mean(np.concatenate([np.asarray(predict(p, jnp.asarray(X[i:i + 500]))) for i in range(0, len(X), 500)]) == Y)) def train(X, Y, aug, steps, seed=0, B=128): p = init(jax.random.PRNGKey(seed)); m = jax.tree.map(jnp.zeros_like, p); v = jax.tree.map(jnp.zeros_like, p) st = make_step(aug); key = jax.random.PRNGKey(seed + 100); Xj, Yj = jnp.asarray(X), jnp.asarray(Y) for t in range(1, steps + 1): key, k1, k2 = jax.random.split(key, 3); b = jax.random.randint(k1, (B,), 0, len(X)); p, m, v, L = st(p, m, v, t, k2, Xj[b], Yj[b]) return p t0 = time.time(); cnn = {} for aug in [False, True]: cnn[aug] = train(Xtr, ytr, aug, 2000) print(f"CNN ({sum(a.size for a in jax.tree.leaves(cnn[aug])):,} parameters), {'with' if aug else 'without'} flip+shift augmentation: " f"test accuracy {acc(cnn[aug], Xte, yte):.3f}; on mirrored test images {acc(cnn[aug], Xte[:, :, ::-1], yte):.3f}; " f"shifted 4 px right {acc(cnn[aug], np.roll(Xte, 4, axis=2), yte):.3f}") print(f"training time for both: {time.time() - t0:.0f} s") x0 = jnp.asarray(Xte[:4]); s = 3 # equivariance: shift then convolve = convolve then shift f_shift = net(cnn[True], jnp.roll(x0, s, axis=2), feats=True); shift_f = jnp.roll(net(cnn[True], x0, feats=True), s, axis=2) inner = (slice(None), slice(None), slice(6, 26)) print(f"first-layer feature maps: max |conv(shift(x)) - shift(conv(x))| away from the borders {float(jnp.abs(f_shift - shift_f)[inner].max()):.1e}") F1 = np.asarray(cnn[True]['c1']) # (5, 5, 3, 16) learned filters spec = (np.abs(np.fft.fft2(F1, s=(16, 16), axes=(0, 1))) ** 2).sum(2) # power spectrum of each filter, summed over colours fr = np.hypot(*np.meshgrid(np.fft.fftfreq(16), np.fft.fftfreq(16), indexing='ij')) low = (spec * (fr < 0.2)[..., None]).sum((0, 1)) / spec.sum((0, 1)) print(f"learned 5x5 filters: share of power below 0.2 cycles/pixel ranges from {low.min():.2f} to {low.max():.2f}; " f"{np.sum(low > 0.5)} of 16 are mainly low-pass (smooth or colour blobs), {np.sum(low < 0.25)} mainly high-pass (edges, texture)") prob = jax.jit(lambda p, x: jax.nn.softmax(net(p, x))) # Theorem 19.1 at test time: average over {identity, mirror} def acc_tta(p, X, Y): P = np.concatenate([np.asarray(prob(p, jnp.asarray(X[i:i + 500])) + prob(p, jnp.asarray(X[i:i + 500, :, ::-1]))) for i in range(0, len(X), 500)]) return float(np.mean(P.argmax(1) == Y)) for aug in [False, True]: print(f"averaging predictions over the image and its mirror ({'with' if aug else 'without'} augmentation): accuracy {acc_tta(cnn[aug], Xte, yte):.3f}, " f"identical on mirrored images: {acc_tta(cnn[aug], Xte[:, :, ::-1], yte) == acc_tta(cnn[aug], Xte, yte)}")
CNN (25,002 parameters), without flip+shift augmentation: test accuracy 0.535; on mirrored test images 0.533; shifted 4 px right 0.515 CNN (25,002 parameters), with flip+shift augmentation: test accuracy 0.515; on mirrored test images 0.524; shifted 4 px right 0.495 training time for both: 518 s first-layer feature maps: max |conv(shift(x)) - shift(conv(x))| away from the borders 0.0e+00 learned 5x5 filters: share of power below 0.2 cycles/pixel ranges from 0.08 to 0.36; 0 of 16 are mainly low-pass (smooth or colour blobs), 9 mainly high-pass (edges, texture) averaging predictions over the image and its mirror (without augmentation): accuracy 0.542, identical on mirrored images: True averaging predictions over the image and its mirror (with augmentation): accuracy 0.523, identical on mirrored images: True
Three results. First, the convolution is exactly shift-equivariant: away from the borders, shifting the image and then filtering gives the same feature maps as filtering and then shifting, to the last digit. Second, learning beats design: the small CNN reaches 0.535 against HOG's 0.483. Third, and less expected, augmentation did not help here: the network trained with random flips and shifts scores 0.515 against 0.535 without them, and is no better on shifted images (0.495 against 0.515). With 25,002 parameters, 10,000 images and 2,000 steps the network is probably short of training rather than short of data, and augmentation makes the training task harder; it usually pays off for larger networks trained for longer. Test-time averaging over the image and its mirror (Theorem 19.1) does what the theorem promises: predictions on an image and on its mirror are now identical. As a bonus, accuracy rises slightly, to 0.542 for the plain network. After this short training the learned filters are still noisy, but they lean towards high frequencies, as edge and texture detectors do: their share of low-frequency power ranges from 0.08 to 0.36, and 9 of the 16 put less than a quarter of their power there.
Symmetry can also be built in completely, without learning. The scattering transform (Mallat, 2012) is a fixed CNN whose filters are wavelets (Lecture 18) and whose nonlinearity is the modulus; it comes with a guarantee that learned networks lack.
Let \(\Phi\) be the windowed scattering transform at scale \(2^J\), and deform an image by \(x_\tau(u)=x(u-\tau(u))\) with a smooth displacement field \(\tau\). Then, for a constant \(C\) depending only on the wavelets and the number of layers,
$$\lVert\Phi(x_\tau)-\Phi(x)\rVert\;\le\;C\,\lVert x\rVert\Big(2^{-J}\lVert\tau\rVert_\infty+\lVert\nabla\tau\rVert_\infty+\lVert\nabla^2\tau\rVert_\infty\Big).$$So translations much smaller than the window \(2^J\) and gentle warps barely change the features, while the representation still separates different signals. (Statement.)
A plain Fourier modulus is invariant to translation too, but it is unstable to deformation: a slight stretch moves high frequencies a long way. Wavelets localise each frequency band in space, which is why the bound holds, and why learned first layers tend to look like wavelets.
A network learns whatever pattern predicts the label in the training data, causal or not. Zech et al. (2018, PLOS Medicine) trained CNNs to detect pneumonia on chest radiographs pooled from several hospital systems. The networks could also identify, almost perfectly, which hospital an image came from, from cues such as scanner type and image markings; because pneumonia rates differed between hospitals, the hospital was a useful shortcut, and performance dropped on hospitals not seen in training. In causal terms, the hospital is a confounder of image and diagnosis (Lecture 13). Remedies include training on data from many sites, testing on held-out sites, and augmentations that destroy known shortcuts.
Training a large network from scratch needs millions of labelled images. Transfer learning reuses a network already trained on a large data set: its early and middle layers compute general features (edges, textures, object parts) that are useful for many tasks. The cheapest form is a linear probe: freeze the pretrained network, compute its output features for your images, and fit only a linear classifier on top. Fine-tuning goes further and also adjusts the pretrained weights, usually with a small learning rate.
We use the image encoder of CLIP ViT-B/32 (Section 4), with weights trained by the OpenCLIP project on the LAION-400M image–text collection (Ilharco et al., 2021; Schuhmann et al., 2021), released on GitHub (mlfoundations/open_clip, MIT licence). The weights were converted to NumPy and the network re-implemented in JAX (clip_jax.py); a check against the published zero-shot accuracy appears in Section 4. CIFAR-10 plays the role of the small target data set; Section 4 ends with the more traditional choice, an ImageNet-trained ResNet-18, on a harder target.
# Transfer learning: features from a pretrained network (CLIP's image encoder) plus a linear classifier, with few labels import sys; sys.path.insert(0, '.'); import clip_jax as C C.load() def clip_feats(X): out = [] for i in range(0, len(X), 50): im = jax.image.resize(jnp.asarray(X[i:i + 50]), (len(X[i:i + 50]), 224, 224, 3), 'bicubic') # 32x32 -> 224x224 out.append(np.asarray(C.encode_image(jnp.clip(im, 0, 1)))) F = np.concatenate(out); return F / np.linalg.norm(F, axis=1, keepdims=True) sub = np.concatenate([np.flatnonzero(ytr == c)[:100] for c in range(10)]) # 100 labelled images per class t0 = time.time(); Ftr, Fte = clip_feats(Xtr[sub]), clip_feats(Xte[:1000]); ysub, y1k = ytr[sub], yte[:1000] print(f"CLIP ViT-B/32 image features: {Ftr.shape[1]} numbers per image; {len(sub) + 1000:,} images encoded in {time.time() - t0:.0f} s") for per in [1, 5, 10, 100]: idx = np.concatenate([np.flatnonzero(ysub == c)[:per] for c in range(10)]) pr = LogisticRegression(max_iter=3000, C=10.0).fit(Ftr[idx], ysub[idx]) small = train(Xtr[sub][idx], ysub[idx], True, 600) if per in (10, 100) else None print(f"{per:3d} labels per class ({10 * per:4d} images): CLIP features + logistic regression {pr.score(Fte, y1k):.3f}" + (f"; small CNN from scratch (with augmentation) {acc(small, Xte[:1000], y1k):.3f}" if small is not None else '')) print(f"for comparison, the small CNN trained on all 10,000 labelled images: {acc(cnn[True], Xte[:1000], y1k):.3f}")
CLIP ViT-B/32 image features: 512 numbers per image; 2,000 images encoded in 168 s 1 labels per class ( 10 images): CLIP features + logistic regression 0.710 5 labels per class ( 50 images): CLIP features + logistic regression 0.885 10 labels per class ( 100 images): CLIP features + logistic regression 0.906; small CNN from scratch (with augmentation) 0.298 100 labels per class (1000 images): CLIP features + logistic regression 0.943; small CNN from scratch (with augmentation) 0.388 for comparison, the small CNN trained on all 10,000 labelled images: 0.507
Pretrained features change the picture completely. With a single labelled image per class, a linear probe on CLIP's 512 features reaches 0.710; with 10 per class it reaches 0.906, and with 100 per class 0.943. The small CNN trained from scratch on the same 100 and 1,000 images reaches only 0.298 and 0.388, and even with all 10,000 labelled images it gets 0.507. The pretrained encoder has already learned, from hundreds of millions of web images, which differences between images matter; the probe only has to draw lines between groups that are already separated. The comparison is not quite fair to the CNN, which sees 32 × 32 images while CLIP sees them enlarged to 224 × 224, but no amount of resizing would let 25,002 parameters learn from 100 images what CLIP learned from 400 million.
A vision transformer (ViT; Dosovitskiy et al., 2021) cuts an image into patches (here 32 × 32 pixels, 49 patches for a 224 × 224 image), turns each patch into a vector, and processes the sequence of patch vectors with the transformer of Lecture 12. It has no built-in convolution or shift symmetry; it learns them from data, which is why it needs much more data than a CNN.
CLIP (Radford et al., 2021) trains an image encoder and a text encoder together on hundreds of millions of image–caption pairs from the web. The objective is the contrastive InfoNCE loss of Lecture 14: in each batch, the embedding of an image should be closer to the embedding of its own caption than to every other caption.
Let \(f\) map images and \(g\) map texts into the same space, with unit-length outputs. For classes with names \(c_1,\dots,c_K\), choose a prompt template, such as "a photo of a {}.", and form one text embedding per class:
$$t_k=g(\text{prompt}(c_k)),\qquad k=1,\dots,K.$$Classify an image \(x\) as the class whose text embedding is most similar:
$$\hat y(x)=\arg\max_k\;f(x)^\top t_k.$$No labelled image of any class is used: the class names alone define the classifier.
# Zero-shot classification with CLIP: no labelled CIFAR images at all, only class names in sentences tok = C.Tokenizer() def text_emb(prompts): T = np.asarray(C.encode_text(jnp.asarray(tok(prompts)))); return T / np.linalg.norm(T, axis=1, keepdims=True) templates = ['{}', 'a photo of a {}.', 'a blurry photo of a {}.', 'a low resolution photo of a {}.', 'a drawing of a {}.', 'an x-ray of a {}.'] Tall = {} for tpl in templates: Tall[tpl] = text_emb([tpl.format(n) for n in names]); pred = (Fte @ Tall[tpl].T).argmax(1) print(f"prompt {tpl!r:38s} zero-shot accuracy {np.mean(pred == y1k):.3f}") ens = np.mean([Tall[t] for t in templates[1:4]], 0); ens /= np.linalg.norm(ens, axis=1, keepdims=True) pz = (Fte @ ens.T).argmax(1); print(f"average of the three 'photo' prompts: {np.mean(pz == y1k):.3f}") cm = np.zeros((10, 10), int); np.add.at(cm, (y1k, pz), 1); off = cm - np.diag(np.diag(cm)) i, j = np.unravel_index(off.argmax(), off.shape) print(f"most common confusion: {names[i]} labelled as {names[j]} ({off[i, j]} of {cm[i].sum()}); " f"per-class accuracy from {np.min(np.diag(cm) / cm.sum(1)):.2f} ({names[np.argmin(np.diag(cm) / cm.sum(1))]}) " f"to {np.max(np.diag(cm) / cm.sum(1)):.2f} ({names[np.argmax(np.diag(cm) / cm.sum(1))]})")
prompt '{}' zero-shot accuracy 0.848
prompt 'a photo of a {}.' zero-shot accuracy 0.919
prompt 'a blurry photo of a {}.' zero-shot accuracy 0.910
prompt 'a low resolution photo of a {}.' zero-shot accuracy 0.915
prompt 'a drawing of a {}.' zero-shot accuracy 0.910
prompt 'an x-ray of a {}.' zero-shot accuracy 0.921
average of the three 'photo' prompts: 0.914
most common confusion: cat labelled as dog (11 of 103); per-class accuracy from 0.83 (cat) to 0.98 (truck)
Without a single labelled CIFAR image, CLIP reaches accuracy 0.919 with the prompt "a photo of a {}.", better than the linear probe with 10 labelled images per class. The bare class name scores only 0.848: the text encoder was trained on captions, and a caption-like sentence puts the class name in the context it knows. Beyond that the prompt matters little: "a blurry photo", "a low resolution photo" and "a drawing" all give 0.910 to 0.915, averaging the three photo prompts gives 0.914, and even the absurd "an x-ray of a {}." gives 0.921. What carries the information is the class name in a sentence, not the description of the image. The commonest error is cat labelled as dog (11 of 103); per-class accuracy runs from 0.83 (cat) to 0.98 (truck).
CIFAR-10's ten classes are coarse, and CLIP was trained on hundreds of millions of captioned photographs, so the comparison so far flatters it. A fine-grained task is harder: telling 37 breeds of cat and dog apart, where a Ragdoll and a Birman differ in the colour of their paws. We compare CLIP's features with those of the network a computer-vision course would traditionally use for transfer, a ResNet-18 (Lecture 11) trained on the 1.28 million labelled images of ImageNet (Deng et al., 2009), in the torchvision release (He et al., 2016).
7,349 photographs of 37 breeds of cat and dog, about 200 per breed, with a standard split into 3,680 training and 3,669 test images (Parkhi, Vedaldi, Zisserman and Jawahar, 2012; CC BY-SA 4.0), in the Parquet copy timm/oxford-iiit-pet on the Hugging Face Hub. Images are resized so that the shorter side is 224 pixels and centre-cropped; we use every other test image.
# Transfer to a fine-grained task: 37 cat and dog breeds (Oxford-IIIT Pets), ImageNet ResNet-18 features against CLIP features import io, torch, torchvision, pyarrow.parquet as pq from PIL import Image pets = {s: pq.read_table(f'data/pets/{s}.parquet').to_pandas() for s in ('train', 'test')} breed = {r.label: ' '.join(w for w in r.image_id.split('_')[:-1]) for r in pets['train'].itertuples()} names37 = [breed[k] for k in range(37)] te_rows = pets['test'].iloc[::2] # every other test image: 1,835 images def images(df, side=224): # resize the short side to 224, centre crop out = [] for b in df.image: im = Image.open(io.BytesIO(b['bytes'])).convert('RGB'); w, h = im.size; s = side / min(w, h) im = im.resize((max(side, round(w * s)), max(side, round(h * s))), Image.BICUBIC); w, h = im.size out.append(np.asarray(im.crop(((w - side) // 2, (h - side) // 2, (w - side) // 2 + side, (h - side) // 2 + side)), np.float32) / 255) return np.stack(out) rn = torchvision.models.resnet18(weights='IMAGENET1K_V1').eval(); rn.fc = torch.nn.Identity() # 512 features from the last pooling layer mean, sd = np.array([0.485, 0.456, 0.406]), np.array([0.229, 0.224, 0.225]) def resnet_feats(X): with torch.no_grad(): F = np.concatenate([rn(torch.tensor(((X[i:i + 64] - mean) / sd).transpose(0, 3, 1, 2), dtype=torch.float32)).numpy() for i in range(0, len(X), 64)]) return F / np.linalg.norm(F, axis=1, keepdims=True) def clip_feats224(X): F = np.concatenate([np.asarray(C.encode_image(jnp.asarray(X[i:i + 50]))) for i in range(0, len(X), 50)]) return F / np.linalg.norm(F, axis=1, keepdims=True) t0 = time.time(); feats = {} for s, df in [('train', pets['train']), ('test', te_rows)]: parts = [(resnet_feats(X), clip_feats224(X)) for X in (images(df.iloc[i:i + 200]) for i in range(0, len(df), 200))] # 200 images at a time feats[s] = (np.concatenate([a for a, _ in parts]), np.concatenate([b for _, b in parts]), df.label.values) print(f"Oxford-IIIT Pets: {len(pets['train']):,} training images (about 100 per breed), {len(te_rows):,} test images; " f"features from both networks in {time.time() - t0:.0f} s") (Rtr, Ctr, yp), (Rte, Cte, yq) = feats['train'], feats['test'] print("labels per breed ImageNet ResNet-18 + logistic CLIP ViT-B/32 + logistic") for per in [1, 5, 10, 100]: idx = np.concatenate([np.flatnonzero(yp == c)[:per] for c in range(37)]) a = [LogisticRegression(max_iter=3000, C=10.0).fit(F[idx], yp[idx]).score(G, yq) for F, G in [(Rtr, Rte), (Ctr, Cte)]] print(f"{per:6d} ({len(idx):5,} images) {a[0]:24.3f} {a[1]:27.3f}") Tp = text_emb([f'a photo of a {n}, a type of pet.' for n in names37]) print(f"CLIP zero-shot, no labelled pets at all ('a photo of a [breed], a type of pet.'): {np.mean((Cte @ Tp.T).argmax(1) == yq):.3f}")
Oxford-IIIT Pets: 3,680 training images (about 100 per breed), 1,835 test images; features from both networks in 511 s
labels per breed ImageNet ResNet-18 + logistic CLIP ViT-B/32 + logistic
1 ( 37 images) 0.647 0.449
5 ( 185 images) 0.831 0.741
10 ( 370 images) 0.858 0.802
100 (3,680 images) 0.901 0.880
CLIP zero-shot, no labelled pets at all ('a photo of a [breed], a type of pet.'): 0.864
Here the traditional encoder wins. With one labelled photograph per breed, the ImageNet ResNet-18 probe is right 64.7% of the time and the CLIP probe only 44.9% (chance: 2.7%); with about 100 per breed, 90.1% against 88.0%. ImageNet's 1,000 classes include about 120 breeds of dog and several of cat, so ResNet-18 was trained, in effect, on a version of this very task, and its features separate breeds. CLIP's training was broader: it learned to match photographs with captions, not to tell a Birman from a Ragdoll. Yet CLIP has an advantage no probe has: it can read. Matching each photograph with the sentence “a photo of a [breed], a type of pet.” gives 86.4% with no labelled pets at all, as good as the ResNet-18 probe with ten labels per breed. Transfer works best when the pretraining task resembles the target, and language is a second route to the target when labels are scarce.
A zero-shot CLIP classifier for ten classes is built from ten prompts. How many labelled images are needed to add an eleventh class?
Lecture 16b's variational autoencoder generates images by decoding random codes. Two other families dominate image generation. A generative adversarial network (GAN; Goodfellow et al., 2014) trains a generator against a discriminator that tries to tell real images from generated ones; GANs produce sharp images but are hard to train. A diffusion model (Sohl-Dickstein et al., 2015; Ho, Jain and Abbeel, 2020) learns to reverse a gradual noising process: corrupt data a little at a time until only noise remains, then train a network to undo one small step. Generating means starting from pure noise and denoising, step by step.
Choose noise levels \(\beta_1,\dots,\beta_T\in(0,1)\) and set
$$\alpha_t=1-\beta_t,\qquad\bar\alpha_t=\prod_{s\le t}\alpha_s.$$The forward process adds Gaussian noise. Its marginal at step \(t\) can be sampled in one jump:
$$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon,\qquad\varepsilon\sim N(0,I).$$A network \(\varepsilon_\theta(x_t,t)\) is trained to predict the added noise, by minimising
$$\mathcal L(\theta)=\mathbb E_{x_0,\,t,\,\varepsilon}\big\lVert\varepsilon_\theta(x_t,t)-\varepsilon\big\rVert^2.$$To generate, draw \(x_T\sim N(0,I)\) and, for \(t=T,\dots,1\), set
$$x_{t-1}=\frac1{\sqrt{\alpha_t}}\Big(x_t-\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\varepsilon_\theta(x_t,t)\Big)+\sqrt{\beta_t}\,z,\qquad z\sim N(0,I),$$with \(z=0\) at the last step.
Why does predicting noise produce new data? The answer is that a noise predictor is a score estimator. The score of a density \(p\) is \(\nabla_x\log p(x)\): an arrow pointing uphill, towards where data are more likely. Following the score from noise leads to the data.
Let \(\tilde x=x+\sigma\varepsilon\) with \(x\sim p\) and \(\varepsilon\sim N(0,I)\), and let \(q_\sigma\) be the density of \(\tilde x\). For any function \(s\),
$$\mathbb E\Big\lVert s(\tilde x)+\frac{\varepsilon}{\sigma}\Big\rVert^2=\mathbb E\big\lVert s(\tilde x)-\nabla\log q_\sigma(\tilde x)\big\rVert^2+\text{const},$$where the constant does not depend on \(s\). So the best denoiser recovers the score:
$$s^\star(\tilde x)=\nabla\log q_\sigma(\tilde x),\qquad\text{i.e.}\qquad\varepsilon_\theta(x_t,t)\approx-\sqrt{1-\bar\alpha_t}\;\nabla\log q_t(x_t).$$Given \(x\), the noisy point has density \(q(\tilde x\mid x)=N(\tilde x;x,\sigma^2I)\), whose score is
$$\nabla_{\tilde x}\log q(\tilde x\mid x)=-\frac{\tilde x-x}{\sigma^2}=-\frac{\varepsilon}{\sigma}.$$Expand both sides. The terms \(\mathbb E\lVert s\rVert^2\) agree, and the terms without \(s\) are constants. It remains to match the cross terms, i.e. to show
$$\mathbb E\big[s(\tilde x)^\top\nabla\log q(\tilde x\mid x)\big]=\mathbb E\big[s(\tilde x)^\top\nabla\log q_\sigma(\tilde x)\big].$$Write \(q_\sigma(\tilde x)=\int p(x)\,q(\tilde x\mid x)\,dx\). Then \(q_\sigma\nabla\log q_\sigma=\nabla q_\sigma=\int p(x)\,q(\tilde x\mid x)\,\nabla\log q(\tilde x\mid x)\,dx\). Multiply by \(s(\tilde x)\) and integrate over \(\tilde x\): both sides equal \(\int s^\top\nabla q_\sigma\,d\tilde x\).
The minimiser of the right-hand side is \(s=\nabla\log q_\sigma\). For DDPM, \(\sigma^2=1-\bar\alpha_t\) after rescaling, and predicting \(\varepsilon\) is the same as predicting \(-\sigma\) times the score. \(\square\)
We train a diffusion model on a two-dimensional "data set" (points on a spiral), small enough to run in a minute and to watch.
# A denoising diffusion model on two-dimensional data: learn to remove noise, then generate by denoising pure noise from sklearn.datasets import make_swiss_roll pts = make_swiss_roll(20000, noise=0.25, random_state=0)[0][:, [0, 2]] / 7.0 # a 2-D spiral Tn = 100; betas = np.linspace(1e-4, 0.05, Tn); alpha = 1 - betas; abar = np.cumprod(alpha) ab = jnp.asarray(abar, jnp.float32) def temb(t): # sinusoidal embedding of the time step f = jnp.exp(jnp.linspace(0, 4, 8)); a = t[:, None] / Tn * f[None] * 10; return jnp.concatenate([jnp.sin(a), jnp.cos(a)], 1) def dinit(key): k = jax.random.split(key, 4); d = lambda k, a, b: jax.random.normal(k, (a, b)) * jnp.sqrt(2 / a) return {'W1': d(k[0], 18, 128), 'b1': jnp.zeros(128), 'W2': d(k[1], 128, 128), 'b2': jnp.zeros(128), 'W3': d(k[2], 128, 128), 'b3': jnp.zeros(128), 'W4': d(k[3], 128, 2) * 0.1, 'b4': jnp.zeros(2)} def eps_net(p, x, t): # predicts the noise that was added h = jnp.concatenate([x, temb(t)], 1) for i in (1, 2, 3): h = jax.nn.silu(h @ p[f'W{i}'] + p[f'b{i}']) return h @ p['W4'] + p['b4'] def dloss(p, key, x0): k1, k2 = jax.random.split(key); t = jax.random.randint(k1, (len(x0),), 0, Tn); e = jax.random.normal(k2, x0.shape) xt = jnp.sqrt(ab[t])[:, None] * x0 + jnp.sqrt(1 - ab[t])[:, None] * e # jump straight to step t (Definition 19.3) return jnp.mean((eps_net(p, xt, t.astype(jnp.float32)) - e) ** 2) @jax.jit def dstep(p, m, v, t, key, x): L, g = jax.value_and_grad(dloss)(p, key, x) m = jax.tree.map(lambda m, g: 0.9 * m + 0.1 * g, m, g); v = jax.tree.map(lambda v, g: 0.999 * v + 0.001 * g * g, v, g) return jax.tree.map(lambda p, m, v: p - 1e-3 * (m / (1 - 0.9**t)) / (jnp.sqrt(v / (1 - 0.999**t)) + 1e-8), p, m, v), m, v, L dp = dinit(jax.random.PRNGKey(0)); dm = jax.tree.map(jnp.zeros_like, dp); dv = jax.tree.map(jnp.zeros_like, dp) key = jax.random.PRNGKey(1); P = jnp.asarray(pts, jnp.float32); t0 = time.time(); Ls = [] for t in range(1, 6001): key, k1, k2 = jax.random.split(key, 3); dp, dm, dv, L = dstep(dp, dm, dv, t, k1, P[jax.random.randint(k2, (512,), 0, len(P))]); Ls.append(float(L)) print(f"trained the noise predictor (a 3-layer MLP) for 6,000 steps in {time.time() - t0:.0f} s; loss {np.mean(Ls[:100]):.3f} -> {np.mean(Ls[-500:]):.3f}") def sample(n, seed, keep_every=2): k = jax.random.PRNGKey(seed); k, k0 = jax.random.split(k); x = jax.random.normal(k0, (n, 2)); traj = [np.asarray(x)] for t in range(Tn - 1, -1, -1): # reverse process (DDPM ancestral sampling) e = eps_net(dp, x, jnp.full((n,), t, jnp.float32)) x = (x - betas[t] / np.sqrt(1 - abar[t]) * e) / np.sqrt(alpha[t]) if t > 0: k, kz = jax.random.split(k); x = x + np.sqrt(betas[t]) * jax.random.normal(kz, x.shape) if t % keep_every == 0: traj.append(np.asarray(x)) return np.asarray(x), traj from sklearn.neighbors import NearestNeighbors nnd = NearestNeighbors(n_neighbors=1).fit(pts[:5000]) gen, traj = sample(2000, 5) gauss = np.random.default_rng(0).normal(size=(2000, 2)) * pts.std(0) + pts.mean(0) dist = lambda Q: float(np.median(nnd.kneighbors(Q)[0])) print(f"median distance to the nearest training point: generated samples {dist(gen):.3f}, fresh data {dist(pts[10000:12000]):.3f}, " f"a Gaussian with the data's mean and spread {dist(gauss):.3f}") for tq in [80, 30, 5]: # the network's noise prediction is a score estimate rng = np.random.default_rng(tq); x0 = pts[rng.choice(len(pts), 400)]; xt = np.sqrt(abar[tq]) * x0 + np.sqrt(1 - abar[tq]) * rng.normal(size=x0.shape) ref = pts[rng.choice(len(pts), 4000)] * np.sqrt(abar[tq]); s2 = 1 - abar[tq] d2 = ((xt[:, None] - ref[None]) ** 2).sum(-1); w = np.exp(-(d2 - d2.min(1, keepdims=True)) / (2 * s2)); w /= w.sum(1, keepdims=True) true_score = (w @ ref - xt) / s2 # score of the noised empirical distribution est = -np.asarray(eps_net(dp, jnp.asarray(xt, jnp.float32), jnp.full((400,), tq, jnp.float32))) / np.sqrt(s2) cos = np.sum(true_score * est, 1) / (np.linalg.norm(true_score, axis=1) * np.linalg.norm(est, axis=1)) print(f"step t = {tq:2d} (noise sd {np.sqrt(s2):.2f}): median cosine between -eps_hat/sigma and the true score {np.median(cos):.3f}")
trained the noise predictor (a 3-layer MLP) for 6,000 steps in 36 s; loss 0.635 -> 0.452 median distance to the nearest training point: generated samples 0.012, fresh data 0.008, a Gaussian with the data's mean and spread 0.243 step t = 80 (noise sd 0.90): median cosine between -eps_hat/sigma and the true score 0.999 step t = 30 (noise sd 0.46): median cosine between -eps_hat/sigma and the true score 0.996 step t = 5 (noise sd 0.09): median cosine between -eps_hat/sigma and the true score 0.983
The samples land on the spiral. Their median distance to the nearest training point is 0.012, close to that of fresh points from the true spiral (0.008) and far below that of a Gaussian with the same mean and spread (0.243). Theorem 19.3 can be checked directly, because for a finite training set the score of the noised data has a closed form. The rescaled noise prediction points in almost exactly the direction of the true score: median cosine 0.999 at noise level 0.90, 0.996 at 0.46 and 0.983 at 0.09. The match is weakest at low noise, where the score changes sharply from one point to the next and is hardest to learn.
Fei-Fei Li and colleagues built ImageNet (Deng et al., 2009), with labels for millions of images collected through Amazon Mechanical Turk, and from 2010 ran an annual challenge on a 1,000-class subset. In 2012 AlexNet (Krizhevsky, Sutskever and Hinton), a deep CNN trained on two graphics cards, won it by a wide margin, which is usually taken as the start of the deep-learning era. ResNets (He et al., 2016) made networks of a hundred layers trainable (Lecture 11). For generation, OpenAI's DALL·E (2021) produced images from text prompts, and Stable Diffusion (Rombach et al., 2022), a diffusion model run in the latent space of an autoencoder, was released openly in August 2022.
A sound recording is a one-dimensional signal: air pressure sampled many thousands of times a second. Its meaning lives in how its frequency content changes over time, so audio models usually start from the spectrogram of Lecture 18 and treat it as an image, with time across and frequency up.
Human hearing does not treat frequencies evenly. We distinguish 100 Hz from 200 Hz easily, but 10,000 Hz from 10,100 Hz hardly at all: pitch perception is roughly logarithmic above about 1,000 Hz. The mel scale (Stevens, Volkmann and Newman, 1937) measures frequency in units of equal perceived pitch steps.
The mel scale maps a frequency \(f\) in hertz to
$$m(f)=2595\,\log_{10}\!\Big(1+\frac f{700}\Big).$$A mel spectrogram pools the power spectrogram into bands that are equally spaced in \(m\): narrow at low frequencies, wide at high ones. One triangular filter per band sums the power of the frequency bins it covers.
Mel-frequency cepstral coefficients (MFCCs; Davis and Mermelstein, 1980) take the logarithm of each band's power in each frame, then a discrete cosine transform across the bands, keeping the first 13 to 20 coefficients. They summarise the smooth shape of the spectrum in each frame.
ESC-50 (Piczak, 2015) contains 2,000 five-second recordings of 50 kinds of everyday sound, taken from the Freesound project, organised into 5 folds for cross-validation, and released under Creative Commons licences on GitHub (karolpiczak/ESC-50). The full archive could not be downloaded from the build environment, so we fetched individual files: the 10 classes of the standard ESC-10 subset (dog, rooster, rain, sea waves, crackling fire, crying baby, sneezing, clock tick, helicopter, chainsaw) plus door knock, which the lecture plan asked us to compare with dog barks. That gives 440 clips, 40 per class.
# Sound: waveform, spectrogram, mel spectrogram and MFCCs for environmental sounds (ESC-50) import librosa, csv meta = [r for r in csv.DictReader(open('data/esc50.csv')) if r['esc10'] == 'True' or r['category'] == 'door_wood_knock'] cats = sorted(set(r['category'] for r in meta)); t0 = time.time() wav = np.stack([librosa.load(f"data/esc/{r['filename']}", sr=22050)[0] for r in meta]) # 5 s clips, resampled to 22,050 Hz ya = np.array([cats.index(r['category']) for r in meta]); fold = np.array([int(r['fold']) for r in meta]) print(f"{len(wav)} clips, {len(cats)} classes, {wav.shape[1]:,} samples each ({wav.shape[1] / 22050:.0f} s at 22,050 Hz); loaded in {time.time() - t0:.0f} s") S = np.abs(librosa.stft(wav[0], n_fft=1024, hop_length=512)) ** 2 M = librosa.feature.melspectrogram(S=S, sr=22050, n_mels=64) mf = librosa.feature.mfcc(S=librosa.power_to_db(M), n_mfcc=20) print(f"one clip: waveform {wav[0].shape}, power spectrogram {S.shape} (frequency bins x frames), " f"mel spectrogram {M.shape}, MFCCs {mf.shape}") for f in [100, 500, 1000, 4000, 8000]: print(f" {f:5d} Hz = {2595 * np.log10(1 + f / 700):6.0f} mel") mb = librosa.mel_frequencies(66, fmin=0, fmax=11025) print(f"widths of the 64 mel bands: lowest {mb[2] - mb[0]:.0f} Hz wide, highest {mb[-1] - mb[-3]:.0f} Hz wide") logmel = np.stack([librosa.power_to_db(librosa.feature.melspectrogram(y=w, sr=22050, n_fft=1024, hop_length=512, n_mels=64)) for w in wav]) cent = librosa.feature.spectral_centroid(y=wav, sr=22050).mean(-1).ravel() for c in ['dog', 'door_wood_knock', 'rooster', 'rain', 'clock_tick', 'helicopter']: print(f" {c:16s} median spectral centroid {np.median(cent[ya == cats.index(c)]):6.0f} Hz")
440 clips, 11 classes, 110,250 samples each (5 s at 22,050 Hz); loaded in 3 s
one clip: waveform (110250,), power spectrogram (513, 216) (frequency bins x frames), mel spectrogram (64, 216), MFCCs (20, 216)
100 Hz = 150 mel
500 Hz = 607 mel
1000 Hz = 1000 mel
4000 Hz = 2146 mel
8000 Hz = 2840 mel
widths of the 64 mel bands: lowest 102 Hz wide, highest 1105 Hz wide
dog median spectral centroid 1331 Hz
door_wood_knock median spectral centroid 696 Hz
rooster median spectral centroid 1132 Hz
rain median spectral centroid 3553 Hz
clock_tick median spectral centroid 2448 Hz
helicopter median spectral centroid 1249 Hz
The mel scale compresses high frequencies: 4,000 Hz is 2,146 mel and 8,000 Hz only 2,840, so the lowest mel band is 102 Hz wide and the highest 1,105 Hz. The 513 frequency bins of each frame shrink to 64 bands and then to 20 MFCCs. The spectral centroid, the power-weighted average frequency of a frame, is a one-number summary of where the energy sits: rain, a broadband hiss, has a median centroid of 3,553 Hz and a clock tick 2,448 Hz, while a dog bark sits at 1,331 Hz and a door knock at 696 Hz.
Using \(m(f)=2595\log_{10}(1+f/700)\), what is 1,000 Hz in mel, to the nearest whole number?
With a log-mel spectrogram as an image, the tools of Section 2 apply directly. A CNN slides small filters over time and frequency; pooling over time makes the prediction insensitive to when in the clip the sound occurs; random time shifts during training are the audio version of the image shifts of Section 2. The traditional alternative summarises each clip by the mean and standard deviation of its MFCCs and fits a linear classifier.
# Audio classification: MFCC statistics + logistic regression against a small CNN on log-mel spectrograms (5-fold CV) from sklearn.preprocessing import StandardScaler mfcc = np.stack([librosa.feature.mfcc(S=l, n_mfcc=20) for l in logmel]); Fm = np.concatenate([mfcc.mean(-1), mfcc.std(-1)], 1) Z = ((logmel - logmel.mean()) / logmel.std()).astype(np.float32)[..., None] # (440, 64, 216, 1) def ainit(key): k = jax.random.split(key, 4); he = lambda k, s, fan: jax.random.normal(k, s) * jnp.sqrt(2 / fan) return {'c1': he(k[0], (3, 3, 1, 16), 9), 'b1': jnp.zeros(16), 'c2': he(k[1], (3, 3, 16, 32), 144), 'b2': jnp.zeros(32), 'c3': he(k[2], (3, 3, 32, 64), 288), 'b3': jnp.zeros(64), 'W': he(k[3], (64, 11), 64) * 0.5, 'b': jnp.zeros(11)} def anet(p, x): h = pool(jax.nn.relu(conv(x, p['c1']) + p['b1'])); h = pool(jax.nn.relu(conv(h, p['c2']) + p['b2'])) h = pool(jax.nn.relu(conv(h, p['c3']) + p['b3'])); return h.mean((1, 2)) @ p['W'] + p['b'] def aloss(p, x, y): z = anet(p, x); return jnp.mean(jax.nn.logsumexp(z, 1) - z[jnp.arange(len(y)), y]) @jax.jit def astep(p, m, v, t, key, x, y): s = jax.random.randint(key, (len(x),), 0, 216); x = jax.vmap(lambda a, k: jnp.roll(a, k, axis=1))(x, s) # random time shift L, g = jax.value_and_grad(aloss)(p, x, y) m = jax.tree.map(lambda m, g: 0.9 * m + 0.1 * g, m, g); v = jax.tree.map(lambda v, g: 0.999 * v + 0.001 * g * g, v, g) return jax.tree.map(lambda p, m, v: p - 1e-3 * (m / (1 - 0.9**t)) / (jnp.sqrt(v / (1 - 0.999**t)) + 1e-8), p, m, v), m, v, L apred = jax.jit(lambda p, x: anet(p, x).argmax(1)) pl, pc = np.zeros(len(ya), int), np.zeros(len(ya), int); t0 = time.time() for f in range(1, 6): tr, te = fold != f, fold == f sc = StandardScaler().fit(Fm[tr]); pl[te] = LogisticRegression(max_iter=3000, C=1.0).fit(sc.transform(Fm[tr]), ya[tr]).predict(sc.transform(Fm[te])) p = ainit(jax.random.PRNGKey(f)); m = jax.tree.map(jnp.zeros_like, p); v = jax.tree.map(jnp.zeros_like, p) Ztr, ytr_a = jnp.asarray(Z[tr]), jnp.asarray(ya[tr]); key = jax.random.PRNGKey(10 + f) for t in range(1, 801): key, k1, k2 = jax.random.split(key, 3); b = jax.random.randint(k1, (32,), 0, int(tr.sum())); p, m, v, L = astep(p, m, v, t, k2, Ztr[b], ytr_a[b]) pc[te] = np.asarray(apred(p, jnp.asarray(Z[te]))); acnn = p print(f"5-fold cross-validation on {len(ya)} clips ({time.time() - t0:.0f} s): MFCC mean+sd + logistic regression {np.mean(pl == ya):.3f}; " f"CNN on log-mel spectrograms {np.mean(pc == ya):.3f}; chance {1 / len(cats):.3f}") for nm, pr in [('logistic', pl), ('CNN', pc)]: cm = np.zeros((11, 11), int); np.add.at(cm, (ya, pr), 1) d, k_ = cats.index('dog'), cats.index('door_wood_knock') worst = sorted(((cm[i, i] / cm[i].sum(), cats[i]) for i in range(11)))[:2] print(f" {nm:8s}: dog -> door knock {cm[d, k_]}, door knock -> dog {cm[k_, d]} (of 40 each); weakest classes " + ", ".join(f"{c} {a:.2f}" for a, c in worst))
5-fold cross-validation on 440 clips (1610 s): MFCC mean+sd + logistic regression 0.764; CNN on log-mel spectrograms 0.798; chance 0.091 logistic: dog -> door knock 1, door knock -> dog 0 (of 40 each); weakest classes clock_tick 0.45, helicopter 0.60 CNN : dog -> door knock 0, door knock -> dog 0 (of 40 each); weakest classes helicopter 0.57, rain 0.60
Both methods are far above chance (0.091). Mean and spread of the MFCCs with logistic regression reach 0.764 in 5-fold cross-validation; the CNN on log-mel spectrograms reaches 0.798, so it beats the hand-designed summary, by a modest margin, with only 352 training clips per fold. Dog barks and door knocks, both short percussive bursts, are almost never confused: one of the 80 clips of the two classes is put in the other class by the logistic model, and none by the CNN. A knock is lower (696 against 1,331 Hz median centroid) and repeats at a steady rhythm, and both models pick this up. The hardest classes are helicopter (0.57) and rain (0.60) for the CNN, and clock tick (0.45) and helicopter (0.60) for the logistic model.
Speech recognition turns audio into text. Two recent systems show the two main routes to doing it with little hand-labelled data.
Both can be put to work on a small task: recognising ten spoken command words. A linear probe on wav2vec 2.0's frame vectors, averaged over the clip, tests what self-supervised pretraining has learned; Whisper needs no training at all, because it can simply be asked to transcribe.
Speech Commands v0.02 (Warden, 2018; CC BY 4.0) contains about 105,000 one-second recordings of 35 words, spoken by 2,618 volunteers through their own computers and phones, at 16 kHz. We take the official test set from the Parquet copy pollen-robotics/speech-commands-v0.02 on the Hugging Face Hub, keep the ten core command words (yes, no, up, down, left, right, on, off, stop, go), 150 clips each, and hold out a third of the speakers, so that every test voice is new to the classifier.
# Speech: ten spoken command words (Speech Commands v0.02), with MFCCs, self-supervised wav2vec 2.0 features, and Whisper zero-shot import glob, soundfile as sf, io as _io, torch, pandas as pd from transformers import Wav2Vec2Model, Wav2Vec2FeatureExtractor, WhisperProcessor, WhisperForConditionalGeneration words10 = ['yes', 'no', 'up', 'down', 'left', 'right', 'on', 'off', 'stop', 'go'] sc = pd.concat([pq.read_table(f).to_pandas() for f in sorted(glob.glob('data/speech/test-*.parquet'))]) sc = sc[sc.label.isin(words10)].groupby('label').head(150).reset_index(drop=True) # 150 clips per word clip = lambda a: (lambda x: np.pad(x[:16000], (0, max(0, 16000 - len(x)))))(sf.read(_io.BytesIO(a['bytes']))[0]) wav = [clip(a) for a in sc.audio] # 1 second at 16 kHz, padded with silence ys = np.array([words10.index(w) for w in sc.label]); spk = sc.speaker_id.values test_spk = set(sorted(set(spk))[::3]); te_m = np.array([s in test_spk for s in spk]) # a third of the speakers are held out print(f"{len(sc):,} one-second clips of 10 words from {len(set(spk))} speakers; {te_m.sum()} test clips from {len(test_spk)} held-out speakers") def probe(F, name): res = [] for per in [5, 20, 'all']: idx = np.flatnonzero(~te_m) if per == 'all' else np.concatenate([np.flatnonzero((ys == c) & ~te_m)[:per] for c in range(10)]) sc_ = StandardScaler().fit(F[idx]); m = LogisticRegression(max_iter=3000, C=1.0).fit(sc_.transform(F[idx]), ys[idx]) res.append(m.score(sc_.transform(F[te_m]), ys[te_m])) print(f"{name:48s} {res[0]:8.3f} {res[1]:9.3f} {res[2]:12.3f}") t0 = time.time() Fm = np.stack([np.concatenate([(m := librosa.feature.mfcc(y=w.astype(np.float32), sr=16000, n_mfcc=20)).mean(1), m.std(1)]) for w in wav]) fe = Wav2Vec2FeatureExtractor.from_pretrained('facebook/wav2vec2-base'); w2v = Wav2Vec2Model.from_pretrained('facebook/wav2vec2-base').eval() H = [] with torch.no_grad(): for i in range(0, len(wav), 32): x = fe(wav[i:i + 32], sampling_rate=16000, return_tensors='pt').input_values hs = w2v(x, output_hidden_states=True).hidden_states # 13 tensors: the CNN output and 12 transformer layers H.append(torch.stack([h.mean(1) for h in hs], 1).numpy()) # average over time H = np.concatenate(H) print(f"features computed in {time.time() - t0:.0f} s; test accuracy with this many labelled clips per word:") print(f"{'':48s} {'5 per word':>8s} {'20 per word':>9s} {'all (~100)':>12s}") probe(Fm, 'MFCC means and standard deviations (40 numbers)') probe(H[:, 6], 'wav2vec 2.0 base, transformer layer 6 (768)') probe(H[:, 12], 'wav2vec 2.0 base, last layer (768)') # Whisper-tiny, trained on 680,000 hours of transcribed audio, never told about this task proc = WhisperProcessor.from_pretrained('openai/whisper-tiny'); wh = WhisperForConditionalGeneration.from_pretrained('openai/whisper-tiny').eval() cand = [proc.tokenizer(' ' + w.capitalize(), add_special_tokens=False).input_ids for w in words10] start = proc.tokenizer.convert_tokens_to_ids(['<|startoftranscript|>', '<|en|>', '<|transcribe|>', '<|notimestamps|>']) idx = np.flatnonzero(te_m); free, closed = [], [] with torch.no_grad(): for i in range(0, len(idx), 16): feats = proc([wav[j] for j in idx[i:i + 16]], sampling_rate=16000, return_tensors='pt').input_features enc_out = wh.model.encoder(feats).last_hidden_state gen = wh.generate(input_features=feats, language='en', task='transcribe', max_new_tokens=8) free += [re.sub(r'[^a-z ]', '', t.lower()).split()[:1] for t in proc.batch_decode(gen, skip_special_tokens=True)] L = max(map(len, cand)); scores = [] for c in cand: # closed set: score each of the 10 words as the transcript dec = torch.tensor([start + c] * len(feats)) lp = torch.log_softmax(wh(encoder_outputs=(enc_out,), decoder_input_ids=dec).logits, -1) scores.append(sum(lp[:, len(start) - 1 + k, t] for k, t in enumerate(c))) closed += torch.stack(scores, 1).argmax(1).tolist() free_acc = np.mean([f[:1] == [words10[ys[j]]] for f, j in zip(free, idx)]) print(f"Whisper-tiny zero-shot: free transcription starts with the right word {free_acc:.3f}; " f"choosing the most probable of the 10 words {np.mean(np.array(closed) == ys[idx]):.3f} ({time.time() - t0:.0f} s in all)")
1,500 one-second clips of 10 words from 157 speakers; 493 test clips from 53 held-out speakers
features computed in 188 s; test accuracy with this many labelled clips per word:
5 per word 20 per word all (~100)
MFCC means and standard deviations (40 numbers) 0.302 0.396 0.467
wav2vec 2.0 base, transformer layer 6 (768) 0.864 0.939 0.959
wav2vec 2.0 base, last layer (768) 0.479 0.665 0.830
Whisper-tiny zero-shot: free transcription starts with the right word 0.836; choosing the most probable of the 10 words 0.970 (614 s in all)
Self-supervised features win by a wide margin. With five labelled clips per word, MFCC statistics give 30.2% on new speakers and wav2vec 2.0's sixth layer 86.4%; with all of them (about 100 per word), 46.7% against 95.9%. The last layer is worse (83.0%): it was trained to solve the pretraining task, and the middle layers carry more general information about the sounds, as the layer-by-layer analyses of wav2vec 2.0 found (Pasad, Chou and Livescu, 2021). Whisper, never trained on this task, transcribes the clip and starts its transcript with the right word 83.6% of the time; asked instead which of the ten words is the most probable transcript, it is right 97.0% of the time, with no labelled clips at all.
Cong, Choi and Lukin (Nature Physics, 2019) proposed a quantum convolutional neural network: a circuit that applies the same small unitary "filter" to neighbouring qubits, then measures and discards some qubits ("pooling") and conditions the next layer on the results, halving the system at each layer. The input is a quantum state, such as the ground state of a chain of interacting spins, and the task is to tell which phase of matter it belongs to, including symmetry-protected topological phases that no local measurement reveals. The circuit needs only a logarithmic number of parameters in the number of qubits, because, as in a CNN, the same filter is reused across positions and scales. It is a natural fit for quantum data, which a classical CNN cannot read directly; for ordinary images there is no demonstrated advantage.
S. S. Stevens, John Volkmann and Edwin Newman built the mel scale in 1937 by asking listeners to set one tone to "half the pitch" of another. Bruce Bogert, Michael Healy and John Tukey coined "cepstrum" (an anagram of "spectrum") in 1963 for the spectrum of a log spectrum, originally to detect echoes in seismograms; Steven Davis and Paul Mermelstein (1980) showed that mel-frequency cepstral coefficients were the best features for recognising spoken words, and MFCCs then dominated speech recognition for three decades, usually fed to the hidden Markov models of Lecture 17a. Deep networks replaced the Gaussian mixtures inside those systems around 2012 (Hinton et al., 2012), and end-to-end transformers such as Whisper replaced the rest.
Learned image and sound models are now routine infrastructure. In medical imaging, CNNs and transformers screen retinal photographs, mammograms and chest radiographs, under the shortcut risks of the causal box above. BirdNET (Kahl et al., 2021) identifies thousands of bird species from their songs in recordings made by citizen scientists and autonomous recorders, and similar models monitor whales, bats and forest health. Wake-word detectors ("Hey Siri", "OK Google") are tiny audio classifiers that run continuously on phones and speakers. Automatic captioning of video and live speech, built on models like Whisper, is an accessibility service for deaf and hard-of-hearing people. And the same generative models that produce images and voices on demand have made deepfake detection, itself a classification problem on images and spectrograms, a field of its own.
By definition,
$$\big(h*(S_ax)\big)_n=\sum_kh_k\,x_{n-k-a}=(h*x)_{n-a}=\big(S_a(h*x)\big)_n.$$With zero padding instead of wrap-around, pixels near the border see zeros on one side, and a shift moves different pixels next to the border. The identity then holds only away from the borders, which is why the code compares feature maps on the inner columns.
Average over the two-element group {identity, flip}:
$$\bar f(x)=\tfrac12\big(f(x)+f(\text{flip}(x))\big).$$Since flipping twice is the identity, \(\bar f(\text{flip}(x))=\bar f(x)\). It costs two forward passes per image. For a group of size \(|G|\) it costs \(|G|\) passes, which is why augmentation during training is preferred for large groups.
Convolution: \(5\cdot5\cdot3\cdot16+16=1{,}216\).
Fully connected: \(3{,}072\times16{,}384+16{,}384\approx5.0\times10^7\), about 41,000 times more. Weight sharing across positions is what makes CNNs trainable on modest data, and it is exactly the shift symmetry of Section 2.
Substitute \(x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon\) with \(\varepsilon_\theta=\varepsilon\): the noise terms cancel and \(\hat x_0=x_0\).
But many different pairs \((x_0,\varepsilon)\) give the same \(x_t\), especially at high noise. The best the network can do is predict the conditional mean of \(\varepsilon\) given \(x_t\), which by Theorem 19.3 is proportional to the score. A one-step estimate is therefore a blurry average of possible images; many small steps, each re-injecting a little noise, produce a sharp sample.
For small \(f/700\), \(\log_{10}(1+u)\approx u/\ln10\), so
$$m(f)\approx\frac{2595}{700\ln10}\,f\approx1.61\,f.$$For large \(f\), \(\log_{10}(1+f/700)\approx\log_{10}f-\log_{10}700\), so \(m(f)\approx2595\log_{10}f+\text{const}\): equal ratios of frequency give equal steps in mel.
Descriptive prompts usually change accuracy by a few points, up or down, and the classes that gain are those whose plain names are ambiguous ("deer" or "frog" in tiny, blurred images). To initialise the probe, set its weight matrix to the zero-shot text embeddings, scaled up, and fine-tune it on the few labelled images with weight decay towards that starting point. This combines prior knowledge from language with a little labelled data, and typically beats both the zero-shot classifier and a probe trained from zero on the same labels.
Each project uses this lecture's methods and public data, and has a clear result.
Data. CIFAR-10: A. Krizhevsky (2009), copy of the official Python files from github.com/0Colin/cifar-10-batches-py. CLIP ViT-B/32 weights: OpenCLIP, trained on LAION-400M, from the mlfoundations/open_clip GitHub release; tokeniser vocabulary from the open_clip_torch package (PyPI). ESC-50: K. J. Piczak (2015), individual clips from github.com/karolpiczak/ESC-50 (Creative Commons licences per clip, listed in the repository). Oxford-IIIT Pets: Parkhi et al. (2012), CC BY-SA 4.0, Parquet copy timm/oxford-iiit-pet (Hugging Face Hub). Speech Commands v0.02: Warden (2018), CC BY 4.0, Parquet copy pollen-robotics/speech-commands-v0.02. Pretrained models: ResNet-18 ImageNet weights from torchvision (BSD licence); facebook/wav2vec2-base and openai/whisper-tiny (Apache 2.0) from the Hugging Face Hub. The LibriVox audio named in the lecture plan is not used.
numbers in the text not printed by a code box are computed in verify_L19.py, figures and widget data in figs_L19.py; code-box outputs are pasted from actual runs (numpy, scikit-learn, scikit-image, librosa and JAX on two CPU cores)