Part III · Deep learning | sentiment as classification, naive Bayes and logistic regression on text, lexicons and domain shift, pre-trained representations, bias in word embeddings and the WEAT, auditing models with counterfactual inputs, fairness criteria, the impossibility theorems, auditing COMPAS and the Lecture 8 income model, counterfactual fairness | reading: Jurafsky & Martin, Speech and Language Processing (3rd ed. draft), ch. 4–6; Barocas, Hardt & Narayanan, Fairness and Machine Learning (2023), ch. 3–5; Chouldechova (2017); Caliskan, Bryson & Narayanan (2017)
The last three lectures built models that read text and images. This lecture closes the deep-learning part of the course by asking two questions about such models. First, a practical one: how do we get a machine to tell whether a piece of text is positive or negative, and how well do the different ways of doing it carry over from one kind of text to another? Second, a harder one: models learn whatever regularities their training data contain, including stereotypes and historical inequities, and are then used to make decisions about people. How do we measure what they have learned wrongly, what would it mean for a decision rule to be fair, and why can't a rule be fair in every sense at once? The second half of the lecture audits two real systems: the COMPAS risk score at the centre of the 2016 "Machine Bias" debate, and the income model we built in Lecture 8.
By the end of this lecture you should be able to build and compare naive Bayes and logistic regression sentiment classifiers and prove that naive Bayes is linear in log space; explain why lexicons and models trained on one domain fail on another; measure bias in word embeddings with analogies, projections and the WEAT, including its permutation test; define demographic parity, equalised odds and calibration; prove Chouldechova's identity and state the Kleinberg–Mullainathan–Raghavan impossibility theorem; audit a classifier's error rates by group and adjust it by post-processing; and explain counterfactual fairness and why fairness is ultimately a causal question.
"A lot of fun, if you can forgive the plot." "Not bad at all." "The worst film I have seen this year." People infer the writer's attitude from such sentences without effort; a machine needs a representation. The simplest is the bag of words: forget the order, and represent a document by the counts \(x_j\) of each word \(j\) of a vocabulary of size \(V\). It throws away a great deal ("not bad" and "bad, not" look the same), and yet it is a strong baseline. A refinement, TF-IDF, weights each count by the word's inverse document frequency \(\log(N/n_j)\), where \(n_j\) of the \(N\) documents contain word \(j\), so that words found in every document ("the", "film") count for little. Adding pairs of adjacent words ("bigrams") restores some order: "not bad" becomes a feature of its own.
50,000 movie reviews from the Internet Movie Database, half positive (rated 7–10 out of 10) and half negative (1–4), at most 30 reviews per film, collected by Maas et al. (2011) at Stanford. The original split has 25,000 reviews for training and 25,000 for testing; we use the CSV copy distributed on GitHub, which merges them, and split it 25,000/25,000 at random. The median review is 171 words long.
Two classifiers are natural for counts. The first is naive Bayes, a generative model: it models how each class produces words, assuming (naively) that the words of a document are drawn independently given its class, with class \(y\) using word \(j\) with probability \(\theta_{yj}\). The second is logistic regression (Lecture 4), which models \(P(y\mid x)\) directly. They turn out to be the same kind of classifier.
Under multinomial naive Bayes with class priors \(\pi_0,\pi_1\) and word probabilities \(\theta_{0j},\theta_{1j}\), the posterior log-odds of a document with word counts \(x\) is linear in \(x\):
$$\log\frac{P(y=1\mid x)}{P(y=0\mid x)}=\sum_{j=1}^V x_j\log\frac{\theta_{1j}}{\theta_{0j}}+\log\frac{\pi_1}{\pi_0}=w^\top x+b.$$So naive Bayes and logistic regression choose from the same family of decision rules (hyperplanes in count space); they differ only in how they pick \(w\) and \(b\): naive Bayes from per-class word frequencies, logistic regression by maximising the conditional likelihood.
Under the model, a document of length \(n=\sum_jx_j\) has
$$P(x\mid y)=\frac{n!}{\prod_jx_j!}\prod_j\theta_{yj}^{x_j}.$$By Bayes' rule, \(P(y\mid x)\propto\pi_yP(x\mid y)\), and in the ratio of the two classes the multinomial coefficient and the normalising constant \(P(x)\) cancel:
$$\frac{P(y=1\mid x)}{P(y=0\mid x)}=\frac{\pi_1\prod_j\theta_{1j}^{x_j}}{\pi_0\prod_j\theta_{0j}^{x_j}}.$$Taking logarithms gives the formula, with \(w_j=\log(\theta_{1j}/\theta_{0j})\) and \(b=\log(\pi_1/\pi_0)\).
The same argument with binary features and Bernoulli naive Bayes gives a linear function of the presence indicators. \(\square\)
The maximum-likelihood estimate \(\hat\theta_{yj}=c_{yj}/\sum_kc_{yk}\) (where \(c_{yj}\) counts word \(j\) in class \(y\)) is zero for any word never seen in class \(y\), and a single such word would then make the posterior of class \(y\) exactly zero. The standard fix, Laplace (add-\(\alpha\)) smoothing, \(\hat\theta_{yj}=(c_{yj}+\alpha)/(\sum_kc_{yk}+\alpha V)\), is the posterior mean of \(\theta_y\) under a symmetric Dirichlet(\(\alpha\)) prior (the beta–binomial calculation, in \(V\) dimensions). And the term \(\log(\pi_1/\pi_0)\) is the prior in the most literal sense: the same words give a different verdict when positive reviews are rare. Base rates will return in Section 7 as the source of the impossibility theorems.
# Sentiment as classification: naive Bayes and logistic regression on 50,000 IMDb reviews import numpy as np, pandas as pd, re, time from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer from sklearn.naive_bayes import MultinomialNB from sklearn.linear_model import LogisticRegression imdb = pd.read_csv('data/IMDB-Dataset.csv') imdb['text'] = imdb.review.str.replace('<br />', ' ', regex=False) y_all = (imdb.sentiment == 'positive').astype(int).values perm = np.random.default_rng(0).permutation(len(imdb)); tr, te = perm[:25000], perm[25000:] # 25,000 / 25,000 print(f'{len(imdb):,} reviews, {y_all.mean():.0%} positive; median length {imdb.text.str.split().str.len().median():.0f} words') cv = CountVectorizer(min_df=5, token_pattern=r"(?u)\b\w[\w']*\b") Xc_tr, Xc_te = cv.fit_transform(imdb.text.values[tr]), cv.transform(imdb.text.values[te]); ytr, yte = y_all[tr], y_all[te] nb = MultinomialNB(alpha=1.0).fit(Xc_tr, ytr) print(f'vocabulary {len(cv.vocabulary_):,} words') print(f'naive Bayes, word counts: test accuracy {nb.score(Xc_te, yte):.4f}') # Theorem 13.1 in action: naive Bayes' log-odds is a linear function w.x + b of the counts w_nb = nb.feature_log_prob_[1] - nb.feature_log_prob_[0]; b_nb = nb.class_log_prior_[1] - nb.class_log_prior_[0] lp = nb.predict_log_proba(Xc_te[:5]); print(' log-odds from predict_log_proba:', np.round(lp[:, 1] - lp[:, 0], 3)) print(' log-odds as w.x + b: ', np.round(Xc_te[:5] @ w_nb + b_nb, 3)) tf = TfidfVectorizer(min_df=5, ngram_range=(1, 2), sublinear_tf=True, token_pattern=r"(?u)\b\w[\w']*\b") Xt_tr, Xt_te = tf.fit_transform(imdb.text.values[tr]), tf.transform(imdb.text.values[te]) lr = LogisticRegression(C=10, max_iter=3000).fit(Xt_tr, ytr) print(f'logistic regression, TF-IDF words + pairs: test accuracy {lr.score(Xt_te, yte):.4f} ({Xt_tr.shape[1]:,} features)') voc = np.array(cv.get_feature_names_out()); o = np.argsort(w_nb); ok = np.asarray(Xc_tr.sum(0)).ravel() >= 50 print('most negative words (NB, seen 50+ times):', ', '.join(voc[[i for i in o if ok[i]][:10]])) print('most positive words (NB, seen 50+ times):', ', '.join(voc[[i for i in o[::-1] if ok[i]][:10]])) fv = np.array(tf.get_feature_names_out()); ol = np.argsort(lr.coef_[0]) print('most negative features (logistic):', ', '.join(fv[ol[:10]])) print('most positive features (logistic):', ', '.join(fv[ol[::-1][:10]]))
50,000 reviews, 50% positive; median length 171 words vocabulary 27,945 words naive Bayes, word counts: test accuracy 0.8385 log-odds from predict_log_proba: [ -4.325 -15.616 0.655 20.923 -10.759] log-odds as w.x + b: [ -4.325 -15.616 0.655 20.923 -10.759] logistic regression, TF-IDF words + pairs: test accuracy 0.9152 (154,851 features) most negative words (NB, seen 50+ times): uwe, boll, stinker, tripe, gamera, unfunny, waste, seagal, mst3k, flimsy most positive words (NB, seen 50+ times): ponyo, devito, custer, macarthur, novak, paulie, georges, kolchak, felix, carla most negative features (logistic): worst, bad, awful, boring, the worst, terrible, waste, worse, nothing, poor most positive features (logistic): great, excellent, perfect, wonderful, amazing, 7, the best, hilarious, brilliant, 7 10
Logistic regression on TF-IDF words and word pairs is right on 91.5% of the 25,000 test reviews; naive Bayes on raw counts, on 83.9%. The code checks Theorem 13.1 numerically: the log-odds from the library and from \(w^\top x+b\) agree to every printed digit. The two lists of top words show why naive Bayes loses. Its most decisive words are proper names: "Uwe" and "Boll" (a director of notoriously bad films), "Seagal", "MST3K" on the negative side, and "Ponyo", "DeVito", "Kolchak" on the positive. A name that appears in a few dozen reviews, nearly all of one class, gets a large weight \(\log(\theta_{1j}/\theta_{0j})\), because naive Bayes estimates each word's weight separately, as if the words were independent. Logistic regression fits all the weights jointly, so a word gets a large weight only if it adds information beyond the others, and its top features are the general words of praise and blame: "worst", "awful", "waste", "great", "excellent", "perfect", and the pairs "the worst" and "7 10" (reviewers rating a film 7/10). This is the classic contrast between generative and discriminative learning: naive Bayes approaches its (higher) asymptotic error with fewer examples, logistic regression reaches a lower error with enough data (Ng and Jordan, 2002).
A naive Bayes classifier has \(\log(\pi_1/\pi_0)=0\). A short review contains "great" once (\(w=+1.2\)), "boring" once (\(w=-1.5\)) and words whose weights sum to 0.6. What is its posterior log-odds of being positive?
Bo Pang, Lillian Lee and Shivakumar Vaithyanathan's "Thumbs up? Sentiment classification using machine learning techniques" (2002) took 1,400 movie reviews from the Internet Movie Database archive of the rec.arts.movies.reviews newsgroup, labelled them by the reviewers' star ratings, and showed that standard text classifiers (naive Bayes, maximum entropy, SVMs) beat lists of sentiment words chosen by people, while doing much worse on sentiment than on topic classification. The paper started the field of sentiment analysis in machine learning. Lexicons had a longer history: the General Inquirer, built at Harvard by Philip Stone and colleagues in the 1960s, tagged thousands of English words with categories such as "Positiv" and "Negativ", and is still in use. Maas et al.'s IMDb dataset (2011) and the Stanford Sentiment Treebank (Socher et al., 2013), with sentiment labels for every phrase of 11,855 sentences, became the benchmarks of the deep-learning era.
A sentiment lexicon is a list of words marked positive or negative, perhaps with strengths. Scoring a text by counting its positive and negative words needs no training data and is transparent, which is why lexicons remain popular in the social sciences and finance. Their weakness is that the meaning of a word depends on the domain. Loughran and McDonald (2011) found that almost three-quarters of the words that the Harvard General Inquirer classes as negative are not negative in companies' annual reports: "tax", "cost", "capital", "board", "liability", "vice" (as in vice-president). They built a lexicon for financial text, now standard in finance research. The same problem afflicts learned models: a classifier trained on movie reviews has learned the vocabulary of movie reviews. When the distribution of inputs at test time differs from that in training, we have domain shift.
4,846 sentences from English-language financial news about companies listed on the Helsinki stock exchange, each labelled positive, negative or neutral "from the viewpoint of an investor" by 16 annotators with backgrounds in finance and business (Malo et al., 2014; CC BY-NC-SA 3.0). We use the 2,264 sentences on which all annotators agreed: 1,391 neutral, 570 positive and 303 negative. Lexicons: the Harvard General Inquirer (Harvard IV-4), VADER (Hutto and Gilbert, 2014; built for social media) and Loughran–McDonald's master dictionary, as distributed in the pysentiment2 and vaderSentiment Python packages.
# Lexicons against learned models, and domain shift: the Financial PhraseBank from sklearn.model_selection import cross_val_predict import collections fpb = pd.read_csv('data/FPB_Sentences_AllAgree.txt', sep='@', header=None, names=['text', 'label'], encoding='latin-1') lab = fpb.label.map({'negative': -1, 'neutral': 0, 'positive': 1}).values print(f'{len(fpb):,} sentences on which all annotators agreed:', dict(collections.Counter(fpb.label))) words = lambda s: re.findall(r"[a-z][a-z'-]*", s.lower()) hiv = pd.read_csv('data/HIV-4.csv', low_memory=False); hiv['w'] = hiv.Entry.str.split('#').str[0].str.lower() HP, HN = set(hiv.w[hiv.Positiv.notna()]), set(hiv.w[hiv.Negativ.notna()]) # Harvard IV (general) lm = pd.read_csv('data/LM.csv'); LP, LN = set(lm.Word[lm.Positive > 0].str.lower()), set(lm.Word[lm.Negative > 0].str.lower()) vd = pd.read_csv('data/vader_lexicon.txt', sep='\t', header=None, usecols=[0, 1], names=['w', 'v'], quoting=3) VAL = dict(zip(vd.w, vd.v)) # VADER (social media) def lex_predict(pos, neg): return np.array([np.sign(sum(w in pos for w in words(s)) - sum(w in neg for w in words(s))) for s in fpb.text]) preds = {'Harvard IV (general, 1960s)': lex_predict(HP, HN), 'VADER (social media, 2014)': np.array([np.sign(sum(VAL.get(w, 0) for w in words(s))) for s in fpb.text]), 'Loughran-McDonald (finance)': lex_predict(LP, LN)} pn = lab != 0 p_imdb = np.where(lr.predict_proba(tf.transform(fpb.text))[:, 1] > 0.5, 1, -1) # the IMDb model, no retraining tf_f = TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True, min_df=2) preds['logistic trained on FPB (5-fold CV)'] = cross_val_predict(LogisticRegression(C=10, max_iter=3000, class_weight='balanced'), tf_f.fit_transform(fpb.text), lab, cv=5) print(f'{pn.sum()} sentences are positive or negative; for those, the share given the right sign, called neutral, or the wrong sign') print(f'{"method":38s} {"3-class":>8s} {"right":>7s} {"neutral":>8s} {"wrong":>6s}') print(f'{"always neutral":38s} {np.mean(lab == 0):8.1%}') for name, p in list(preds.items()) + [('logistic trained on IMDb reviews', None)]: acc = ' -' if p is None else f'{np.mean(p == lab):8.1%}' q = p_imdb[pn] if p is None else p[pn]; l = lab[pn] print(f'{name:38s} {acc} {np.mean(q == l):7.0%} {np.mean(q == 0):8.0%} {np.mean((q != l) & (q != 0)):6.0%}') cnt = collections.Counter(w for s, l in zip(fpb.text, lab) if l >= 0 for w in set(words(s)) if w in HN and w not in LN) print('Harvard-negative words (not negative for Loughran-McDonald) most common in neutral or positive sentences:') print(' ' + ', '.join(f'{w} ({c})' for w, c in cnt.most_common(10)))
2,264 sentences on which all annotators agreed: {'neutral': 1391, 'positive': 570, 'negative': 303}
873 sentences are positive or negative; for those, the share given the right sign, called neutral, or the wrong sign
method 3-class right neutral wrong
always neutral 61.4%
Harvard IV (general, 1960s) 45.8% 44% 36% 20%
VADER (social media, 2014) 56.4% 56% 21% 23%
Loughran-McDonald (finance) 65.0% 25% 66% 10%
logistic trained on FPB (5-fold CV) 84.0% 64% 21% 15%
logistic trained on IMDb reviews - 53% 0% 47%
Harvard-negative words (not negative for Loughran-McDonald) most common in neutral or positive sentences:
order (52), capital (52), board (43), service (36), deal (33), division (23), press (13), cost (12), vice (10), make (9)
On financial news, every general-purpose method struggles. Always answering "neutral" is right 61.4% of the time; the Harvard lexicon gets only 45.8% of the three-way labels right, because it finds sentiment where there is none: in neutral or good news it marks "order" (as in order intake), "capital", "board", "service", "deal" and "division" as negative. VADER, built for tweets, does a little better. The Loughran–McDonald lexicon makes the fewest wrong-sign errors (10% of the positive and negative sentences), but its list of positive words is short (354 words against 2,355 negative), so it calls two-thirds of the clearly positive or negative sentences neutral: "operating profit rose to EUR 23.5 mn" contains no lexicon word at all. Financial sentiment is carried by verbs of change attached to financial quantities, which a word list handles badly. A logistic regression trained on the financial sentences themselves (in five-fold cross-validation, so always predicting sentences it has not seen) reaches 84.0%. And the IMDb model, 91.5% accurate on reviews, gives the right sign to only 53% of the financial sentences: its weights belong to another domain.
The IMDb model gives the right sign to only 53% of the positive and negative financial sentences, barely better than a coin. Which explanation fits best?
Labelled data is expensive; unlabelled text is plentiful. The transfer-learning recipe of Lecture 12 is to learn a representation from unlabelled text and then fit a small model on top with few labels. The oldest version uses static word vectors: represent a review by the average of the GloVe vectors (Pennington, Socher and Manning, 2014; trained on six billion words of Wikipedia and newswire) of its words, and fit a logistic regression to the 100 averaged dimensions. A slightly richer version learns to weight the words: an attention-pooling classifier (Lecture 12, Section 3) computes a score for each word from its vector, turns the scores into weights with a softmax, and classifies the weighted average. The modern version fine-tunes a whole pre-trained transformer such as BERT, which we cannot download here; we report published numbers for it below.
# Transfer: pre-trained GloVe word vectors, from 100 labelled reviews to 25,000; and an attention-pooling classifier import gzip, jax, jax.numpy as jnp gv_words, gv_vecs = [], [] with gzip.open('data/glove100.gz', 'rt', encoding='utf8') as f: next(f) # header line: 400000 100 for i, line in enumerate(f): if i == 100000: break # the 100,000 most frequent words p = line.rstrip().split(' '); gv_words.append(p[0]); gv_vecs.append(np.array(p[1:], np.float32)) G = np.stack(gv_vecs); gidx = {w: i for i, w in enumerate(gv_words)} print(f'GloVe: {G.shape[0]:,} words x {G.shape[1]} dimensions (trained on Wikipedia + Gigaword, 6 billion tokens)') tokens = lambda s: re.findall(r"[a-z]+(?:'[a-z]+)?|[!?]", s.lower()) def glove_avg(texts): out = np.zeros((len(texts), 100), np.float32) for j, s in enumerate(texts): ix = [gidx[w] for w in tokens(s) if w in gidx] if ix: out[j] = G[ix].mean(0) return out A_tr, A_te = glove_avg(imdb.text.values[tr]), glove_avg(imdb.text.values[te]); mu, sd = A_tr.mean(0), A_tr.std(0) print(f'{"labelled reviews":>16s} {"TF-IDF logistic":>16s} {"GloVe average + logistic":>25s}') for n in [100, 1000, 10000, 25000]: s_ = np.arange(n) a1 = LogisticRegression(C=10, max_iter=3000).fit(Xt_tr[s_], ytr[s_]).score(Xt_te, yte) a2 = LogisticRegression(C=1, max_iter=3000).fit((A_tr[s_] - mu) / sd, ytr[s_]).score((A_te - mu) / sd, yte) print(f'{n:16,d} {a1:16.4f} {a2:25.4f}') glove_lr = LogisticRegression(C=1, max_iter=3000).fit((A_tr - mu) / sd, ytr) # kept for Section 4 # Attention pooling over frozen GloVe vectors: weights a_i = softmax(v . tanh(W e_i)), document = sum_i a_i tanh(U e_i) L_max = 300 def ids_of(texts): M = np.zeros((len(texts), L_max), np.int32); mask = np.zeros((len(texts), L_max), bool) for j, s in enumerate(texts): ix = [gidx[w] for w in tokens(s) if w in gidx][:L_max]; M[j, :len(ix)] = ix; mask[j, :len(ix)] = True return M, mask I_tr, M_tr = ids_of(imdb.text.values[tr]); I_te, M_te = ids_of(imdb.text.values[te]); Gj = jnp.array(G) k1, k2, k3 = jax.random.split(jax.random.PRNGKey(0), 3) ap = {'W': jax.random.normal(k1, (100, 64)) * 0.1, 'v': jax.random.normal(k2, (64,)) * 0.1, 'U': jax.random.normal(k3, (100, 128)) * 0.1, 'u': jnp.zeros(128), 'c': 0.0} def attend(p, I, M): E = Gj[I] # (batch, L, 100), frozen s = jnp.where(M, jnp.tanh(E @ p['W']) @ p['v'], -1e9); a = jax.nn.softmax(s, axis=1) return a, jnp.einsum('bl,bld->bd', a, jnp.tanh(E @ p['U'])) @ p['u'] + p['c'] def ap_loss(p, I, M, yb): z = attend(p, I, M)[1]; return jnp.mean(jnp.logaddexp(0, z) - yb * z) @jax.jit def ap_step(p, m, v, t, I, M, yb): g = jax.grad(ap_loss)(p, I, M, yb) m = jax.tree_util.tree_map(lambda a, b: 0.9 * a + 0.1 * b, m, g); v = jax.tree_util.tree_map(lambda a, b: 0.999 * a + 0.001 * b * b, v, g) return jax.tree_util.tree_map(lambda q, a, b: q - 2e-3 * (a / (1 - 0.9 ** t)) / (jnp.sqrt(b / (1 - 0.999 ** t)) + 1e-8), p, m, v), m, v m_, v_ = jax.tree_util.tree_map(jnp.zeros_like, ap), jax.tree_util.tree_map(jnp.zeros_like, ap); t = 0; rng = np.random.default_rng(0) ap_pred = jax.jit(lambda p, I, M: attend(p, I, M)[1] > 0) for ep in range(4): for b_ in np.array_split(rng.permutation(25000), 25000 // 64): t += 1; ap, m_, v_ = ap_step(ap, m_, v_, t, I_tr[b_], M_tr[b_], ytr[b_].astype(np.float32)) acc = np.mean(np.concatenate([np.asarray(ap_pred(ap, I_te[i:i + 2500], M_te[i:i + 2500])) for i in range(0, 25000, 2500)]) == yte) print(f'attention pooling, epoch {ep + 1}: test accuracy {acc:.4f}') a_ex = np.asarray(attend(ap, I_te[:1], M_te[:1])[0][0]); n_ex = M_te[0].sum() top = np.argsort(-a_ex[:n_ex])[:8] print('first test review:', imdb.text.values[te[0]][:150].strip() + '...') print(' label', 'positive' if yte[0] else 'negative', '| most attended words:', ', '.join(f'{gv_words[I_te[0, i]]} ({a_ex[i]:.2f})' for i in top))
GloVe: 100,000 words x 100 dimensions (trained on Wikipedia + Gigaword, 6 billion tokens)
labelled reviews TF-IDF logistic GloVe average + logistic
100 0.7350 0.6894
1,000 0.8387 0.7726
10,000 0.8983 0.7951
25,000 0.9152 0.8012
attention pooling, epoch 1: test accuracy 0.8396
attention pooling, epoch 2: test accuracy 0.8457
attention pooling, epoch 3: test accuracy 0.8502
attention pooling, epoch 4: test accuracy 0.8502
first test review: These slasher pics are past their sell by date, but this one is good fun. The valentine cards themselves are witty, and well thought out. The film h...
label positive | most attended words: sizzling (0.06), sorry (0.05), nothing (0.02), feeling (0.02), fun (0.02), enough (0.02), moments (0.02), scary (0.02)
The pre-trained vectors do not help here. With 100 labelled reviews, TF-IDF logistic regression is right 73.5% of the time and the GloVe average 68.9%; with all 25,000 the gap widens to 91.5% against 80.1%. (Tuning the regularisation does not change the order.) Averaging throws away which words occur, and a review's sentiment rests on a few decisive words ("waste", "excellent") whose information is diluted in the average of 200 vectors. The attention-pooling model, which learns how much each word should count, recovers part of the loss: 85.0% after four epochs. But its weights are spread thinly; on the first test review the most attended word gets only 6% of the weight, and the top words ("sizzling", "sorry", "nothing", "fun") are a mixture of sentiment words and words a reader would not single out. Attention weights are not explanations: Jain and Wallace (2019) showed that very different attention distributions can produce the same predictions, so reading a model's reasons off its attention needs care.
The real gains from pre-training come from contextual representations: a transformer (Lecture 12) gives each word a vector that depends on the whole sentence, so "not good" and "good" no longer share their evidence. Fine-tuned end to end on the IMDb training set, BERT-base reaches about 94% test accuracy (Sun et al., 2019, report 94.6% with careful fine-tuning), and DistilBERT, a model 40% smaller, 92.8% (Sanh et al., 2019). Fine-tuning is beyond our CPU budget, but we can measure a public DistilBERT that was fine-tuned on a different data set, the Stanford Sentiment Treebank's single sentences from film reviews (SST-2; Socher et al., 2013), and has never seen an IMDb review. The same box scores the Financial PhraseBank of Section 2 with FinBERT (Araci, 2019), a BERT fine-tuned for financial sentiment.
# Pre-trained transformers, measured: DistilBERT (fine-tuned on SST-2 movie-review sentences) on IMDb, and FinBERT on the PhraseBank import torch, time from transformers import AutoTokenizer, AutoModelForSequenceClassification def classify(name, texts, max_len=512, head_tail=False, bs=16): tk = AutoTokenizer.from_pretrained(name); m = AutoModelForSequenceClassification.from_pretrained(name).eval(); P = [] order = np.argsort([len(t) for t in texts]) # batch similar lengths together: much less padding for i in range(0, len(texts), bs): ids = [tk(texts[j], add_special_tokens=False).input_ids for j in order[i:i + bs]] if head_tail: ids = [x if len(x) <= max_len - 2 else x[:128] + x[-(max_len - 130):] for x in ids] # first 128 + last 382 tokens else: ids = [x[:max_len - 2] for x in ids] ids = [[tk.cls_token_id] + x + [tk.sep_token_id] for x in ids]; T = max(map(len, ids)) x = torch.tensor([s + [tk.pad_token_id] * (T - len(s)) for s in ids]); att = (x != tk.pad_token_id).long() with torch.no_grad(): P.append(torch.softmax(m(input_ids=x, attention_mask=att).logits, -1).numpy()) P = np.concatenate(P); out = np.empty_like(P); out[order] = P; return out, m.config.id2label sub = te[:1000]; texts = list(imdb.text.values[sub]); ys = y_all[sub]; t0 = time.time() P1, lab1 = classify('distilbert/distilbert-base-uncased-finetuned-sst-2-english', texts) P2, _ = classify('distilbert/distilbert-base-uncased-finetuned-sst-2-english', texts, head_tail=True) ntok = AutoTokenizer.from_pretrained('distilbert/distilbert-base-uncased-finetuned-sst-2-english') long_ = np.array([len(ntok(t).input_ids) > 512 for t in texts]) print(f"IMDb, the first 1,000 test reviews ({long_.mean():.0%} longer than 512 tokens); labels {lab1}; {time.time() - t0:.0f} s") print(f" logistic regression, TF-IDF, trained on 25,000 IMDb reviews: {lr.score(Xt_te[:1000], yte[:1000]):.1%}") print(f" DistilBERT-SST-2, never trained on IMDb, first 512 tokens: {np.mean(P1.argmax(1) == ys):.1%}") print(f" the same, first 128 + last 382 tokens (Sun et al., 2019): {np.mean(P2.argmax(1) == ys):.1%}" f" (on the long reviews alone: {np.mean(P1.argmax(1)[long_] == ys[long_]):.1%} -> {np.mean(P2.argmax(1)[long_] == ys[long_]):.1%})") fpb = pd.read_csv('data/FPB_Sentences_AllAgree.txt', sep='@', header=None, names=['text', 'label'], encoding='latin-1') labf = fpb.label.map({'negative': -1, 'neutral': 0, 'positive': 1}).values; pol = labf != 0 P3, lab3 = classify('ProsusAI/finbert', list(fpb.text)) pf = np.array([{'positive': 1, 'negative': -1, 'neutral': 0}[lab3[k]] for k in P3.argmax(1)]) P4, _ = classify('distilbert/distilbert-base-uncased-finetuned-sst-2-english', list(fpb.text)) pd_ = np.where(P4.argmax(1) == 1, 1, -1) # SST-2 has no neutral class row = lambda p: f"{np.mean(p[pol] == labf[pol]):5.0%} {np.mean(p[pol] == 0):6.0%} {np.mean(p[pol] == -labf[pol]):5.0%}" print("Financial PhraseBank, 2,264 all-agree sentences 3-class right neutral wrong (on the 873 polar sentences)") print(f" DistilBERT-SST-2 (general sentiment, 2 classes) - {row(pd_)}") print(f" FinBERT (fine-tuned on the PhraseBank itself) {np.mean(pf == labf):5.1%} {row(pf)}")
IMDb, the first 1,000 test reviews (15% longer than 512 tokens); labels {0: 'NEGATIVE', 1: 'POSITIVE'}; 267 s
logistic regression, TF-IDF, trained on 25,000 IMDb reviews: 91.7%
DistilBERT-SST-2, never trained on IMDb, first 512 tokens: 88.7%
the same, first 128 + last 382 tokens (Sun et al., 2019): 89.6% (on the long reviews alone: 80.8% -> 87.0%)
Financial PhraseBank, 2,264 all-agree sentences 3-class right neutral wrong (on the 873 polar sentences)
DistilBERT-SST-2 (general sentiment, 2 classes) - 67% 0% 33%
FinBERT (fine-tuned on the PhraseBank itself) 97.2% 98% 0% 2%
Without a single IMDb label, DistilBERT is right on 88.7% of the first 1,000 test reviews, close to the 91.7% of logistic regression trained on 25,000 of them. BERT-style models read at most 512 tokens, and 15% of these reviews are longer; Sun et al. found that keeping the first 128 and the last 382 tokens works better than keeping the first 512, because reviews tend to end with the verdict, and so it does here: 80.8% to 87.0% on the long reviews, 89.6% overall. On financial news the same model fails like the movie-review logistic regression of Section 2, if less badly: it has no neutral class, and it gives the wrong sign to 33% of the sentences that do have a sentiment (the movie-review model: 47%). FinBERT scores 97.2% on the three classes, far above every other method in this lecture. The number is not a fair test: FinBERT was fine-tuned on the Financial PhraseBank itself (Araci held out a random 20% for testing), so most of these sentences were in its training data. A model evaluated on its own training data looks excellent, and Lecture 23 returns to this kind of leakage. On fresh financial text its accuracy would be lower, by an amount this data set cannot tell us.
Word vectors learn meaning from co-occurrence (Lecture 12, Theorem 12.1), so they learn whatever associations the training text contains: that Paris is to France as Tokyo is to Japan, and also that some occupations are mentioned mostly alongside women and others alongside men. Bolukbasi et al. (2016) put it in their title: "Man is to computer programmer as woman is to homemaker?" Such associations matter because embeddings are inputs to systems that rank CVs, route customer complaints and flag content. Three tools measure them: analogies, projections onto a direction, and association tests.
# Bias in representations: analogies and a gender direction in GloVe Gn = G / np.linalg.norm(G, axis=1, keepdims=True) def analogy(a, b, c, exclude=True, k=3): # a : b :: c : ? by the vector b - a + c q = Gn[gidx[b]] - Gn[gidx[a]] + Gn[gidx[c]]; s = Gn @ (q / np.linalg.norm(q)) if exclude: s[[gidx[a], gidx[b], gidx[c]]] = -np.inf return [gv_words[i] for i in np.argsort(-s)[:k]] for a, b, c in [('man', 'king', 'woman'), ('paris', 'france', 'tokyo'), ('man', 'doctor', 'woman'), ('man', 'programmer', 'woman'), ('he', 'surgeon', 'she'), ('man', 'engineer', 'woman')]: print(f'{a}:{b} :: {c}:? excluding inputs: {", ".join(analogy(a, b, c))} allowing them: {", ".join(analogy(a, b, c, False))}') pairs = [('she', 'he'), ('her', 'his'), ('woman', 'man'), ('mother', 'father'), ('daughter', 'son'), ('girl', 'boy'), ('female', 'male'), ('herself', 'himself')] pairs_c = np.concatenate([np.stack([Gn[gidx[f]], Gn[gidx[m]]]) - (Gn[gidx[f]] + Gn[gidx[m]]) / 2 for f, m in pairs]) _, sv, Vt = np.linalg.svd(pairs_c, full_matrices=False); g = Vt[0] * np.sign(Vt[0] @ (Gn[gidx['she']] - Gn[gidx['he']])) print(f'gender direction: first principal component of the {len(pairs)} centred pairs explains {sv[0] ** 2 / (sv ** 2).sum():.0%} of their variance') jobs = ['nurse', 'receptionist', 'librarian', 'hairdresser', 'secretary', 'teacher', 'dancer', 'designer', 'writer', 'lawyer', 'doctor', 'scientist', 'professor', 'manager', 'pilot', 'engineer', 'architect', 'programmer', 'surgeon', 'carpenter', 'mechanic', 'plumber'] proj = sorted(((Gn[gidx[w]] @ g, w) for w in jobs), reverse=True) print('projection on the she-he direction (positive = closer to "she"):') print(' ' + ', '.join(f'{w} {p:+.2f}' for p, w in proj))
man:king :: woman:? excluding inputs: queen, monarch, throne allowing them: king, queen, monarch paris:france :: tokyo:? excluding inputs: japan, korea, japanese allowing them: japan, tokyo, korea man:doctor :: woman:? excluding inputs: nurse, physician, doctors allowing them: doctor, nurse, physician man:programmer :: woman:? excluding inputs: educator, programmers, linguist allowing them: programmer, educator, programmers he:surgeon :: she:? excluding inputs: nurse, dentist, physician allowing them: surgeon, nurse, dentist man:engineer :: woman:? excluding inputs: technician, educator, engineers allowing them: engineer, technician, educator gender direction: first principal component of the 8 centred pairs explains 74% of their variance projection on the she-he direction (positive = closer to "she"): nurse +0.30, receptionist +0.29, hairdresser +0.25, dancer +0.23, librarian +0.12, designer +0.06, teacher +0.05, doctor +0.01, writer -0.00, surgeon -0.03, carpenter -0.08, pilot -0.09, plumber -0.09, lawyer -0.10, professor -0.11, scientist -0.13, programmer -0.13, secretary -0.18, mechanic -0.20, engineer -0.24, architect -0.25, manager -0.33
The analogy man : king :: woman : ? returns "queen", and man : doctor :: woman : ? returns "nurse", the kind of result that made Bolukbasi et al.'s paper famous; he : surgeon :: she : ? also gives "nurse". But look at the second column. The standard analogy search excludes the three input words from the answers. When they are allowed, the nearest word to \(\text{doctor}-\text{man}+\text{woman}\) is "doctor" itself, and similarly for "programmer", "engineer" and "surgeon": the vectors say mainly that a woman doctor is a doctor. Projections onto a gender direction are a more direct measurement. Taking the first principal component of eight centred pairs (she/he, her/his, woman/man, …), which explains 74% of their variance, "nurse", "receptionist", "hairdresser" and "dancer" lie on the "she" side, and "manager", "architect", "engineer" and "mechanic" on the "he" side, mirroring occupational statistics (Caliskan et al. found a correlation of 0.90 between such projections and the percentage of women in each occupation in US labour statistics). Two cautions: "secretary" falls on the "he" side, because in news text "secretary" is mostly "Secretary of State"; and a single direction captures only part of how gender is encoded. Bolukbasi et al. proposed "debiasing" by removing the gender direction from words that should be neutral, but Gonen and Goldberg (2019) showed that the bias remained recoverable from the debiased vectors' neighbourhoods: projecting out one direction hides the symptom more than it removes the cause.
Analogies are a vivid but unreliable measuring instrument. A systematic test asks whether a whole set of target words is more strongly associated with one set of attribute words than another. Caliskan, Bryson and Narayanan (2017) adapted the Implicit Association Test of psychology (Greenwald, McGhee and Schwartz, 1998), which measures how much faster people pair, say, flowers with pleasant words than insects with pleasant words, into a test on vectors.
Let \(X,Y\) be two sets of target words of equal size (flowers, insects) and \(A,B\) two sets of attribute words (pleasant, unpleasant). The association of a word \(w\) with the attributes is
$$s(w,A,B)=\frac1{|A|}\sum_{a\in A}\cos(w,a)-\frac1{|B|}\sum_{b\in B}\cos(w,b),$$the test statistic is
$$S(X,Y,A,B)=\sum_{x\in X}s(x,A,B)-\sum_{y\in Y}s(y,A,B),$$and the effect size is
$$d=\frac{\operatorname{mean}_{x\in X}s(x,A,B)-\operatorname{mean}_{y\in Y}s(y,A,B)}{\operatorname{sd}_{w\in X\cup Y}\,s(w,A,B)},$$a standardised difference of means like Cohen's \(d\) (values around 0.8 and above are conventionally "large").
Let \(\{X_i,Y_i\}\) range over all ways of splitting \(X\cup Y\) into two sets of equal size, and define the one-sided \(p\)-value
$$p=\Pr_i\big[S(X_i,Y_i,A,B)\ge S(X,Y,A,B)\big].$$If the labels "\(X\)" and "\(Y\)" are exchangeable (the null hypothesis that the two target sets do not differ in their association with \(A\) and \(B\)), then \(\Pr(p\le\alpha)\le\alpha\) for every \(\alpha\in[0,1]\).
The same holds when \(p\) is estimated from \(M\) random splits as
$$(1+\#\{m:S_m\ge S\})/(M+1).$$Condition on the multiset of association values \(\{s(w,A,B):w\in X\cup Y\}\). Under exchangeability, the observed split is equally likely to be any of the \(K=\binom{2n}{n}\) splits, so the observed statistic is a uniformly random draw from the list \(S_1,\dots,S_K\) of the statistics of all splits. Its rank from the top is then uniform, and \(p\) is its rank divided by \(K\). Hence \(\Pr(p\le k/K)\le k/K\) for each \(k\) (with equality when there are no ties), and so \(\Pr(p\le\alpha)\le\alpha\).
For the Monte Carlo version, under the null the observed statistic and the \(M\) random-split statistics are exchangeable, so the observed one is equally likely to occupy any of the \(M+1\) ranks, which gives the same bound for \((1+\#\{S_m\ge S\})/(M+1)\).
Note what the test does not say: the randomness is over which words were called \(X\), so it treats the word lists as given. A different, equally reasonable choice of words can change \(d\), and Ethayarajh, Duvenaud and Hirst (2019) showed how strongly WEAT results can depend on the choice of word lists. \(\square\)
# The Word-Embedding Association Test (WEAT), reproducing Caliskan, Bryson and Narayanan (2017) on GloVe W_ = lambda s: s.split() weat = { 'flowers vs insects / pleasant vs unpleasant': (W_('aster clover hyacinth marigold poppy azalea crocus iris orchid rose bluebell daffodil lilac pansy tulip buttercup daisy lily peony violet carnation gladiola magnolia petunia zinnia'), W_('ant caterpillar flea locust spider bedbug centipede fly maggot tarantula bee cockroach gnat mosquito termite beetle cricket hornet moth wasp blackfly dragonfly horsefly roach weevil'), 'P', 'U', 1.50), 'instruments vs weapons / pleasant vs unpleasant': (W_('bagpipe cello guitar lute trombone banjo clarinet harmonica mandolin trumpet bassoon drum harp oboe tuba bell fiddle harpsichord piano viola bongo flute horn saxophone violin'), W_('arrow club gun missile spear axe dagger harpoon pistol sword blade dynamite hatchet rifle tank bomb firearm knife shotgun teargas cannon grenade mace slingshot whip'), 'P', 'U', 1.53), 'European- vs African-American names / pleasant vs unpleasant': (W_('adam harry josh roger alan frank justin ryan andrew jack matthew stephen brad greg paul jonathan peter amanda courtney heather melanie katie betsy kristin nancy stephanie ellen lauren colleen emily megan rachel'), W_('alonzo jamel theo alphonse jerome leroy torrance darnell lamar lionel tyree deion lamont malik terrence tyrone lavon marcellus wardell nichelle shereen ebony latisha shaniqua jasmine tanisha tia lakisha latoya yolanda malika yvette'), 'P', 'U', 1.41), 'male vs female names / career vs family': (W_('john paul mike kevin steve greg jeff bill'), W_('amy joan lisa sarah diana kate ann donna'), W_('executive management professional corporation salary office business career'), W_('home parents children family cousins marriage wedding relatives'), 1.81), 'maths vs arts / male vs female terms': (W_('math algebra geometry calculus equations computation numbers addition'), W_('poetry art dance literature novel symphony drama sculpture'), W_('male man boy brother he him his son'), W_('female woman girl sister she her hers daughter'), 1.06), 'science vs arts / male vs female terms': (W_('science technology physics chemistry einstein nasa experiment astronomy'), W_('poetry art shakespeare dance literature novel symphony drama'), W_('brother father uncle grandfather son he his him'), W_('sister mother aunt grandmother daughter she hers her'), 1.24)} PLEAS = W_('caress freedom health love peace cheer friend heaven loyal pleasure diamond gentle honest lucky rainbow diploma gift honor miracle sunrise family happy laughter paradise vacation') UNPL = W_('abuse crash filth murder sickness accident death grief poison stink assault disaster hatred pollute tragedy divorce jail poverty ugly cancer kill rotten vomit agony prison') def weat_test(X, Y, A, B, n_perm=100000, seed=0): X, Y, A, B = ([w for w in S if w in gidx] for S in (X, Y, A, B)) s = lambda w: Gn[[gidx[a] for a in A]] @ Gn[gidx[w]] # cosines with A assoc = np.array([s(w).mean() - (Gn[[gidx[b] for b in B]] @ Gn[gidx[w]]).mean() for w in X + Y]) sx, sy = assoc[:len(X)], assoc[len(X):] d = (sx.mean() - sy.mean()) / assoc.std(ddof=1) # effect size r = np.random.default_rng(seed); n = len(X); stat = sx.sum() - sy.sum(); more = 0 for _ in range(n_perm): # permutation test over equal-size splits of X u Y p = r.permutation(len(assoc)); more += assoc[p[:n]].sum() - assoc[p[n:2 * n]].sum() >= stat return d, (more + 1) / (n_perm + 1), len(X), len(Y) print(f'{"test":60s} {"words":>7s} {"effect size d":>14s} {"p (one-sided)":>14s} {"Caliskan et al.":>16s}') for name, (X, Y, A, B, ref) in weat.items(): A, B = (PLEAS if A == 'P' else A), (UNPL if B == 'U' else B) d, p, nx, ny = weat_test(X, Y, A, B) print(f'{name:60s} {nx:3d}+{ny:<3d} {d:14.2f} {p:14.1e} {ref:16.2f}')
test words effect size d p (one-sided) Caliskan et al. flowers vs insects / pleasant vs unpleasant 22+22 1.22 1.0e-05 1.50 instruments vs weapons / pleasant vs unpleasant 25+25 1.30 1.0e-05 1.53 European- vs African-American names / pleasant vs unpleasant 32+24 0.80 1.4e-03 1.41 male vs female names / career vs family 8+8 1.72 1.0e-04 1.81 maths vs arts / male vs female terms 8+8 1.28 2.6e-03 1.06 science vs arts / male vs female terms 8+8 1.12 8.1e-03 1.24
All six associations are reproduced, with large effect sizes and small permutation \(p\)-values: flowers and musical instruments are closer to pleasant words than insects and weapons; European-American names are closer to pleasant words than African-American names; male names to career words and female names to family words; and male terms to mathematics and science than to the arts. The effect sizes differ from Caliskan et al.'s, mostly downwards, for three reasons worth knowing. They used vectors trained on 840 billion words of web text; ours come from six billion words of Wikipedia and newswire. Some words in their lists are not among our 100,000 vocabulary words (three flowers and three insects, and eight of the African-American names), and the test drops them. And the effect size has a large standard error with eight words per set: a single ambiguous word ("jasmine" is also a flower, "jack" a tool) can move it noticeably. The \(p\)-values say the associations are not artefacts of which words were called \(X\); they do not say what the associations will do in an application. For that, test the application.
The standard analogy code returns "nurse" for man : doctor :: woman : ?, but when the input words are allowed as answers, it returns "doctor". What does this show?
Associations in an embedding are not yet harm; harm happens when a system built on them treats people differently. The direct test is an audit with counterfactual inputs: feed the system inputs that differ only in a protected attribute, or a proxy for one such as a name, and compare the outputs. Kiritchenko and Mohammad (2018) built the Equity Evaluation Corpus of 8,640 template sentences for this purpose and found that many of 219 sentiment systems scored sentences with African-American names or female terms systematically differently. Speer (2017) showed that a sentiment classifier built in the obvious way on GloVe vectors rated "Let's go get Mexican food" well below "Let's go get Italian food". We can check both on our own models.
# Does representation bias reach decisions? Score template sentences that differ only in a name EA, AA = weat['European- vs African-American names / pleasant vs unpleasant'][:2] EA, AA = [w for w in EA if w in gidx], [w for w in AA if w in gidx] templates = ['I went to see a movie with {} last night.', '{} wrote the screenplay.', 'The film stars {} as the detective.', 'My friend {} recommended it.', '{} directed this one.'] def logodds_glove(texts): return glove_lr.decision_function((glove_avg(texts) - mu) / sd) def logodds_tfidf(texts): return lr.decision_function(tf.transform(texts)) print(f'{len(templates)} templates such as "{templates[0]}"; {len(EA)} European-American and {len(AA)} African-American names') for name, f in [('GloVe average + logistic', logodds_glove), ('TF-IDF logistic', logodds_tfidf)]: score = lambda names: np.mean([f([t.format(n.capitalize()) for n in names]) for t in templates], axis=0) # one number per name se, sa = score(EA), score(AA) diff = se.mean() - sa.mean(); allv = np.concatenate([se, sa]); r = np.random.default_rng(0) perm_d = np.array([(lambda p: allv[p[:len(se)]].mean() - allv[p[len(se):]].mean())(r.permutation(len(allv))) for _ in range(20000)]) print(f'{name:25s} mean log-odds: European-American {se.mean():+.3f}, African-American {sa.mean():+.3f}; ' f'difference {diff:+.3f} (permutation p = {(np.sum(perm_d >= diff) + 1) / 20001:.3f})') for cuisine in ['Italian', 'French', 'Chinese', 'Mexican', 'Indian']: s_ = f"Let's go get {cuisine} food tonight." print(f' "{s_}" GloVe model log-odds {logodds_glove([s_])[0]:+.3f} TF-IDF model {logodds_tfidf([s_])[0]:+.3f}')
5 templates such as "I went to see a movie with {} last night."; 32 European-American and 24 African-American names
GloVe average + logistic mean log-odds: European-American +0.808, African-American +0.394; difference +0.415 (permutation p = 0.289)
TF-IDF logistic mean log-odds: European-American +0.153, African-American +0.019; difference +0.134 (permutation p = 0.091)
"Let's go get Italian food tonight." GloVe model log-odds -2.276 TF-IDF model +0.100
"Let's go get French food tonight." GloVe model log-odds -1.724 TF-IDF model +0.321
"Let's go get Chinese food tonight." GloVe model log-odds -5.851 TF-IDF model +0.026
"Let's go get Mexican food tonight." GloVe model log-odds -3.952 TF-IDF model -0.312
"Let's go get Indian food tonight." GloVe model log-odds -3.370 TF-IDF model -0.158
Averaged over five templates, both models score sentences with European-American names more positively than the same sentences with African-American names, as the embedding association would predict, but the differences are small and not statistically significant (\(p=0.29\) for the GloVe model and \(p=0.09\) for the TF-IDF model, by a permutation test over names). With 56 names, an audit like this cannot rule out a modest effect, and it does not show one either: a larger set of names and templates, as in the Equity Evaluation Corpus, would be needed. The TF-IDF model hardly knows most of the names, so it barely reacts to them; its small differences come from names that happen to be characters or actors in reviews. The cuisine sentences are more striking. The GloVe model calls every one negative, and its scores range from \(-1.7\) for French food to \(-5.9\) for Chinese food, a factor of \(e^{4.1}\approx60\) in odds, for sentences with nothing to do with films. The TF-IDF model, which has seen these words only in reviews, scores them all near zero. A model built on general-purpose representations imports their associations into a domain where they have no business.
Does a contextual model behave better? The same audit on DistilBERT:
# The name audit above, repeated on DistilBERT: P(positive) as log-odds, averaged over the five templates tkd = AutoTokenizer.from_pretrained('distilbert/distilbert-base-uncased-finetuned-sst-2-english') mdl = AutoModelForSequenceClassification.from_pretrained('distilbert/distilbert-base-uncased-finetuned-sst-2-english').eval() def logodds_bert(texts): with torch.no_grad(): z = mdl(**tkd(texts, return_tensors='pt', padding=True)).logits return (z[:, 1] - z[:, 0]).numpy() score = lambda names: np.mean([logodds_bert([t.format(n.capitalize()) for n in names]) for t in templates], axis=0) se, sa = score(EA), score(AA); diff = se.mean() - sa.mean(); allv = np.concatenate([se, sa]); r = np.random.default_rng(0) perm_d = np.array([(lambda p: allv[p[:len(se)]].mean() - allv[p[len(se):]].mean())(r.permutation(len(allv))) for _ in range(20000)]) print(f'DistilBERT-SST-2 on the name templates: mean log-odds European-American {se.mean():+.3f}, African-American {sa.mean():+.3f}; ' f'difference {diff:+.3f} (permutation p = {(np.sum(perm_d >= diff) + 1) / 20001:.3f})') for cuisine in ['Italian', 'French', 'Chinese', 'Mexican', 'Indian']: s_ = f"Let's go get {cuisine} food tonight."; print(f' "{s_}" DistilBERT log-odds {logodds_bert([s_])[0]:+.3f}')
DistilBERT-SST-2 on the name templates: mean log-odds European-American +5.962, African-American +5.800; difference +0.162 (permutation p = 0.150) "Let's go get Italian food tonight." DistilBERT log-odds +2.440 "Let's go get French food tonight." DistilBERT log-odds +3.018 "Let's go get Chinese food tonight." DistilBERT log-odds +3.565 "Let's go get Mexican food tonight." DistilBERT log-odds +2.808 "Let's go get Indian food tonight." DistilBERT log-odds +4.221
DistilBERT scores every template as strongly positive (mean log-odds +5.962 with European-American names, +5.800 with African-American names), and the difference, +0.162, is again small and not significant (\(p=0.150\)). The cuisine sentences are all positive too, from +2.440 for Italian food to +4.221 for Indian, a spread of a factor of about 6 in odds, against 60 for the GloVe model. On this small audit the contextual model shows neither of the GloVe model's patterns. That is a statement about these 56 names and five sentences, not a certificate: larger audits have found such gaps in many sentiment systems (Kiritchenko and Mohammad, 2018), and in the representations of BERT itself (Kurita et al., 2019). An audit has to be repeated for every model and every use.
In 2018 Joy Buolamwini and Timnit Gebru tested three commercial face-analysis systems (from IBM, Microsoft and Face++) on 1,270 photographs of parliamentarians from three African and three European countries, balanced by gender and skin type. Classifying gender, the systems erred on at most 0.8% of lighter-skinned men but on up to 34.7% of darker-skinned women. The benchmarks the companies had used were dominated by lighter-skinned faces, so their headline accuracies had hidden the gap. The companies released improved models within months, and a follow-up audit (Raji and Buolamwini, 2019) found the gaps had narrowed for the audited firms but not for others. The study made disaggregated evaluation (reporting performance separately for each group, and for intersections of groups) a standard part of responsible machine learning, and fed into the documentation practices of "datasheets for datasets" and "model cards".
To say what a fair decision rule would be, fix notation. A model produces a score \(R\) (a risk, a probability) and perhaps a decision \(\hat Y\) (detain, approve); \(Y\) is the outcome (reoffends, repays); \(A\) is a protected attribute (race, sex). Most proposed definitions of group fairness are conditional independence statements among these three variables, and there are only three basic ones (Barocas, Hardt and Narayanan, 2023).
Each has an argument for it. Independence says outcomes should be shared out equally, whatever the base rates; it is the idea behind the "four-fifths rule" of US employment law, under which a selection rate for any group below 80% of the highest group's is evidence of adverse impact. Separation says people who will turn out the same way should be treated the same way: a person who would not have reoffended should not be more likely to be detained because of their group. Sufficiency says a score should mean what it says for everyone, so that a judge can read "70% risk" the same way for any defendant. In 2016 these three arguments collided in public.
COMPAS (Correctional Offender Management Profiling for Alternative Sanctions), sold by Northpointe (now Equivant), scores defendants' risk of reoffending from a questionnaire and criminal record, on a scale of deciles 1–10. ProPublica obtained the scores of 7,214 defendants assessed in Broward County in 2013–2014 through public-records requests, matched them with criminal records over the following two years, and published the data and analysis (Angwin, Larson, Mattu and Kirchner, 2016; github.com/propublica/compas-analysis). We apply ProPublica's filters and keep the 5,278 African-American and white defendants; "reoffended" means a new arrest within two years, which measures policing as well as crime, a limitation discussed in Section 9. ProPublica, like us, calls a decile of 5 or more "medium or high risk". Names and other identifying fields in the file are not used.
# Auditing COMPAS: error rates, predictive values and calibration by race (ProPublica's Broward County data) cp = pd.read_csv('data/compas-scores-two-years.csv') cp = cp[(cp.days_b_screening_arrest.abs() <= 30) & (cp.is_recid != -1) & (cp.c_charge_degree != 'O') & (cp.score_text != 'N/A')] # ProPublica's filters cp = cp[cp.race.isin(['African-American', 'Caucasian'])].copy() cp['high'] = (cp.decile_score >= 5).astype(int); y_c = cp.two_year_recid.values # "medium or high risk" = decile 5-10 print(f'{len(cp):,} defendants (African-American or white), two-year recidivism rate {y_c.mean():.1%}') def rates(pred, y): tp, fp = np.sum((pred == 1) & (y == 1)), np.sum((pred == 1) & (y == 0)); fn, tn = np.sum((pred == 0) & (y == 1)), np.sum((pred == 0) & (y == 0)) return dict(n=len(y), base=y.mean(), flagged=pred.mean(), FPR=fp / (fp + tn), FNR=fn / (fn + tp), PPV=tp / (tp + fp), NPV=tn / (tn + fn), acc=(tp + tn) / len(y)) print(f'{"group":17s} {"n":>5s} {"base rate":>9s} {"flagged":>8s} {"FPR":>6s} {"FNR":>6s} {"PPV":>6s} {"NPV":>6s} {"accuracy":>8s}') R = {} for grp, lbl in [('African-American', 'Black'), ('Caucasian', 'white')]: s_ = cp.race.values == grp; R[lbl] = r_ = rates(cp.high.values[s_], y_c[s_]) print(f'{lbl:17s} {r_["n"]:5d} {r_["base"]:9.1%} {r_["flagged"]:8.1%} {r_["FPR"]:6.1%} {r_["FNR"]:6.1%} {r_["PPV"]:6.1%} {r_["NPV"]:6.1%} {r_["acc"]:8.1%}') for lbl, r_ in R.items(): # Chouldechova's identity: FPR = p/(1-p) * (1-PPV)/PPV * (1-FNR) print(f'Chouldechova identity, {lbl}: FPR = {r_["FPR"]:.4f}; p/(1-p) (1-PPV)/PPV (1-FNR) = {r_["base"] / (1 - r_["base"]) * (1 - r_["PPV"]) / r_["PPV"] * (1 - r_["FNR"]):.4f}') print('recidivism rate by COMPAS decile (calibration):') tab = cp.groupby(['decile_score', 'race']).two_year_recid.agg(['mean', 'size']).unstack() print(' decile ' + ' '.join(f'{d:>5d}' for d in range(1, 11))) for grp, lbl in [('African-American', 'Black'), ('Caucasian', 'white')]: print(f' {lbl:8s} ' + ' '.join(f'{v:5.2f}' for v in tab[('mean', grp)].values)) # Dressel and Farid (2018): two features do as well from sklearn.model_selection import cross_val_score Xdf = cp[['age', 'priors_count']].values print(f'COMPAS (decile >= 5) accuracy {np.mean(cp.high.values == y_c):.3f}; logistic regression on age and number of priors, ' f'10-fold CV accuracy {cross_val_score(LogisticRegression(), Xdf, y_c, cv=10).mean():.3f}')
5,278 defendants (African-American or white), two-year recidivism rate 47.0% group n base rate flagged FPR FNR PPV NPV accuracy Black 3175 52.3% 57.6% 42.3% 28.5% 65.0% 64.9% 64.9% white 2103 39.1% 33.1% 22.0% 49.6% 59.5% 71.0% 67.2% Chouldechova identity, Black: FPR = 0.4234; p/(1-p) (1-PPV)/PPV (1-FNR) = 0.4234 Chouldechova identity, white: FPR = 0.2201; p/(1-p) (1-PPV)/PPV (1-FNR) = 0.2201 recidivism rate by COMPAS decile (calibration): decile 1 2 3 4 5 6 7 8 9 10 Black 0.23 0.30 0.42 0.47 0.49 0.59 0.61 0.71 0.72 0.84 white 0.21 0.31 0.34 0.40 0.46 0.58 0.60 0.75 0.71 0.70 COMPAS (decile >= 5) accuracy 0.658; logistic regression on age and number of priors, 10-fold CV accuracy 0.677
The table reproduces ProPublica's central finding. Among defendants who were not rearrested within two years, 42.3% of Black defendants had been classed as medium or high risk, against 22.0% of white defendants: Black defendants bore almost twice the false-positive rate. Among those who were rearrested, white defendants were more often classed as low risk (false-negative rate 49.6% against 28.5%). Separation fails. (ProPublica's published figures, 44.9% and 23.5%, come from a slightly different sample; the pattern is the same.) Yet the positive predictive values are close (65.0% and 59.5%), and the calibration table shows that, decile by decile, rearrest rates are similar for the two groups: a defendant with a score of 7 was rearrested about 60% of the time whichever their race. Sufficiency approximately holds, except perhaps in the top decile, where there are only 50 white defendants. Finally, a logistic regression on just two inputs, age and the number of prior offences, matches COMPAS's accuracy (67.7% in cross-validation against 65.8%), as Dressel and Farid found.
The COMPAS numbers are not a coincidence of one tool. Two short theorems, found independently in 2016, show that when base rates differ between groups, no score or decision rule short of a perfect one can satisfy separation and sufficiency together.
For a binary decision in a group with base rate \(p=P(Y=1)\), positive predictive value PPV, false-positive rate FPR and false-negative rate FNR,
$$\mathrm{FPR}=\frac{p}{1-p}\cdot\frac{1-\mathrm{PPV}}{\mathrm{PPV}}\cdot(1-\mathrm{FNR}).$$Consequently, if two groups have different base rates and the same PPV, they cannot also have the same FPR and the same FNR (unless both rates are zero for one of them, or the PPV is 1).
Write the confusion matrix of a group of size \(N\) as counts TP, FP, FN, TN. Then \(\mathrm{TP}+\mathrm{FN}=pN\) and \(\mathrm{FP}+\mathrm{TN}=(1-p)N\), so \(\mathrm{TP}=(1-\mathrm{FNR})\,pN\) and \(\mathrm{FP}=\mathrm{FPR}\,(1-p)N\).
By definition \(\mathrm{PPV}=\mathrm{TP}/(\mathrm{TP}+\mathrm{FP})\), so
$$\mathrm{FP}=\mathrm{TP}\,(1-\mathrm{PPV})/\mathrm{PPV}.$$Substituting,
$$\mathrm{FPR}\,(1-p)N=(1-\mathrm{FNR})\,pN\,\frac{1-\mathrm{PPV}}{\mathrm{PPV}},$$and dividing by \((1-p)N\) gives the identity.
If the PPV and FNR are equal in two groups, the right-hand side differs between the groups through the factor \(p/(1-p)\), which is strictly increasing in \(p\); so the FPRs are equal only if the remaining factor \((1-\mathrm{PPV})(1-\mathrm{FNR})/\mathrm{PPV}\) is zero. \(\square\)
With the Broward numbers the identity holds to four decimal places (the code box checks it). Because more Black than white defendants were rearrested (52% against 39%), a tool with similar predictive values in both groups must have a higher false-positive rate for Black defendants. Kleinberg, Mullainathan and Raghavan proved the corresponding statement for scores rather than decisions.
Consider a score \(S\in[0,1]\) and two groups with base rates \(p_1\ne p_2\). Suppose the score is
Then the score is perfect: \(S=Y\) with probability 1. (Kleinberg et al. also show an approximate version: approximate satisfaction of all three forces approximately equal base rates or approximately perfect prediction.)
Step 1: calibration fixes the average score. In group \(t\), by the tower property,
$$E[S]=E\big[E[Y\mid S]\big]=E[Y]=p_t.$$Writing \(x_t=E[S\mid Y=0]\) and \(y_t=E[S\mid Y=1]\) in group \(t\), the same mean is \((1-p_t)x_t+p_ty_t\), so
$$(1-p_t)\,x_t+p_t\,y_t=p_t,\qquad t=1,2.$$Step 2: balance makes the unknowns common. With \(x_1=x_2=x\) and \(y_1=y_2=y\), both equations hold for the same \((x,y)\). Rearranged, each says \(x+p_t(y-x-1)=0\). Subtracting the two,
$$(p_1-p_2)(y-x-1)=0,$$and since \(p_1\ne p_2\), \(y=x+1\). As \(x,y\in[0,1]\), this forces \(x=0\), \(y=1\).
Step 3: conclude. \(E[S\mid Y=0]=0\) with \(S\ge0\) means \(S=0\) whenever \(Y=0\), and \(E[S\mid Y=1]=1\) with \(S\le1\) means \(S=1\) whenever \(Y=1\), almost surely. \(\square\)
The theorems turn a political argument into arithmetic. ProPublica measured separation and found it violated; Northpointe replied that its scores satisfied sufficiency (Dieterich, Mendoza and Brennan, 2016), which our calibration table confirms to within sampling error. Both were right, and both could not be satisfied at once. Which criterion should give way depends on the decision and on whose interests are at stake, and that is not a question that mathematics can settle. Explore the trade-off yourself.
In a group with base rate \(p=0.4\), a classifier has PPV \(=0.6\) and FNR \(=0.4\). Use Theorem 13.3 to find its false-positive rate (three decimal places).
On 23 May 2016 ProPublica published "Machine Bias", by Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner, under the subheading "There's software used across the country to predict future criminals. And it's biased against blacks." Northpointe's rebuttal argued that the tool was equally accurate for both groups. Within months, Alexandra Chouldechova (then at Carnegie Mellon), Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan, and Sam Corbett-Davies and colleagues at Stanford independently showed that the disagreement was mathematically inevitable. In Wisconsin, the state Supreme Court ruled in State v. Loomis (2016) that judges may consider COMPAS scores at sentencing, provided they are warned of the tool's limitations. In 2018 Julia Dressel and Hany Farid showed that untrained people recruited online, given seven facts about each defendant, predicted reoffending about as accurately as COMPAS (67% against 65%), and that a logistic regression on two features, age and number of prior convictions, did as well as the tool's 137 inputs.
In Lecture 8 we trained XGBoost to predict whether a working Californian earns more than $50,000 a year, from the 2018 American Community Survey as packaged by Folktables (Ding et al., 2021). Folktables was built to replace the 1994 Census extract known as "Adult", which is small, old, and labelled with an income threshold fixed in 1994 dollars (Ding et al., 2021). Suppose such a model were used to decide who is offered a premium financial product. Here is the audit by sex, followed by two common remedies: deleting the sensitive attribute, and adjusting the threshold.
# Auditing the Lecture 8 income model (XGBoost on the 2018 American Community Survey, California) by sex import xgboost as xgb from sklearn.model_selection import train_test_split from sklearn.metrics import roc_auc_score from folktables import ACSIncome acs = pd.read_csv('data/acs/2018/1-Year/psam_p06.csv', usecols=ACSIncome.features + ['PINCP', 'PWGTP']) Xa, ya_, _ = ACSIncome.df_to_pandas(acs); ad = Xa.copy(); ad['y'] = ya_.iloc[:, 0].astype(int).values use = list(ACSIncome.features); cat = ['COW', 'MAR', 'OCCP', 'POBP', 'RELP', 'SEX', 'RAC1P'] for c in cat: ad[c] = ad[c].astype(int).astype('category') Atr, Ate = train_test_split(ad, test_size=0.3, stratify=ad['y'], random_state=0) # the same split as Lecture 8 Afit, Aval = train_test_split(Atr, test_size=0.15, stratify=Atr['y'], random_state=1) def fit_xgb(features, target='y'): return xgb.XGBClassifier(n_estimators=3000, learning_rate=0.05, max_depth=6, subsample=0.8, colsample_bytree=0.8, max_cat_to_onehot=1, enable_categorical=True, tree_method='hist', early_stopping_rounds=100, random_state=0 ).fit(Afit[features], Afit[target].astype(int), eval_set=[(Aval[features], Aval[target].astype(int))], verbose=False) female = (Ate.SEX == 2).values; ya = Ate.y.values # ACS codes sex as 1 = male, 2 = female def audit(name, p, thr_f=0.5, thr_m=0.5): pred = np.where(female, p > thr_f, p > thr_m).astype(int) print(f'{name}: accuracy {np.mean(pred == ya):.4f}') for g, s_ in [(' women', female), (' men', ~female)]: r_ = rates(pred[s_], ya[s_]) print(f'{g:7s} base rate {r_["base"]:5.1%} predicted >$50K {r_["flagged"]:5.1%} TPR {1 - r_["FNR"]:5.1%} FPR {r_["FPR"]:5.1%} ' f'PPV {r_["PPV"]:5.1%} mean score {p[s_].mean():5.1%}') xa = fit_xgb(use); pa = xa.predict_proba(Ate[use])[:, 1] audit('XGBoost, all features (Lecture 8)', pa) no_sex = [c for c in use if c != 'SEX']; pb = fit_xgb(no_sex).predict_proba(Ate[no_sex])[:, 1] audit('XGBoost without the sex column ("unawareness")', pb) for d_ in (Afit, Aval): d_['is_f'] = (d_.SEX == 2) ps_ = fit_xgb(no_sex, 'is_f').predict_proba(Ate[no_sex])[:, 1] # how much do the other columns reveal about sex? print(f' sex predicted from the other nine columns: AUC {roc_auc_score(female, ps_):.3f}') tpr_m = np.mean(pa[~female & (ya == 1)] > 0.5) # equal opportunity: match women's TPR to men's thr_f = np.quantile(pa[female & (ya == 1)], 1 - tpr_m) audit(f'XGBoost with threshold {thr_f:.3f} for women (equal opportunity)', pa, thr_f=thr_f)
XGBoost, all features (Lecture 8): accuracy 0.8270
women base rate 35.0% predicted >$50K 34.0% TPR 75.1% FPR 11.9% PPV 77.3% mean score 35.0%
men base rate 46.4% predicted >$50K 47.1% TPR 81.2% FPR 17.4% PPV 80.2% mean score 46.5%
XGBoost without the sex column ("unawareness"): accuracy 0.8268
women base rate 35.0% predicted >$50K 35.7% TPR 77.3% FPR 13.3% PPV 75.8% mean score 36.7%
men base rate 46.4% predicted >$50K 45.4% TPR 79.6% FPR 15.8% PPV 81.4% mean score 45.0%
sex predicted from the other nine columns: AUC 0.832
XGBoost with threshold 0.415 for women (equal opportunity): accuracy 0.8261
women base rate 35.0% predicted >$50K 38.6% TPR 81.2% FPR 15.5% PPV 73.8% mean score 35.0%
men base rate 46.4% predicted >$50K 47.1% TPR 81.2% FPR 17.4% PPV 80.2% mean score 46.5%
The model (the same model as in Lecture 8, 82.7% accurate) predicts high income for 34.0% of women and 47.1% of men. The ratio of selection rates is 0.72, below the four-fifths rule's 0.8, so independence fails; but so do the base rates themselves (35.0% against 46.4%), because the data record a real income gap. Sufficiency nearly holds: the positive predictive values are close (77.3% and 80.2%), and the average score in each group matches its base rate (35.0% and 46.5%), as calibration requires (Theorem 13.4, step 1). Separation fails: the model finds 75.1% of high-earning women but 81.2% of high-earning men, and its false-positive rate is higher for men (17.4% against 11.9%). Theorem 13.3 says this pattern is forced by the different base rates once the predictive values are (nearly) equal.
The first remedy, "fairness through unawareness", deletes the sex column. It changes little (women's true-positive rate moves from 75.1% to 77.3%), because the other features carry much of the same information: a model given only the other nine columns tells women from men with AUC 0.832. What it does change is calibration within groups: without the sex column the model can no longer adjust for it, and women's average score (36.7%) now exceeds their base rate (35.0%) while men's (45.0%) falls below theirs (46.4%). Removing a protected attribute from the inputs rarely removes it from the model, and it can make the scores mean different things for different groups. The second remedy, post-processing (Hardt, Price and Srebro, 2016), keeps the model and chooses group-specific thresholds to equalise a criterion. Lowering the threshold for women from 0.5 to 0.415 equalises the true-positive rates at 81.2% (equal opportunity) at a cost of only 0.09 points of accuracy; but women's false-positive rate rises from 11.9% to 15.5% and their positive predictive value falls from 77.3% to 73.8%, so the scores now mean different things for the two groups. Equal opportunity was bought with sufficiency, as the theorems said it must be. And using a person's sex or race explicitly to set their threshold is itself unlawful in some settings: the US Civil Rights Act of 1991, for example, forbids adjusting employment test scores or cut-offs by race, sex or national origin. Fairness interventions are made inside a legal framework, not outside it.
A bank removes race from its credit model's inputs and finds that the approval rates by race barely change. What is the most likely reason?
Sentiment analysis is a large industry. Brand-monitoring services score millions of social-media posts a day for companies; trading firms and data vendors turn news and earnings-call sentiment into market signals, often with Loughran–McDonald-style dictionaries alongside fine-tuned language models; platforms use related classifiers to flag toxic content, where the same name and dialect biases matter even more (Sap et al., 2019, found that tweets in African-American English were up to twice as likely to be flagged as offensive). Fairness audits have moved from research into law. New York City's Local Law 144, enforced since July 2023, requires employers who use automated employment decision tools to commission an independent bias audit every year, publish the selection rates and impact ratios by sex and race or ethnicity (and their intersections), and notify candidates. The EU's AI Act (Regulation 2024/1689) classes systems used in employment, credit, education and access to essential services as high-risk and requires their providers to examine training data for possible biases; under the 2026 "Digital Omnibus" amendment (Regulation 2026/1744), those obligations apply from 2 December 2027.
The three criteria of Section 6 are statistical: they are properties of the joint distribution of \(R\), \(Y\) and \(A\). Hardt, Price and Srebro (2016) pointed out their limit: two scenarios can produce exactly the same joint distribution while differing in whether the decision is intuitively fair, because they differ in why \(A\) and \(Y\) are related. Whether a disparity is unfair depends on the causal path that produces it. The 1973 graduate admissions at Berkeley are the classic example (Bickel, Hammel and O'Connell, 1975): women were admitted at a lower rate overall, but within departments their rates were similar or higher, because women applied more often to the departments that admitted fewer applicants. The path from sex to admission through choice of department was widely judged acceptable; a direct path would not have been. Answering such questions needs the causal models of Lectures 3, 4 and 8.
Given a structural causal model with background variables \(U\), a predictor \(\hat Y\) is counterfactually fair if, for every individual (every \(x\) and \(a\)) and every \(y\) and \(a'\),
$$P\big(\hat Y_{A\leftarrow a}(U)=y\mid X=x,A=a\big)=P\big(\hat Y_{A\leftarrow a'}(U)=y\mid X=x,A=a\big):$$the prediction would have been the same had the individual's protected attribute been different, with everything not caused by it held fixed.
Computing it follows Pearl's three steps: infer the background variables from what we observe (abduction), set \(A\) to \(a'\) (action), and recompute the features and the prediction (prediction).
A predictor that uses only variables that are not descendants of \(A\) is automatically counterfactually fair.
Path-specific fairness relaxes the requirement to some causal paths from \(A\), treating others, like the choice of department, as legitimate (Kilbertus et al., 2017; Nabi and Shpitser, 2018).
# Counterfactual fairness in a simulated world where we know the causal model (after Kusner et al., 2017) from sklearn.linear_model import LinearRegression r = np.random.default_rng(0); N = 20000 A = r.integers(0, 2, N) # group membership K = r.normal(0, 1, N) # unobserved "knowledge", independent of A e1, e2, e3 = r.normal(0, 0.6, (3, N)) def world(A): # structural equations: the test and the grades are depressed for group 1 X1 = K - 1.0 * A + e1; X2 = K - 0.5 * A + e2; return np.c_[X1, X2] X = world(A); Yv = K + e3 # the outcome depends only on K Xcf = world(1 - A) # counterfactual: same people, same K and noise, group flipped res = lambda X_, A_: X_ - np.c_[[X[A == a].mean(0) for a in (0, 1)]][A_] # residuals of the features given the group models = {'aware (features and group)': (np.c_[X, A], np.c_[Xcf, 1 - A]), 'unaware (features only)': (X, Xcf), 'counterfactually fair (residuals)': (res(X, A), res(Xcf, 1 - A))} print(f'{"predictor":36s} {"MSE":>6s} {"mean prediction, group 0 / 1":>30s} {"mean |change| if group flipped":>31s}') for name, (Z, Zcf) in models.items(): m = LinearRegression().fit(Z, Yv); yh, ycf = m.predict(Z), m.predict(Zcf) print(f'{name:36s} {np.mean((yh - Yv) ** 2):6.3f} {yh[A == 0].mean():+14.3f} / {yh[A == 1].mean():+.3f} {np.mean(np.abs(ycf - yh)):31.3f}')
predictor MSE mean prediction, group 0 / 1 mean |change| if group flipped aware (features and group) 0.511 +0.006 / -0.005 0.000 unaware (features only) 0.594 +0.268 / -0.267 0.525 counterfactually fair (residuals) 0.511 +0.000 / +0.000 0.011
In the simulated world, the outcome depends only on an unobserved "knowledge" \(K\), which is the same in both groups; but group 1's test scores and grades are depressed (by 1.0 and 0.5) by barriers unrelated to knowledge. The unaware model, which uses the scores and grades but not the group, is the unfair one: it predicts 0.53 lower for group 1 than for group 0 on average, and flipping a person's group (holding their knowledge and luck fixed) would change their prediction by 0.53 on average. The aware linear model, given the group, learns to add back the penalty and is counterfactually fair, with lower error. The third predictor uses the residuals of the features after removing their group means, \(X-E[X\mid A]\), which in this additive model are functions of \(K\) and the noise alone: it is counterfactually fair by construction (the 0.011 is estimation error in the group means) and just as accurate. The example is idealised, since a real causal model is never known this precisely, but it makes the point that blindness to a protected attribute can produce exactly the unfairness it was meant to avoid.
Every number in this lecture's audits depends on the label \(Y\), and labels are measurements. COMPAS's "reoffended" is a new arrest, and arrests depend on where police patrol; if one group is policed more heavily, the same behaviour produces more arrests, the base rates of Theorem 13.3 are partly artefacts of measurement, and a perfectly calibrated score inherits the distortion. Obermeyer, Powers, Vogeli and Mullainathan (2019) found the same mechanism in a widely used US health-care algorithm: it predicted future medical costs as a proxy for medical need, and because less money was spent on Black patients with the same illnesses, Black patients had to be considerably sicker than white patients to receive the same score. Changing the label to a direct measure of health would have raised the share of Black patients flagged for extra care from 17.7% to 46.5%. No statistical criterion computed from \((R,Y,A)\) can detect a biased \(Y\); asking what the label measures, and along which paths the protected attribute affects features, outcomes and measurements, is causal reasoning.
Next: Part IV begins. Lecture 14 drops the labels altogether and asks what can be learned from the structure of unlabelled data.
Each class has 4 word tokens and \(V=6\), so \(\theta_{yj}=(c_{yj}+1)/10\). Positive: great 3/10, plot 1/10. Negative: great 2/10, plot 2/10. Priors are equal. Log-odds \(=\log\frac{0.3}{0.2}+\log\frac{0.1}{0.2}=0.405-0.693=-0.288\): slightly negative, because "plot" appeared only in a negative review.
Sufficiency gives \(P(Y=1\mid A=a)=\int P(Y=1\mid R=r)\,dP(r\mid A=a)\), with the integrand not depending on \(a\). Independence makes the distribution of \(R\) the same for every \(a\). Then \(P(Y=1\mid A=a)\) is the same for every \(a\), contradicting \(Y\not\perp A\). (In the language of conditional independence, \(R\perp A\) and \(Y\perp A\mid R\) together imply \(Y\perp A\).)
With common TPR and FPR, the selection rate in group \(a\) is \(\mathrm{TPR}\,p_a+\mathrm{FPR}\,(1-p_a)=\mathrm{FPR}+p_a(\mathrm{TPR}-\mathrm{FPR})\). Independence requires this to be the same for \(p_1\ne p_2\), so \(\mathrm{TPR}=\mathrm{FPR}\): the decision is positive with the same probability whatever the outcome.
Means 0.08 and 0.00; the four values have mean 0.04 and sample standard deviation \(\sqrt{(0.0036+0.0004+0.0036+0.0004)/3}=0.0516\), so \(d=0.08/0.0516=1.55\). There are \(\binom42=6\) splits; the statistic \(S=\sum_Xs-\sum_Ys\) is 0.16 for the observed split and smaller for every other (the next largest is 0.08), so \(p=1/6\). With two words per set no result can be significant at 5%: the smallest possible \(p\) is \(1/6\).
The PPV at threshold \(t\) is the average rearrest rate over deciles \(\ge t\), weighted by the number of defendants in each decile. Even if the rate in each decile were identical for the two groups, the weights differ (white defendants are concentrated in low deciles), so the averages differ: 65.0% against 59.5% at \(t=5\). Lowering the threshold adds a decile of defendants, among whom those not rearrested become new false positives, so the FPR can only rise.
\(E[X\mid A]=E[K]+\beta A+E[\varepsilon]\), so \(\tilde X=(K-E[K])+(\varepsilon-E[\varepsilon])\), which does not involve \(A\). If the effect \(\beta_i\) differs by person, \(X-E[X\mid A]=K+\varepsilon+(\beta_i-\bar\beta)A-\text{const}\) still depends on \(A\) through \((\beta_i-\bar\beta)A\), so flipping \(A\) changes it. Counterfactual fairness then needs the individual-level model, which observational data rarely identify.
Each project uses this lecture's methods and public data, and has a clear result.
Data. IMDb reviews: A. L. Maas et al. (2011), CSV copy from github.com/Ankit152/IMDB-sentiment-analysis (the Stanford archive was not reachable from the build environment). Financial PhraseBank v1.0: Malo et al. (2014), CC BY-NC-SA 3.0, copy from github.com/maxwellsarpong/NLP-financial-text-processing-dataset. Lexicons: Harvard IV-4 and Loughran–McDonald master dictionary from the pysentiment2 package (PyPI); VADER lexicon from the vaderSentiment package (MIT licence). GloVe 6B 100-dimensional vectors (Pennington et al., 2014; Public Domain Dedication and License), from the gensim-data release on GitHub. COMPAS: ProPublica, github.com/propublica/compas-analysis. American Community Survey 2018 (US Census Bureau, public domain) via Folktables, as in Lecture 8. Pretrained models from the Hugging Face Hub: DistilBERT fine-tuned on SST-2 (distilbert/distilbert-base-uncased-finetuned-sst-2-english, Apache 2.0) and FinBERT (ProsusAI/finbert, CC BY-NC-SA 4.0).
numbers in the text not printed by a code box are computed in verify_L13.py, widget data in figs_L13.py; code-box outputs are pasted from actual runs (scikit-learn, XGBoost and JAX on two CPU cores)