Machine Learning · Lecture 13

Lecture 13
Sentiment and Bias Analysis

Part III · Deep learning  |  sentiment as classification, naive Bayes and logistic regression on text, lexicons and domain shift, pre-trained representations, bias in word embeddings and the WEAT, auditing models with counterfactual inputs, fairness criteria, the impossibility theorems, auditing COMPAS and the Lecture 8 income model, counterfactual fairness  |  reading: Jurafsky & Martin, Speech and Language Processing (3rd ed. draft), ch. 4–6; Barocas, Hardt & Narayanan, Fairness and Machine Learning (2023), ch. 3–5; Chouldechova (2017); Caliskan, Bryson & Narayanan (2017)

Cartoon: accurate for whom?One stick figure proudly points at a board reading "accuracy 66%". A second figure points at two bars of different heights labelled "false alarms, group A" and "false alarms, group B" and asks: accurate for whom? accuracy66% group Agroup B Our risk score is righttwo times in three! Then why are its falsealarms so unequal?
One accuracy number can hide very different errors for different groups, and which errors should be equal is a choice
In this lecture
  1. Sentiment as classification
  2. Lexicons, learned models and domain shift
  3. Pre-trained representations
  4. Bias in word embeddings
  5. From representations to decisions
  6. Fairness criteria
  7. The impossibility theorems
  8. Auditing the Lecture 8 income model
  9. Causal notions of fairness
  10. Summary, exercises, projects, reading

The last three lectures built models that read text and images. This lecture closes the deep-learning part of the course by asking two questions about such models. First, a practical one: how do we get a machine to tell whether a piece of text is positive or negative, and how well do the different ways of doing it carry over from one kind of text to another? Second, a harder one: models learn whatever regularities their training data contain, including stereotypes and historical inequities, and are then used to make decisions about people. How do we measure what they have learned wrongly, what would it mean for a decision rule to be fair, and why can't a rule be fair in every sense at once? The second half of the lecture audits two real systems: the COMPAS risk score at the centre of the 2016 "Machine Bias" debate, and the income model we built in Lecture 8.

By the end of this lecture you should be able to build and compare naive Bayes and logistic regression sentiment classifiers and prove that naive Bayes is linear in log space; explain why lexicons and models trained on one domain fail on another; measure bias in word embeddings with analogies, projections and the WEAT, including its permutation test; define demographic parity, equalised odds and calibration; prove Chouldechova's identity and state the Kleinberg–Mullainathan–Raghavan impossibility theorem; audit a classifier's error rates by group and adjust it by post-processing; and explain counterfactual fairness and why fairness is ultimately a causal question.

1 — Sentiment as classification

"A lot of fun, if you can forgive the plot." "Not bad at all." "The worst film I have seen this year." People infer the writer's attitude from such sentences without effort; a machine needs a representation. The simplest is the bag of words: forget the order, and represent a document by the counts \(x_j\) of each word \(j\) of a vocabulary of size \(V\). It throws away a great deal ("not bad" and "bad, not" look the same), and yet it is a strong baseline. A refinement, TF-IDF, weights each count by the word's inverse document frequency \(\log(N/n_j)\), where \(n_j\) of the \(N\) documents contain word \(j\), so that words found in every document ("the", "film") count for little. Adding pairs of adjacent words ("bigrams") restores some order: "not bad" becomes a feature of its own.

data — the IMDb Large Movie Review dataset

50,000 movie reviews from the Internet Movie Database, half positive (rated 7–10 out of 10) and half negative (1–4), at most 30 reviews per film, collected by Maas et al. (2011) at Stanford. The original split has 25,000 reviews for training and 25,000 for testing; we use the CSV copy distributed on GitHub, which merges them, and split it 25,000/25,000 at random. The median review is 171 words long.

Two classifiers are natural for counts. The first is naive Bayes, a generative model: it models how each class produces words, assuming (naively) that the words of a document are drawn independently given its class, with class \(y\) using word \(j\) with probability \(\theta_{yj}\). The second is logistic regression (Lecture 4), which models \(P(y\mid x)\) directly. They turn out to be the same kind of classifier.

theorem 13.1 — naive Bayes is a linear classifier in log space

Under multinomial naive Bayes with class priors \(\pi_0,\pi_1\) and word probabilities \(\theta_{0j},\theta_{1j}\), the posterior log-odds of a document with word counts \(x\) is linear in \(x\):

$$\log\frac{P(y=1\mid x)}{P(y=0\mid x)}=\sum_{j=1}^V x_j\log\frac{\theta_{1j}}{\theta_{0j}}+\log\frac{\pi_1}{\pi_0}=w^\top x+b.$$

So naive Bayes and logistic regression choose from the same family of decision rules (hyperplanes in count space); they differ only in how they pick \(w\) and \(b\): naive Bayes from per-class word frequencies, logistic regression by maximising the conditional likelihood.

Under the model, a document of length \(n=\sum_jx_j\) has

$$P(x\mid y)=\frac{n!}{\prod_jx_j!}\prod_j\theta_{yj}^{x_j}.$$

By Bayes' rule, \(P(y\mid x)\propto\pi_yP(x\mid y)\), and in the ratio of the two classes the multinomial coefficient and the normalising constant \(P(x)\) cancel:

$$\frac{P(y=1\mid x)}{P(y=0\mid x)}=\frac{\pi_1\prod_j\theta_{1j}^{x_j}}{\pi_0\prod_j\theta_{0j}^{x_j}}.$$

Taking logarithms gives the formula, with \(w_j=\log(\theta_{1j}/\theta_{0j})\) and \(b=\log(\pi_1/\pi_0)\).

The same argument with binary features and Bernoulli naive Bayes gives a linear function of the presence indicators. \(\square\)

Bayes — smoothing is a prior, and the base rate is a prior too

The maximum-likelihood estimate \(\hat\theta_{yj}=c_{yj}/\sum_kc_{yk}\) (where \(c_{yj}\) counts word \(j\) in class \(y\)) is zero for any word never seen in class \(y\), and a single such word would then make the posterior of class \(y\) exactly zero. The standard fix, Laplace (add-\(\alpha\)) smoothing, \(\hat\theta_{yj}=(c_{yj}+\alpha)/(\sum_kc_{yk}+\alpha V)\), is the posterior mean of \(\theta_y\) under a symmetric Dirichlet(\(\alpha\)) prior (the beta–binomial calculation, in \(V\) dimensions). And the term \(\log(\pi_1/\pi_0)\) is the prior in the most literal sense: the same words give a different verdict when positive reviews are rare. Base rates will return in Section 7 as the source of the impossibility theorems.

code — naive Bayes and logistic regression on 50,000 reviews
# Sentiment as classification: naive Bayes and logistic regression on 50,000 IMDb reviews
import numpy as np, pandas as pd, re, time
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
imdb = pd.read_csv('data/IMDB-Dataset.csv')
imdb['text'] = imdb.review.str.replace('<br />', ' ', regex=False)
y_all = (imdb.sentiment == 'positive').astype(int).values
perm = np.random.default_rng(0).permutation(len(imdb)); tr, te = perm[:25000], perm[25000:]   # 25,000 / 25,000
print(f'{len(imdb):,} reviews, {y_all.mean():.0%} positive; median length {imdb.text.str.split().str.len().median():.0f} words')
cv = CountVectorizer(min_df=5, token_pattern=r"(?u)\b\w[\w']*\b")
Xc_tr, Xc_te = cv.fit_transform(imdb.text.values[tr]), cv.transform(imdb.text.values[te]); ytr, yte = y_all[tr], y_all[te]
nb = MultinomialNB(alpha=1.0).fit(Xc_tr, ytr)
print(f'vocabulary {len(cv.vocabulary_):,} words')
print(f'naive Bayes, word counts:                   test accuracy {nb.score(Xc_te, yte):.4f}')
# Theorem 13.1 in action: naive Bayes' log-odds is a linear function w.x + b of the counts
w_nb = nb.feature_log_prob_[1] - nb.feature_log_prob_[0]; b_nb = nb.class_log_prior_[1] - nb.class_log_prior_[0]
lp = nb.predict_log_proba(Xc_te[:5]); print('  log-odds from predict_log_proba:', np.round(lp[:, 1] - lp[:, 0], 3))
print('  log-odds as w.x + b:            ', np.round(Xc_te[:5] @ w_nb + b_nb, 3))
tf = TfidfVectorizer(min_df=5, ngram_range=(1, 2), sublinear_tf=True, token_pattern=r"(?u)\b\w[\w']*\b")
Xt_tr, Xt_te = tf.fit_transform(imdb.text.values[tr]), tf.transform(imdb.text.values[te])
lr = LogisticRegression(C=10, max_iter=3000).fit(Xt_tr, ytr)
print(f'logistic regression, TF-IDF words + pairs: test accuracy {lr.score(Xt_te, yte):.4f}  ({Xt_tr.shape[1]:,} features)')
voc = np.array(cv.get_feature_names_out()); o = np.argsort(w_nb); ok = np.asarray(Xc_tr.sum(0)).ravel() >= 50
print('most negative words (NB, seen 50+ times):', ', '.join(voc[[i for i in o if ok[i]][:10]]))
print('most positive words (NB, seen 50+ times):', ', '.join(voc[[i for i in o[::-1] if ok[i]][:10]]))
fv = np.array(tf.get_feature_names_out()); ol = np.argsort(lr.coef_[0])
print('most negative features (logistic):', ', '.join(fv[ol[:10]]))
print('most positive features (logistic):', ', '.join(fv[ol[::-1][:10]]))
50,000 reviews, 50% positive; median length 171 words
vocabulary 27,945 words
naive Bayes, word counts:                   test accuracy 0.8385
  log-odds from predict_log_proba: [ -4.325 -15.616   0.655  20.923 -10.759]
  log-odds as w.x + b:             [ -4.325 -15.616   0.655  20.923 -10.759]
logistic regression, TF-IDF words + pairs: test accuracy 0.9152  (154,851 features)
most negative words (NB, seen 50+ times): uwe, boll, stinker, tripe, gamera, unfunny, waste, seagal, mst3k, flimsy
most positive words (NB, seen 50+ times): ponyo, devito, custer, macarthur, novak, paulie, georges, kolchak, felix, carla
most negative features (logistic): worst, bad, awful, boring, the worst, terrible, waste, worse, nothing, poor
most positive features (logistic): great, excellent, perfect, wonderful, amazing, 7, the best, hilarious, brilliant, 7 10

Logistic regression on TF-IDF words and word pairs is right on 91.5% of the 25,000 test reviews; naive Bayes on raw counts, on 83.9%. The code checks Theorem 13.1 numerically: the log-odds from the library and from \(w^\top x+b\) agree to every printed digit. The two lists of top words show why naive Bayes loses. Its most decisive words are proper names: "Uwe" and "Boll" (a director of notoriously bad films), "Seagal", "MST3K" on the negative side, and "Ponyo", "DeVito", "Kolchak" on the positive. A name that appears in a few dozen reviews, nearly all of one class, gets a large weight \(\log(\theta_{1j}/\theta_{0j})\), because naive Bayes estimates each word's weight separately, as if the words were independent. Logistic regression fits all the weights jointly, so a word gets a large weight only if it adds information beyond the others, and its top features are the general words of praise and blame: "worst", "awful", "waste", "great", "excellent", "perfect", and the pairs "the worst" and "7 10" (reviewers rating a film 7/10). This is the classic contrast between generative and discriminative learning: naive Bayes approaches its (higher) asymptotic error with fewer examples, logistic regression reaches a lower error with enough data (Ng and Jordan, 2002).

A live sentiment scorer

A logistic regression on the presence or absence of the 5,000 most common words, trained on the 25,000 training reviews. Each word's colour shows its weight (sky pushes towards positive, rose towards negative); the score is the sum of the weights of the distinct words present, plus an intercept. Type your own review, or load one from the test set. The default text shows a weakness of the bag of words: "not" counts against the review and "bad" counts against it again.

spot check 13.1

A naive Bayes classifier has \(\log(\pi_1/\pi_0)=0\). A short review contains "great" once (\(w=+1.2\)), "boring" once (\(w=-1.5\)) and words whose weights sum to 0.6. What is its posterior log-odds of being positive?

Theorem 13.1: add the weights, each multiplied by its count, and the intercept.
\(1.2-1.5+0.6=0.3\), so \(P(\text{positive})=1/(1+e^{-0.3})=0.57\). Every word votes independently with a fixed weight; neither naive Bayes nor a bag-of-words logistic regression can learn that "boring" means something different after "never".
history — thumbs up?

Bo Pang, Lillian Lee and Shivakumar Vaithyanathan's "Thumbs up? Sentiment classification using machine learning techniques" (2002) took 1,400 movie reviews from the Internet Movie Database archive of the rec.arts.movies.reviews newsgroup, labelled them by the reviewers' star ratings, and showed that standard text classifiers (naive Bayes, maximum entropy, SVMs) beat lists of sentiment words chosen by people, while doing much worse on sentiment than on topic classification. The paper started the field of sentiment analysis in machine learning. Lexicons had a longer history: the General Inquirer, built at Harvard by Philip Stone and colleagues in the 1960s, tagged thousands of English words with categories such as "Positiv" and "Negativ", and is still in use. Maas et al.'s IMDb dataset (2011) and the Stanford Sentiment Treebank (Socher et al., 2013), with sentiment labels for every phrase of 11,855 sentences, became the benchmarks of the deep-learning era.

2 — Lexicons, learned models and domain shift

A sentiment lexicon is a list of words marked positive or negative, perhaps with strengths. Scoring a text by counting its positive and negative words needs no training data and is transparent, which is why lexicons remain popular in the social sciences and finance. Their weakness is that the meaning of a word depends on the domain. Loughran and McDonald (2011) found that almost three-quarters of the words that the Harvard General Inquirer classes as negative are not negative in companies' annual reports: "tax", "cost", "capital", "board", "liability", "vice" (as in vice-president). They built a lexicon for financial text, now standard in finance research. The same problem afflicts learned models: a classifier trained on movie reviews has learned the vocabulary of movie reviews. When the distribution of inputs at test time differs from that in training, we have domain shift.

data — the Financial PhraseBank

4,846 sentences from English-language financial news about companies listed on the Helsinki stock exchange, each labelled positive, negative or neutral "from the viewpoint of an investor" by 16 annotators with backgrounds in finance and business (Malo et al., 2014; CC BY-NC-SA 3.0). We use the 2,264 sentences on which all annotators agreed: 1,391 neutral, 570 positive and 303 negative. Lexicons: the Harvard General Inquirer (Harvard IV-4), VADER (Hutto and Gilbert, 2014; built for social media) and Loughran–McDonald's master dictionary, as distributed in the pysentiment2 and vaderSentiment Python packages.

code — four ways to score financial news, and a movie-review model out of its depth
# Lexicons against learned models, and domain shift: the Financial PhraseBank
from sklearn.model_selection import cross_val_predict
import collections
fpb = pd.read_csv('data/FPB_Sentences_AllAgree.txt', sep='@', header=None, names=['text', 'label'], encoding='latin-1')
lab = fpb.label.map({'negative': -1, 'neutral': 0, 'positive': 1}).values
print(f'{len(fpb):,} sentences on which all annotators agreed:', dict(collections.Counter(fpb.label)))
words = lambda s: re.findall(r"[a-z][a-z'-]*", s.lower())
hiv = pd.read_csv('data/HIV-4.csv', low_memory=False); hiv['w'] = hiv.Entry.str.split('#').str[0].str.lower()
HP, HN = set(hiv.w[hiv.Positiv.notna()]), set(hiv.w[hiv.Negativ.notna()])                # Harvard IV (general)
lm = pd.read_csv('data/LM.csv'); LP, LN = set(lm.Word[lm.Positive > 0].str.lower()), set(lm.Word[lm.Negative > 0].str.lower())
vd = pd.read_csv('data/vader_lexicon.txt', sep='\t', header=None, usecols=[0, 1], names=['w', 'v'], quoting=3)
VAL = dict(zip(vd.w, vd.v))                                                               # VADER (social media)
def lex_predict(pos, neg):
    return np.array([np.sign(sum(w in pos for w in words(s)) - sum(w in neg for w in words(s))) for s in fpb.text])
preds = {'Harvard IV (general, 1960s)': lex_predict(HP, HN), 'VADER (social media, 2014)': np.array([np.sign(sum(VAL.get(w, 0) for w in words(s))) for s in fpb.text]),
         'Loughran-McDonald (finance)': lex_predict(LP, LN)}
pn = lab != 0
p_imdb = np.where(lr.predict_proba(tf.transform(fpb.text))[:, 1] > 0.5, 1, -1)             # the IMDb model, no retraining
tf_f = TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True, min_df=2)
preds['logistic trained on FPB (5-fold CV)'] = cross_val_predict(LogisticRegression(C=10, max_iter=3000, class_weight='balanced'), tf_f.fit_transform(fpb.text), lab, cv=5)
print(f'{pn.sum()} sentences are positive or negative; for those, the share given the right sign, called neutral, or the wrong sign')
print(f'{"method":38s} {"3-class":>8s} {"right":>7s} {"neutral":>8s} {"wrong":>6s}')
print(f'{"always neutral":38s} {np.mean(lab == 0):8.1%}')
for name, p in list(preds.items()) + [('logistic trained on IMDb reviews', None)]:
    acc = '       -' if p is None else f'{np.mean(p == lab):8.1%}'
    q = p_imdb[pn] if p is None else p[pn]; l = lab[pn]
    print(f'{name:38s} {acc} {np.mean(q == l):7.0%} {np.mean(q == 0):8.0%} {np.mean((q != l) & (q != 0)):6.0%}')
cnt = collections.Counter(w for s, l in zip(fpb.text, lab) if l >= 0 for w in set(words(s)) if w in HN and w not in LN)
print('Harvard-negative words (not negative for Loughran-McDonald) most common in neutral or positive sentences:')
print('  ' + ', '.join(f'{w} ({c})' for w, c in cnt.most_common(10)))
2,264 sentences on which all annotators agreed: {'neutral': 1391, 'positive': 570, 'negative': 303}
873 sentences are positive or negative; for those, the share given the right sign, called neutral, or the wrong sign
method                                  3-class   right  neutral  wrong
always neutral                            61.4%
Harvard IV (general, 1960s)               45.8%     44%      36%    20%
VADER (social media, 2014)                56.4%     56%      21%    23%
Loughran-McDonald (finance)               65.0%     25%      66%    10%
logistic trained on FPB (5-fold CV)       84.0%     64%      21%    15%
logistic trained on IMDb reviews              -     53%       0%    47%
Harvard-negative words (not negative for Loughran-McDonald) most common in neutral or positive sentences:
  order (52), capital (52), board (43), service (36), deal (33), division (23), press (13), cost (12), vice (10), make (9)

On financial news, every general-purpose method struggles. Always answering "neutral" is right 61.4% of the time; the Harvard lexicon gets only 45.8% of the three-way labels right, because it finds sentiment where there is none: in neutral or good news it marks "order" (as in order intake), "capital", "board", "service", "deal" and "division" as negative. VADER, built for tweets, does a little better. The Loughran–McDonald lexicon makes the fewest wrong-sign errors (10% of the positive and negative sentences), but its list of positive words is short (354 words against 2,355 negative), so it calls two-thirds of the clearly positive or negative sentences neutral: "operating profit rose to EUR 23.5 mn" contains no lexicon word at all. Financial sentiment is carried by verbs of change attached to financial quantities, which a word list handles badly. A logistic regression trained on the financial sentences themselves (in five-fold cross-validation, so always predicting sentences it has not seen) reaches 84.0%. And the IMDb model, 91.5% accurate on reviews, gives the right sign to only 53% of the financial sentences: its weights belong to another domain.

spot check 13.2

The IMDb model gives the right sign to only 53% of the positive and negative financial sentences, barely better than a coin. Which explanation fits best?

Which words decide whether company news is good or bad?
This is domain shift: \(P(x)\) changes, and with it the relationship between words and labels. A model trained on 2,000 financial sentences beats one trained on 25,000 reviews, and a lexicon built for the domain beats general ones at avoiding wrong signs.

3 — Pre-trained representations

Labelled data is expensive; unlabelled text is plentiful. The transfer-learning recipe of Lecture 12 is to learn a representation from unlabelled text and then fit a small model on top with few labels. The oldest version uses static word vectors: represent a review by the average of the GloVe vectors (Pennington, Socher and Manning, 2014; trained on six billion words of Wikipedia and newswire) of its words, and fit a logistic regression to the 100 averaged dimensions. A slightly richer version learns to weight the words: an attention-pooling classifier (Lecture 12, Section 3) computes a score for each word from its vector, turns the scores into weights with a softmax, and classifies the weighted average. The modern version fine-tunes a whole pre-trained transformer such as BERT, which we cannot download here; we report published numbers for it below.

code — GloVe features from 100 labels to 25,000, and attention pooling
# Transfer: pre-trained GloVe word vectors, from 100 labelled reviews to 25,000; and an attention-pooling classifier
import gzip, jax, jax.numpy as jnp
gv_words, gv_vecs = [], []
with gzip.open('data/glove100.gz', 'rt', encoding='utf8') as f:
    next(f)                                                    # header line: 400000 100
    for i, line in enumerate(f):
        if i == 100000: break                                  # the 100,000 most frequent words
        p = line.rstrip().split(' '); gv_words.append(p[0]); gv_vecs.append(np.array(p[1:], np.float32))
G = np.stack(gv_vecs); gidx = {w: i for i, w in enumerate(gv_words)}
print(f'GloVe: {G.shape[0]:,} words x {G.shape[1]} dimensions (trained on Wikipedia + Gigaword, 6 billion tokens)')
tokens = lambda s: re.findall(r"[a-z]+(?:'[a-z]+)?|[!?]", s.lower())
def glove_avg(texts):
    out = np.zeros((len(texts), 100), np.float32)
    for j, s in enumerate(texts):
        ix = [gidx[w] for w in tokens(s) if w in gidx]
        if ix: out[j] = G[ix].mean(0)
    return out
A_tr, A_te = glove_avg(imdb.text.values[tr]), glove_avg(imdb.text.values[te]); mu, sd = A_tr.mean(0), A_tr.std(0)
print(f'{"labelled reviews":>16s} {"TF-IDF logistic":>16s} {"GloVe average + logistic":>25s}')
for n in [100, 1000, 10000, 25000]:
    s_ = np.arange(n)
    a1 = LogisticRegression(C=10, max_iter=3000).fit(Xt_tr[s_], ytr[s_]).score(Xt_te, yte)
    a2 = LogisticRegression(C=1, max_iter=3000).fit((A_tr[s_] - mu) / sd, ytr[s_]).score((A_te - mu) / sd, yte)
    print(f'{n:16,d} {a1:16.4f} {a2:25.4f}')
glove_lr = LogisticRegression(C=1, max_iter=3000).fit((A_tr - mu) / sd, ytr)       # kept for Section 4
# Attention pooling over frozen GloVe vectors: weights a_i = softmax(v . tanh(W e_i)), document = sum_i a_i tanh(U e_i)
L_max = 300
def ids_of(texts):
    M = np.zeros((len(texts), L_max), np.int32); mask = np.zeros((len(texts), L_max), bool)
    for j, s in enumerate(texts):
        ix = [gidx[w] for w in tokens(s) if w in gidx][:L_max]; M[j, :len(ix)] = ix; mask[j, :len(ix)] = True
    return M, mask
I_tr, M_tr = ids_of(imdb.text.values[tr]); I_te, M_te = ids_of(imdb.text.values[te]); Gj = jnp.array(G)
k1, k2, k3 = jax.random.split(jax.random.PRNGKey(0), 3)
ap = {'W': jax.random.normal(k1, (100, 64)) * 0.1, 'v': jax.random.normal(k2, (64,)) * 0.1,
      'U': jax.random.normal(k3, (100, 128)) * 0.1, 'u': jnp.zeros(128), 'c': 0.0}
def attend(p, I, M):
    E = Gj[I]                                                    # (batch, L, 100), frozen
    s = jnp.where(M, jnp.tanh(E @ p['W']) @ p['v'], -1e9); a = jax.nn.softmax(s, axis=1)
    return a, jnp.einsum('bl,bld->bd', a, jnp.tanh(E @ p['U'])) @ p['u'] + p['c']
def ap_loss(p, I, M, yb):
    z = attend(p, I, M)[1]; return jnp.mean(jnp.logaddexp(0, z) - yb * z)
@jax.jit
def ap_step(p, m, v, t, I, M, yb):
    g = jax.grad(ap_loss)(p, I, M, yb)
    m = jax.tree_util.tree_map(lambda a, b: 0.9 * a + 0.1 * b, m, g); v = jax.tree_util.tree_map(lambda a, b: 0.999 * a + 0.001 * b * b, v, g)
    return jax.tree_util.tree_map(lambda q, a, b: q - 2e-3 * (a / (1 - 0.9 ** t)) / (jnp.sqrt(b / (1 - 0.999 ** t)) + 1e-8), p, m, v), m, v
m_, v_ = jax.tree_util.tree_map(jnp.zeros_like, ap), jax.tree_util.tree_map(jnp.zeros_like, ap); t = 0; rng = np.random.default_rng(0)
ap_pred = jax.jit(lambda p, I, M: attend(p, I, M)[1] > 0)
for ep in range(4):
    for b_ in np.array_split(rng.permutation(25000), 25000 // 64):
        t += 1; ap, m_, v_ = ap_step(ap, m_, v_, t, I_tr[b_], M_tr[b_], ytr[b_].astype(np.float32))
    acc = np.mean(np.concatenate([np.asarray(ap_pred(ap, I_te[i:i + 2500], M_te[i:i + 2500])) for i in range(0, 25000, 2500)]) == yte)
    print(f'attention pooling, epoch {ep + 1}: test accuracy {acc:.4f}')
a_ex = np.asarray(attend(ap, I_te[:1], M_te[:1])[0][0]); n_ex = M_te[0].sum()
top = np.argsort(-a_ex[:n_ex])[:8]
print('first test review:', imdb.text.values[te[0]][:150].strip() + '...')
print('  label', 'positive' if yte[0] else 'negative', '| most attended words:', ', '.join(f'{gv_words[I_te[0, i]]} ({a_ex[i]:.2f})' for i in top))
GloVe: 100,000 words x 100 dimensions (trained on Wikipedia + Gigaword, 6 billion tokens)
labelled reviews  TF-IDF logistic  GloVe average + logistic
             100           0.7350                    0.6894
           1,000           0.8387                    0.7726
          10,000           0.8983                    0.7951
          25,000           0.9152                    0.8012
attention pooling, epoch 1: test accuracy 0.8396
attention pooling, epoch 2: test accuracy 0.8457
attention pooling, epoch 3: test accuracy 0.8502
attention pooling, epoch 4: test accuracy 0.8502
first test review: These slasher pics are past their sell by date, but this one is good fun.  The valentine cards themselves are witty, and well thought out.  The film h...
  label positive | most attended words: sizzling (0.06), sorry (0.05), nothing (0.02), feeling (0.02), fun (0.02), enough (0.02), moments (0.02), scary (0.02)

The pre-trained vectors do not help here. With 100 labelled reviews, TF-IDF logistic regression is right 73.5% of the time and the GloVe average 68.9%; with all 25,000 the gap widens to 91.5% against 80.1%. (Tuning the regularisation does not change the order.) Averaging throws away which words occur, and a review's sentiment rests on a few decisive words ("waste", "excellent") whose information is diluted in the average of 200 vectors. The attention-pooling model, which learns how much each word should count, recovers part of the loss: 85.0% after four epochs. But its weights are spread thinly; on the first test review the most attended word gets only 6% of the weight, and the top words ("sizzling", "sorry", "nothing", "fun") are a mixture of sentiment words and words a reader would not single out. Attention weights are not explanations: Jain and Wallace (2019) showed that very different attention distributions can produce the same predictions, so reading a model's reasons off its attention needs care.

The real gains from pre-training come from contextual representations: a transformer (Lecture 12) gives each word a vector that depends on the whole sentence, so "not good" and "good" no longer share their evidence. Fine-tuned end to end on the IMDb training set, BERT-base reaches about 94% test accuracy (Sun et al., 2019, report 94.6% with careful fine-tuning), and DistilBERT, a model 40% smaller, 92.8% (Sanh et al., 2019). Fine-tuning is beyond our CPU budget, but we can measure a public DistilBERT that was fine-tuned on a different data set, the Stanford Sentiment Treebank's single sentences from film reviews (SST-2; Socher et al., 2013), and has never seen an IMDb review. The same box scores the Financial PhraseBank of Section 2 with FinBERT (Araci, 2019), a BERT fine-tuned for financial sentiment.

code — two pretrained transformers, measured: DistilBERT on IMDb and on financial news, FinBERT on financial news
# Pre-trained transformers, measured: DistilBERT (fine-tuned on SST-2 movie-review sentences) on IMDb, and FinBERT on the PhraseBank
import torch, time
from transformers import AutoTokenizer, AutoModelForSequenceClassification
def classify(name, texts, max_len=512, head_tail=False, bs=16):
    tk = AutoTokenizer.from_pretrained(name); m = AutoModelForSequenceClassification.from_pretrained(name).eval(); P = []
    order = np.argsort([len(t) for t in texts])                  # batch similar lengths together: much less padding
    for i in range(0, len(texts), bs):
        ids = [tk(texts[j], add_special_tokens=False).input_ids for j in order[i:i + bs]]
        if head_tail: ids = [x if len(x) <= max_len - 2 else x[:128] + x[-(max_len - 130):] for x in ids]   # first 128 + last 382 tokens
        else: ids = [x[:max_len - 2] for x in ids]
        ids = [[tk.cls_token_id] + x + [tk.sep_token_id] for x in ids]; T = max(map(len, ids))
        x = torch.tensor([s + [tk.pad_token_id] * (T - len(s)) for s in ids]); att = (x != tk.pad_token_id).long()
        with torch.no_grad(): P.append(torch.softmax(m(input_ids=x, attention_mask=att).logits, -1).numpy())
    P = np.concatenate(P); out = np.empty_like(P); out[order] = P; return out, m.config.id2label
sub = te[:1000]; texts = list(imdb.text.values[sub]); ys = y_all[sub]; t0 = time.time()
P1, lab1 = classify('distilbert/distilbert-base-uncased-finetuned-sst-2-english', texts)
P2, _ = classify('distilbert/distilbert-base-uncased-finetuned-sst-2-english', texts, head_tail=True)
ntok = AutoTokenizer.from_pretrained('distilbert/distilbert-base-uncased-finetuned-sst-2-english')
long_ = np.array([len(ntok(t).input_ids) > 512 for t in texts])
print(f"IMDb, the first 1,000 test reviews ({long_.mean():.0%} longer than 512 tokens); labels {lab1}; {time.time() - t0:.0f} s")
print(f"  logistic regression, TF-IDF, trained on 25,000 IMDb reviews:   {lr.score(Xt_te[:1000], yte[:1000]):.1%}")
print(f"  DistilBERT-SST-2, never trained on IMDb, first 512 tokens:      {np.mean(P1.argmax(1) == ys):.1%}")
print(f"  the same, first 128 + last 382 tokens (Sun et al., 2019):       {np.mean(P2.argmax(1) == ys):.1%}"
      f"   (on the long reviews alone: {np.mean(P1.argmax(1)[long_] == ys[long_]):.1%} -> {np.mean(P2.argmax(1)[long_] == ys[long_]):.1%})")
fpb = pd.read_csv('data/FPB_Sentences_AllAgree.txt', sep='@', header=None, names=['text', 'label'], encoding='latin-1')
labf = fpb.label.map({'negative': -1, 'neutral': 0, 'positive': 1}).values; pol = labf != 0
P3, lab3 = classify('ProsusAI/finbert', list(fpb.text))
pf = np.array([{'positive': 1, 'negative': -1, 'neutral': 0}[lab3[k]] for k in P3.argmax(1)])
P4, _ = classify('distilbert/distilbert-base-uncased-finetuned-sst-2-english', list(fpb.text))
pd_ = np.where(P4.argmax(1) == 1, 1, -1)                         # SST-2 has no neutral class
row = lambda p: f"{np.mean(p[pol] == labf[pol]):5.0%}   {np.mean(p[pol] == 0):6.0%}   {np.mean(p[pol] == -labf[pol]):5.0%}"
print("Financial PhraseBank, 2,264 all-agree sentences   3-class   right  neutral  wrong  (on the 873 polar sentences)")
print(f"  DistilBERT-SST-2 (general sentiment, 2 classes)      -     {row(pd_)}")
print(f"  FinBERT (fine-tuned on the PhraseBank itself)    {np.mean(pf == labf):5.1%}   {row(pf)}")
IMDb, the first 1,000 test reviews (15% longer than 512 tokens); labels {0: 'NEGATIVE', 1: 'POSITIVE'}; 267 s
  logistic regression, TF-IDF, trained on 25,000 IMDb reviews:   91.7%
  DistilBERT-SST-2, never trained on IMDb, first 512 tokens:      88.7%
  the same, first 128 + last 382 tokens (Sun et al., 2019):       89.6%   (on the long reviews alone: 80.8% -> 87.0%)
Financial PhraseBank, 2,264 all-agree sentences   3-class   right  neutral  wrong  (on the 873 polar sentences)
  DistilBERT-SST-2 (general sentiment, 2 classes)      -       67%       0%     33%
  FinBERT (fine-tuned on the PhraseBank itself)    97.2%     98%       0%      2%

Without a single IMDb label, DistilBERT is right on 88.7% of the first 1,000 test reviews, close to the 91.7% of logistic regression trained on 25,000 of them. BERT-style models read at most 512 tokens, and 15% of these reviews are longer; Sun et al. found that keeping the first 128 and the last 382 tokens works better than keeping the first 512, because reviews tend to end with the verdict, and so it does here: 80.8% to 87.0% on the long reviews, 89.6% overall. On financial news the same model fails like the movie-review logistic regression of Section 2, if less badly: it has no neutral class, and it gives the wrong sign to 33% of the sentences that do have a sentiment (the movie-review model: 47%). FinBERT scores 97.2% on the three classes, far above every other method in this lecture. The number is not a fair test: FinBERT was fine-tuned on the Financial PhraseBank itself (Araci held out a random 20% for testing), so most of these sentences were in its training data. A model evaluated on its own training data looks excellent, and Lecture 23 returns to this kind of leakage. On fresh financial text its accuracy would be lower, by an amount this data set cannot tell us.

4 — Bias in word embeddings

Word vectors learn meaning from co-occurrence (Lecture 12, Theorem 12.1), so they learn whatever associations the training text contains: that Paris is to France as Tokyo is to Japan, and also that some occupations are mentioned mostly alongside women and others alongside men. Bolukbasi et al. (2016) put it in their title: "Man is to computer programmer as woman is to homemaker?" Such associations matter because embeddings are inputs to systems that rank CVs, route customer complaints and flag content. Three tools measure them: analogies, projections onto a direction, and association tests.

code — analogies and a gender direction in GloVe
# Bias in representations: analogies and a gender direction in GloVe
Gn = G / np.linalg.norm(G, axis=1, keepdims=True)
def analogy(a, b, c, exclude=True, k=3):                       # a : b :: c : ?   by the vector b - a + c
    q = Gn[gidx[b]] - Gn[gidx[a]] + Gn[gidx[c]]; s = Gn @ (q / np.linalg.norm(q))
    if exclude: s[[gidx[a], gidx[b], gidx[c]]] = -np.inf
    return [gv_words[i] for i in np.argsort(-s)[:k]]
for a, b, c in [('man', 'king', 'woman'), ('paris', 'france', 'tokyo'), ('man', 'doctor', 'woman'), ('man', 'programmer', 'woman'),
                ('he', 'surgeon', 'she'), ('man', 'engineer', 'woman')]:
    print(f'{a}:{b} :: {c}:?   excluding inputs: {", ".join(analogy(a, b, c))}   allowing them: {", ".join(analogy(a, b, c, False))}')
pairs = [('she', 'he'), ('her', 'his'), ('woman', 'man'), ('mother', 'father'), ('daughter', 'son'), ('girl', 'boy'), ('female', 'male'), ('herself', 'himself')]
pairs_c = np.concatenate([np.stack([Gn[gidx[f]], Gn[gidx[m]]]) - (Gn[gidx[f]] + Gn[gidx[m]]) / 2 for f, m in pairs])
_, sv, Vt = np.linalg.svd(pairs_c, full_matrices=False); g = Vt[0] * np.sign(Vt[0] @ (Gn[gidx['she']] - Gn[gidx['he']]))
print(f'gender direction: first principal component of the {len(pairs)} centred pairs explains {sv[0] ** 2 / (sv ** 2).sum():.0%} of their variance')
jobs = ['nurse', 'receptionist', 'librarian', 'hairdresser', 'secretary', 'teacher', 'dancer', 'designer', 'writer', 'lawyer', 'doctor',
        'scientist', 'professor', 'manager', 'pilot', 'engineer', 'architect', 'programmer', 'surgeon', 'carpenter', 'mechanic', 'plumber']
proj = sorted(((Gn[gidx[w]] @ g, w) for w in jobs), reverse=True)
print('projection on the she-he direction (positive = closer to "she"):')
print('  ' + ', '.join(f'{w} {p:+.2f}' for p, w in proj))
man:king :: woman:?   excluding inputs: queen, monarch, throne   allowing them: king, queen, monarch
paris:france :: tokyo:?   excluding inputs: japan, korea, japanese   allowing them: japan, tokyo, korea
man:doctor :: woman:?   excluding inputs: nurse, physician, doctors   allowing them: doctor, nurse, physician
man:programmer :: woman:?   excluding inputs: educator, programmers, linguist   allowing them: programmer, educator, programmers
he:surgeon :: she:?   excluding inputs: nurse, dentist, physician   allowing them: surgeon, nurse, dentist
man:engineer :: woman:?   excluding inputs: technician, educator, engineers   allowing them: engineer, technician, educator
gender direction: first principal component of the 8 centred pairs explains 74% of their variance
projection on the she-he direction (positive = closer to "she"):
  nurse +0.30, receptionist +0.29, hairdresser +0.25, dancer +0.23, librarian +0.12, designer +0.06, teacher +0.05, doctor +0.01, writer -0.00, surgeon -0.03, carpenter -0.08, pilot -0.09, plumber -0.09, lawyer -0.10, professor -0.11, scientist -0.13, programmer -0.13, secretary -0.18, mechanic -0.20, engineer -0.24, architect -0.25, manager -0.33

The analogy man : king :: woman : ? returns "queen", and man : doctor :: woman : ? returns "nurse", the kind of result that made Bolukbasi et al.'s paper famous; he : surgeon :: she : ? also gives "nurse". But look at the second column. The standard analogy search excludes the three input words from the answers. When they are allowed, the nearest word to \(\text{doctor}-\text{man}+\text{woman}\) is "doctor" itself, and similarly for "programmer", "engineer" and "surgeon": the vectors say mainly that a woman doctor is a doctor. Projections onto a gender direction are a more direct measurement. Taking the first principal component of eight centred pairs (she/he, her/his, woman/man, …), which explains 74% of their variance, "nurse", "receptionist", "hairdresser" and "dancer" lie on the "she" side, and "manager", "architect", "engineer" and "mechanic" on the "he" side, mirroring occupational statistics (Caliskan et al. found a correlation of 0.90 between such projections and the percentage of women in each occupation in US labour statistics). Two cautions: "secretary" falls on the "he" side, because in news text "secretary" is mostly "Secretary of State"; and a single direction captures only part of how gender is encoded. Bolukbasi et al. proposed "debiasing" by removing the gender direction from words that should be neutral, but Gonen and Goldberg (2019) showed that the bias remained recoverable from the debiased vectors' neighbourhoods: projecting out one direction hides the symptom more than it removes the cause.

Analogies are a vivid but unreliable measuring instrument. A systematic test asks whether a whole set of target words is more strongly associated with one set of attribute words than another. Caliskan, Bryson and Narayanan (2017) adapted the Implicit Association Test of psychology (Greenwald, McGhee and Schwartz, 1998), which measures how much faster people pair, say, flowers with pleasant words than insects with pleasant words, into a test on vectors.

definition 13.1 — the Word-Embedding Association Test (WEAT)

Let \(X,Y\) be two sets of target words of equal size (flowers, insects) and \(A,B\) two sets of attribute words (pleasant, unpleasant). The association of a word \(w\) with the attributes is

$$s(w,A,B)=\frac1{|A|}\sum_{a\in A}\cos(w,a)-\frac1{|B|}\sum_{b\in B}\cos(w,b),$$

the test statistic is

$$S(X,Y,A,B)=\sum_{x\in X}s(x,A,B)-\sum_{y\in Y}s(y,A,B),$$

and the effect size is

$$d=\frac{\operatorname{mean}_{x\in X}s(x,A,B)-\operatorname{mean}_{y\in Y}s(y,A,B)}{\operatorname{sd}_{w\in X\cup Y}\,s(w,A,B)},$$

a standardised difference of means like Cohen's \(d\) (values around 0.8 and above are conventionally "large").

proposition 13.2 — the WEAT permutation test

Let \(\{X_i,Y_i\}\) range over all ways of splitting \(X\cup Y\) into two sets of equal size, and define the one-sided \(p\)-value

$$p=\Pr_i\big[S(X_i,Y_i,A,B)\ge S(X,Y,A,B)\big].$$

If the labels "\(X\)" and "\(Y\)" are exchangeable (the null hypothesis that the two target sets do not differ in their association with \(A\) and \(B\)), then \(\Pr(p\le\alpha)\le\alpha\) for every \(\alpha\in[0,1]\).

The same holds when \(p\) is estimated from \(M\) random splits as

$$(1+\#\{m:S_m\ge S\})/(M+1).$$

Condition on the multiset of association values \(\{s(w,A,B):w\in X\cup Y\}\). Under exchangeability, the observed split is equally likely to be any of the \(K=\binom{2n}{n}\) splits, so the observed statistic is a uniformly random draw from the list \(S_1,\dots,S_K\) of the statistics of all splits. Its rank from the top is then uniform, and \(p\) is its rank divided by \(K\). Hence \(\Pr(p\le k/K)\le k/K\) for each \(k\) (with equality when there are no ties), and so \(\Pr(p\le\alpha)\le\alpha\).

For the Monte Carlo version, under the null the observed statistic and the \(M\) random-split statistics are exchangeable, so the observed one is equally likely to occupy any of the \(M+1\) ranks, which gives the same bound for \((1+\#\{S_m\ge S\})/(M+1)\).

Note what the test does not say: the randomness is over which words were called \(X\), so it treats the word lists as given. A different, equally reasonable choice of words can change \(d\), and Ethayarajh, Duvenaud and Hirst (2019) showed how strongly WEAT results can depend on the choice of word lists. \(\square\)

code — reproducing six of Caliskan et al.'s tests
# The Word-Embedding Association Test (WEAT), reproducing Caliskan, Bryson and Narayanan (2017) on GloVe
W_ = lambda s: s.split()
weat = {
 'flowers vs insects / pleasant vs unpleasant': (W_('aster clover hyacinth marigold poppy azalea crocus iris orchid rose bluebell daffodil lilac pansy tulip buttercup daisy lily peony violet carnation gladiola magnolia petunia zinnia'),
    W_('ant caterpillar flea locust spider bedbug centipede fly maggot tarantula bee cockroach gnat mosquito termite beetle cricket hornet moth wasp blackfly dragonfly horsefly roach weevil'), 'P', 'U', 1.50),
 'instruments vs weapons / pleasant vs unpleasant': (W_('bagpipe cello guitar lute trombone banjo clarinet harmonica mandolin trumpet bassoon drum harp oboe tuba bell fiddle harpsichord piano viola bongo flute horn saxophone violin'),
    W_('arrow club gun missile spear axe dagger harpoon pistol sword blade dynamite hatchet rifle tank bomb firearm knife shotgun teargas cannon grenade mace slingshot whip'), 'P', 'U', 1.53),
 'European- vs African-American names / pleasant vs unpleasant': (W_('adam harry josh roger alan frank justin ryan andrew jack matthew stephen brad greg paul jonathan peter amanda courtney heather melanie katie betsy kristin nancy stephanie ellen lauren colleen emily megan rachel'),
    W_('alonzo jamel theo alphonse jerome leroy torrance darnell lamar lionel tyree deion lamont malik terrence tyrone lavon marcellus wardell nichelle shereen ebony latisha shaniqua jasmine tanisha tia lakisha latoya yolanda malika yvette'), 'P', 'U', 1.41),
 'male vs female names / career vs family': (W_('john paul mike kevin steve greg jeff bill'), W_('amy joan lisa sarah diana kate ann donna'),
    W_('executive management professional corporation salary office business career'), W_('home parents children family cousins marriage wedding relatives'), 1.81),
 'maths vs arts / male vs female terms': (W_('math algebra geometry calculus equations computation numbers addition'), W_('poetry art dance literature novel symphony drama sculpture'),
    W_('male man boy brother he him his son'), W_('female woman girl sister she her hers daughter'), 1.06),
 'science vs arts / male vs female terms': (W_('science technology physics chemistry einstein nasa experiment astronomy'), W_('poetry art shakespeare dance literature novel symphony drama'),
    W_('brother father uncle grandfather son he his him'), W_('sister mother aunt grandmother daughter she hers her'), 1.24)}
PLEAS = W_('caress freedom health love peace cheer friend heaven loyal pleasure diamond gentle honest lucky rainbow diploma gift honor miracle sunrise family happy laughter paradise vacation')
UNPL = W_('abuse crash filth murder sickness accident death grief poison stink assault disaster hatred pollute tragedy divorce jail poverty ugly cancer kill rotten vomit agony prison')
def weat_test(X, Y, A, B, n_perm=100000, seed=0):
    X, Y, A, B = ([w for w in S if w in gidx] for S in (X, Y, A, B))
    s = lambda w: Gn[[gidx[a] for a in A]] @ Gn[gidx[w]]                  # cosines with A
    assoc = np.array([s(w).mean() - (Gn[[gidx[b] for b in B]] @ Gn[gidx[w]]).mean() for w in X + Y])
    sx, sy = assoc[:len(X)], assoc[len(X):]
    d = (sx.mean() - sy.mean()) / assoc.std(ddof=1)                       # effect size
    r = np.random.default_rng(seed); n = len(X); stat = sx.sum() - sy.sum(); more = 0
    for _ in range(n_perm):                                                # permutation test over equal-size splits of X u Y
        p = r.permutation(len(assoc)); more += assoc[p[:n]].sum() - assoc[p[n:2 * n]].sum() >= stat
    return d, (more + 1) / (n_perm + 1), len(X), len(Y)
print(f'{"test":60s} {"words":>7s} {"effect size d":>14s} {"p (one-sided)":>14s} {"Caliskan et al.":>16s}')
for name, (X, Y, A, B, ref) in weat.items():
    A, B = (PLEAS if A == 'P' else A), (UNPL if B == 'U' else B)
    d, p, nx, ny = weat_test(X, Y, A, B)
    print(f'{name:60s} {nx:3d}+{ny:<3d} {d:14.2f} {p:14.1e} {ref:16.2f}')
test                                                           words  effect size d  p (one-sided)  Caliskan et al.
flowers vs insects / pleasant vs unpleasant                   22+22            1.22        1.0e-05             1.50
instruments vs weapons / pleasant vs unpleasant               25+25            1.30        1.0e-05             1.53
European- vs African-American names / pleasant vs unpleasant  32+24            0.80        1.4e-03             1.41
male vs female names / career vs family                        8+8             1.72        1.0e-04             1.81
maths vs arts / male vs female terms                           8+8             1.28        2.6e-03             1.06
science vs arts / male vs female terms                         8+8             1.12        8.1e-03             1.24

All six associations are reproduced, with large effect sizes and small permutation \(p\)-values: flowers and musical instruments are closer to pleasant words than insects and weapons; European-American names are closer to pleasant words than African-American names; male names to career words and female names to family words; and male terms to mathematics and science than to the arts. The effect sizes differ from Caliskan et al.'s, mostly downwards, for three reasons worth knowing. They used vectors trained on 840 billion words of web text; ours come from six billion words of Wikipedia and newswire. Some words in their lists are not among our 100,000 vocabulary words (three flowers and three insects, and eight of the African-American names), and the test drops them. And the effect size has a large standard error with eight words per set: a single ambiguous word ("jasmine" is also a flower, "jack" a tool) can move it noticeably. The \(p\)-values say the associations are not artefacts of which words were called \(X\); they do not say what the associations will do in an application. For that, test the application.

Exploring GloVe word vectors

Choose a set of words. "Principal components" projects their 100-dimensional GloVe vectors onto the two directions of greatest variance (Lecture 2's SVD, computed here in your browser). "Concept axes" uses two interpretable directions instead: horizontal, the she−he direction (from eight gendered word pairs); vertical, the mean of the 25 pleasant words minus the mean of the 25 unpleasant words.

spot check 13.3

The standard analogy code returns "nurse" for man : doctor :: woman : ?, but when the input words are allowed as answers, it returns "doctor". What does this show?

Look at the "allowing them" column for man : king :: woman : ?.
Nissim, van Noord and van der Goot (2020) made exactly this point. Measuring bias needs measurements that do not build in the answer, such as projections and association tests with stated word lists and uncertainty.

5 — From representations to decisions

Associations in an embedding are not yet harm; harm happens when a system built on them treats people differently. The direct test is an audit with counterfactual inputs: feed the system inputs that differ only in a protected attribute, or a proxy for one such as a name, and compare the outputs. Kiritchenko and Mohammad (2018) built the Equity Evaluation Corpus of 8,640 template sentences for this purpose and found that many of 219 sentiment systems scored sentences with African-American names or female terms systematically differently. Speer (2017) showed that a sentiment classifier built in the obvious way on GloVe vectors rated "Let's go get Mexican food" well below "Let's go get Italian food". We can check both on our own models.

code — scoring sentences that differ only in a name
# Does representation bias reach decisions? Score template sentences that differ only in a name
EA, AA = weat['European- vs African-American names / pleasant vs unpleasant'][:2]
EA, AA = [w for w in EA if w in gidx], [w for w in AA if w in gidx]
templates = ['I went to see a movie with {} last night.', '{} wrote the screenplay.', 'The film stars {} as the detective.',
             'My friend {} recommended it.', '{} directed this one.']
def logodds_glove(texts): return glove_lr.decision_function((glove_avg(texts) - mu) / sd)
def logodds_tfidf(texts): return lr.decision_function(tf.transform(texts))
print(f'{len(templates)} templates such as "{templates[0]}"; {len(EA)} European-American and {len(AA)} African-American names')
for name, f in [('GloVe average + logistic', logodds_glove), ('TF-IDF logistic', logodds_tfidf)]:
    score = lambda names: np.mean([f([t.format(n.capitalize()) for n in names]) for t in templates], axis=0)   # one number per name
    se, sa = score(EA), score(AA)
    diff = se.mean() - sa.mean(); allv = np.concatenate([se, sa]); r = np.random.default_rng(0)
    perm_d = np.array([(lambda p: allv[p[:len(se)]].mean() - allv[p[len(se):]].mean())(r.permutation(len(allv))) for _ in range(20000)])
    print(f'{name:25s} mean log-odds: European-American {se.mean():+.3f}, African-American {sa.mean():+.3f}; '
          f'difference {diff:+.3f} (permutation p = {(np.sum(perm_d >= diff) + 1) / 20001:.3f})')
for cuisine in ['Italian', 'French', 'Chinese', 'Mexican', 'Indian']:
    s_ = f"Let's go get {cuisine} food tonight."
    print(f'  "{s_}"  GloVe model log-odds {logodds_glove([s_])[0]:+.3f}   TF-IDF model {logodds_tfidf([s_])[0]:+.3f}')
5 templates such as "I went to see a movie with {} last night."; 32 European-American and 24 African-American names
GloVe average + logistic  mean log-odds: European-American +0.808, African-American +0.394; difference +0.415 (permutation p = 0.289)
TF-IDF logistic           mean log-odds: European-American +0.153, African-American +0.019; difference +0.134 (permutation p = 0.091)
  "Let's go get Italian food tonight."  GloVe model log-odds -2.276   TF-IDF model +0.100
  "Let's go get French food tonight."  GloVe model log-odds -1.724   TF-IDF model +0.321
  "Let's go get Chinese food tonight."  GloVe model log-odds -5.851   TF-IDF model +0.026
  "Let's go get Mexican food tonight."  GloVe model log-odds -3.952   TF-IDF model -0.312
  "Let's go get Indian food tonight."  GloVe model log-odds -3.370   TF-IDF model -0.158

Averaged over five templates, both models score sentences with European-American names more positively than the same sentences with African-American names, as the embedding association would predict, but the differences are small and not statistically significant (\(p=0.29\) for the GloVe model and \(p=0.09\) for the TF-IDF model, by a permutation test over names). With 56 names, an audit like this cannot rule out a modest effect, and it does not show one either: a larger set of names and templates, as in the Equity Evaluation Corpus, would be needed. The TF-IDF model hardly knows most of the names, so it barely reacts to them; its small differences come from names that happen to be characters or actors in reviews. The cuisine sentences are more striking. The GloVe model calls every one negative, and its scores range from \(-1.7\) for French food to \(-5.9\) for Chinese food, a factor of \(e^{4.1}\approx60\) in odds, for sentences with nothing to do with films. The TF-IDF model, which has seen these words only in reviews, scores them all near zero. A model built on general-purpose representations imports their associations into a domain where they have no business.

Does a contextual model behave better? The same audit on DistilBERT:

code — the name and cuisine audit on DistilBERT
# The name audit above, repeated on DistilBERT: P(positive) as log-odds, averaged over the five templates
tkd = AutoTokenizer.from_pretrained('distilbert/distilbert-base-uncased-finetuned-sst-2-english')
mdl = AutoModelForSequenceClassification.from_pretrained('distilbert/distilbert-base-uncased-finetuned-sst-2-english').eval()
def logodds_bert(texts):
    with torch.no_grad(): z = mdl(**tkd(texts, return_tensors='pt', padding=True)).logits
    return (z[:, 1] - z[:, 0]).numpy()
score = lambda names: np.mean([logodds_bert([t.format(n.capitalize()) for n in names]) for t in templates], axis=0)
se, sa = score(EA), score(AA); diff = se.mean() - sa.mean(); allv = np.concatenate([se, sa]); r = np.random.default_rng(0)
perm_d = np.array([(lambda p: allv[p[:len(se)]].mean() - allv[p[len(se):]].mean())(r.permutation(len(allv))) for _ in range(20000)])
print(f'DistilBERT-SST-2 on the name templates: mean log-odds European-American {se.mean():+.3f}, African-American {sa.mean():+.3f}; '
      f'difference {diff:+.3f} (permutation p = {(np.sum(perm_d >= diff) + 1) / 20001:.3f})')
for cuisine in ['Italian', 'French', 'Chinese', 'Mexican', 'Indian']:
    s_ = f"Let's go get {cuisine} food tonight."; print(f'  "{s_}"  DistilBERT log-odds {logodds_bert([s_])[0]:+.3f}')
DistilBERT-SST-2 on the name templates: mean log-odds European-American +5.962, African-American +5.800; difference +0.162 (permutation p = 0.150)
  "Let's go get Italian food tonight."  DistilBERT log-odds +2.440
  "Let's go get French food tonight."  DistilBERT log-odds +3.018
  "Let's go get Chinese food tonight."  DistilBERT log-odds +3.565
  "Let's go get Mexican food tonight."  DistilBERT log-odds +2.808
  "Let's go get Indian food tonight."  DistilBERT log-odds +4.221

DistilBERT scores every template as strongly positive (mean log-odds +5.962 with European-American names, +5.800 with African-American names), and the difference, +0.162, is again small and not significant (\(p=0.150\)). The cuisine sentences are all positive too, from +2.440 for Italian food to +4.221 for Indian, a spread of a factor of about 6 in odds, against 60 for the GloVe model. On this small audit the contextual model shows neither of the GloVe model's patterns. That is a statement about these 56 names and five sentences, not a certificate: larger audits have found such gaps in many sentiment systems (Kiritchenko and Mohammad, 2018), and in the representations of BERT itself (Kurita et al., 2019). An audit has to be repeated for every model and every use.

history — Gender Shades

In 2018 Joy Buolamwini and Timnit Gebru tested three commercial face-analysis systems (from IBM, Microsoft and Face++) on 1,270 photographs of parliamentarians from three African and three European countries, balanced by gender and skin type. Classifying gender, the systems erred on at most 0.8% of lighter-skinned men but on up to 34.7% of darker-skinned women. The benchmarks the companies had used were dominated by lighter-skinned faces, so their headline accuracies had hidden the gap. The companies released improved models within months, and a follow-up audit (Raji and Buolamwini, 2019) found the gaps had narrowed for the audited firms but not for others. The study made disaggregated evaluation (reporting performance separately for each group, and for intersections of groups) a standard part of responsible machine learning, and fed into the documentation practices of "datasheets for datasets" and "model cards".

6 — Fairness criteria

To say what a fair decision rule would be, fix notation. A model produces a score \(R\) (a risk, a probability) and perhaps a decision \(\hat Y\) (detain, approve); \(Y\) is the outcome (reoffends, repays); \(A\) is a protected attribute (race, sex). Most proposed definitions of group fairness are conditional independence statements among these three variables, and there are only three basic ones (Barocas, Hardt and Narayanan, 2023).

definition 13.2 — independence, separation, sufficiency

Each has an argument for it. Independence says outcomes should be shared out equally, whatever the base rates; it is the idea behind the "four-fifths rule" of US employment law, under which a selection rate for any group below 80% of the highest group's is evidence of adverse impact. Separation says people who will turn out the same way should be treated the same way: a person who would not have reoffended should not be more likely to be detained because of their group. Sufficiency says a score should mean what it says for everyone, so that a judge can read "70% risk" the same way for any defendant. In 2016 these three arguments collided in public.

data — COMPAS scores in Broward County, Florida

COMPAS (Correctional Offender Management Profiling for Alternative Sanctions), sold by Northpointe (now Equivant), scores defendants' risk of reoffending from a questionnaire and criminal record, on a scale of deciles 1–10. ProPublica obtained the scores of 7,214 defendants assessed in Broward County in 2013–2014 through public-records requests, matched them with criminal records over the following two years, and published the data and analysis (Angwin, Larson, Mattu and Kirchner, 2016; github.com/propublica/compas-analysis). We apply ProPublica's filters and keep the 5,278 African-American and white defendants; "reoffended" means a new arrest within two years, which measures policing as well as crime, a limitation discussed in Section 9. ProPublica, like us, calls a decile of 5 or more "medium or high risk". Names and other identifying fields in the file are not used.

code — auditing COMPAS by race
# Auditing COMPAS: error rates, predictive values and calibration by race (ProPublica's Broward County data)
cp = pd.read_csv('data/compas-scores-two-years.csv')
cp = cp[(cp.days_b_screening_arrest.abs() <= 30) & (cp.is_recid != -1) & (cp.c_charge_degree != 'O') & (cp.score_text != 'N/A')]   # ProPublica's filters
cp = cp[cp.race.isin(['African-American', 'Caucasian'])].copy()
cp['high'] = (cp.decile_score >= 5).astype(int); y_c = cp.two_year_recid.values                   # "medium or high risk" = decile 5-10
print(f'{len(cp):,} defendants (African-American or white), two-year recidivism rate {y_c.mean():.1%}')
def rates(pred, y):
    tp, fp = np.sum((pred == 1) & (y == 1)), np.sum((pred == 1) & (y == 0)); fn, tn = np.sum((pred == 0) & (y == 1)), np.sum((pred == 0) & (y == 0))
    return dict(n=len(y), base=y.mean(), flagged=pred.mean(), FPR=fp / (fp + tn), FNR=fn / (fn + tp), PPV=tp / (tp + fp), NPV=tn / (tn + fn), acc=(tp + tn) / len(y))
print(f'{"group":17s} {"n":>5s} {"base rate":>9s} {"flagged":>8s} {"FPR":>6s} {"FNR":>6s} {"PPV":>6s} {"NPV":>6s} {"accuracy":>8s}')
R = {}
for grp, lbl in [('African-American', 'Black'), ('Caucasian', 'white')]:
    s_ = cp.race.values == grp; R[lbl] = r_ = rates(cp.high.values[s_], y_c[s_])
    print(f'{lbl:17s} {r_["n"]:5d} {r_["base"]:9.1%} {r_["flagged"]:8.1%} {r_["FPR"]:6.1%} {r_["FNR"]:6.1%} {r_["PPV"]:6.1%} {r_["NPV"]:6.1%} {r_["acc"]:8.1%}')
for lbl, r_ in R.items():       # Chouldechova's identity: FPR = p/(1-p) * (1-PPV)/PPV * (1-FNR)
    print(f'Chouldechova identity, {lbl}: FPR = {r_["FPR"]:.4f}; p/(1-p) (1-PPV)/PPV (1-FNR) = {r_["base"] / (1 - r_["base"]) * (1 - r_["PPV"]) / r_["PPV"] * (1 - r_["FNR"]):.4f}')
print('recidivism rate by COMPAS decile (calibration):')
tab = cp.groupby(['decile_score', 'race']).two_year_recid.agg(['mean', 'size']).unstack()
print('  decile    ' + ' '.join(f'{d:>5d}' for d in range(1, 11)))
for grp, lbl in [('African-American', 'Black'), ('Caucasian', 'white')]:
    print(f'  {lbl:8s}  ' + ' '.join(f'{v:5.2f}' for v in tab[('mean', grp)].values))
# Dressel and Farid (2018): two features do as well
from sklearn.model_selection import cross_val_score
Xdf = cp[['age', 'priors_count']].values
print(f'COMPAS (decile >= 5) accuracy {np.mean(cp.high.values == y_c):.3f}; logistic regression on age and number of priors, '
      f'10-fold CV accuracy {cross_val_score(LogisticRegression(), Xdf, y_c, cv=10).mean():.3f}')
5,278 defendants (African-American or white), two-year recidivism rate 47.0%
group                 n base rate  flagged    FPR    FNR    PPV    NPV accuracy
Black              3175     52.3%    57.6%  42.3%  28.5%  65.0%  64.9%    64.9%
white              2103     39.1%    33.1%  22.0%  49.6%  59.5%  71.0%    67.2%
Chouldechova identity, Black: FPR = 0.4234; p/(1-p) (1-PPV)/PPV (1-FNR) = 0.4234
Chouldechova identity, white: FPR = 0.2201; p/(1-p) (1-PPV)/PPV (1-FNR) = 0.2201
recidivism rate by COMPAS decile (calibration):
  decile        1     2     3     4     5     6     7     8     9    10
  Black      0.23  0.30  0.42  0.47  0.49  0.59  0.61  0.71  0.72  0.84
  white      0.21  0.31  0.34  0.40  0.46  0.58  0.60  0.75  0.71  0.70
COMPAS (decile >= 5) accuracy 0.658; logistic regression on age and number of priors, 10-fold CV accuracy 0.677

The table reproduces ProPublica's central finding. Among defendants who were not rearrested within two years, 42.3% of Black defendants had been classed as medium or high risk, against 22.0% of white defendants: Black defendants bore almost twice the false-positive rate. Among those who were rearrested, white defendants were more often classed as low risk (false-negative rate 49.6% against 28.5%). Separation fails. (ProPublica's published figures, 44.9% and 23.5%, come from a slightly different sample; the pattern is the same.) Yet the positive predictive values are close (65.0% and 59.5%), and the calibration table shows that, decile by decile, rearrest rates are similar for the two groups: a defendant with a score of 7 was rearrested about 60% of the time whichever their race. Sufficiency approximately holds, except perhaps in the top decile, where there are only 50 white defendants. Finally, a logistic regression on just two inputs, age and the number of prior offences, matches COMPAS's accuracy (67.7% in cross-validation against 65.8%), as Dressel and Farid found.

7 — The impossibility theorems

The COMPAS numbers are not a coincidence of one tool. Two short theorems, found independently in 2016, show that when base rates differ between groups, no score or decision rule short of a perfect one can satisfy separation and sufficiency together.

theorem 13.3 — Chouldechova's identity (2017)

For a binary decision in a group with base rate \(p=P(Y=1)\), positive predictive value PPV, false-positive rate FPR and false-negative rate FNR,

$$\mathrm{FPR}=\frac{p}{1-p}\cdot\frac{1-\mathrm{PPV}}{\mathrm{PPV}}\cdot(1-\mathrm{FNR}).$$

Consequently, if two groups have different base rates and the same PPV, they cannot also have the same FPR and the same FNR (unless both rates are zero for one of them, or the PPV is 1).

Write the confusion matrix of a group of size \(N\) as counts TP, FP, FN, TN. Then \(\mathrm{TP}+\mathrm{FN}=pN\) and \(\mathrm{FP}+\mathrm{TN}=(1-p)N\), so \(\mathrm{TP}=(1-\mathrm{FNR})\,pN\) and \(\mathrm{FP}=\mathrm{FPR}\,(1-p)N\).

By definition \(\mathrm{PPV}=\mathrm{TP}/(\mathrm{TP}+\mathrm{FP})\), so

$$\mathrm{FP}=\mathrm{TP}\,(1-\mathrm{PPV})/\mathrm{PPV}.$$

Substituting,

$$\mathrm{FPR}\,(1-p)N=(1-\mathrm{FNR})\,pN\,\frac{1-\mathrm{PPV}}{\mathrm{PPV}},$$

and dividing by \((1-p)N\) gives the identity.

If the PPV and FNR are equal in two groups, the right-hand side differs between the groups through the factor \(p/(1-p)\), which is strictly increasing in \(p\); so the FPRs are equal only if the remaining factor \((1-\mathrm{PPV})(1-\mathrm{FNR})/\mathrm{PPV}\) is zero. \(\square\)

With the Broward numbers the identity holds to four decimal places (the code box checks it). Because more Black than white defendants were rearrested (52% against 39%), a tool with similar predictive values in both groups must have a higher false-positive rate for Black defendants. Kleinberg, Mullainathan and Raghavan proved the corresponding statement for scores rather than decisions.

theorem 13.4 — Kleinberg, Mullainathan and Raghavan (2017)

Consider a score \(S\in[0,1]\) and two groups with base rates \(p_1\ne p_2\). Suppose the score is

Then the score is perfect: \(S=Y\) with probability 1. (Kleinberg et al. also show an approximate version: approximate satisfaction of all three forces approximately equal base rates or approximately perfect prediction.)

Step 1: calibration fixes the average score. In group \(t\), by the tower property,

$$E[S]=E\big[E[Y\mid S]\big]=E[Y]=p_t.$$

Writing \(x_t=E[S\mid Y=0]\) and \(y_t=E[S\mid Y=1]\) in group \(t\), the same mean is \((1-p_t)x_t+p_ty_t\), so

$$(1-p_t)\,x_t+p_t\,y_t=p_t,\qquad t=1,2.$$

Step 2: balance makes the unknowns common. With \(x_1=x_2=x\) and \(y_1=y_2=y\), both equations hold for the same \((x,y)\). Rearranged, each says \(x+p_t(y-x-1)=0\). Subtracting the two,

$$(p_1-p_2)(y-x-1)=0,$$

and since \(p_1\ne p_2\), \(y=x+1\). As \(x,y\in[0,1]\), this forces \(x=0\), \(y=1\).

Step 3: conclude. \(E[S\mid Y=0]=0\) with \(S\ge0\) means \(S=0\) whenever \(Y=0\), and \(E[S\mid Y=1]=1\) with \(S\le1\) means \(S=1\) whenever \(Y=1\), almost surely. \(\square\)

The theorems turn a political argument into arithmetic. ProPublica measured separation and found it violated; Northpointe replied that its scores satisfied sufficiency (Dieterich, Mendoza and Brennan, 2016), which our calibration table confirms to within sampling error. Both were right, and both could not be satisfied at once. Which criterion should give way depends on the decision and on whose interests are at stake, and that is not a question that mathematics can settle. Explore the trade-off yourself.

Moving the thresholds on COMPAS

The 5,278 defendants of the code box, by COMPAS decile and race. Defendants with a decile at or above their group's threshold are classed as high risk. Move the thresholds separately for Black and white defendants, and watch which criteria hold. The "match" buttons keep the threshold for Black defendants and choose the white defendants' threshold that comes closest to equalising the false-positive rate or the selection rate (thresholds are whole deciles, so matches are approximate). Bars show the rates for Black (rose) and white (sky) defendants.

spot check 13.4

In a group with base rate \(p=0.4\), a classifier has PPV \(=0.6\) and FNR \(=0.4\). Use Theorem 13.3 to find its false-positive rate (three decimal places).

\(\frac{p}{1-p}=\frac23\) and \(\frac{1-\mathrm{PPV}}{\mathrm{PPV}}=\frac23\).
\(\tfrac23\cdot\tfrac23\cdot0.6=0.267\). A group with base rate 0.3 and the same PPV and FNR would have FPR \(\tfrac37\cdot\tfrac23\cdot0.6=0.171\): equal predictive values and equal miss rates force unequal false alarms.
history — "Machine Bias" and the debate it started

On 23 May 2016 ProPublica published "Machine Bias", by Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner, under the subheading "There's software used across the country to predict future criminals. And it's biased against blacks." Northpointe's rebuttal argued that the tool was equally accurate for both groups. Within months, Alexandra Chouldechova (then at Carnegie Mellon), Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan, and Sam Corbett-Davies and colleagues at Stanford independently showed that the disagreement was mathematically inevitable. In Wisconsin, the state Supreme Court ruled in State v. Loomis (2016) that judges may consider COMPAS scores at sentencing, provided they are warned of the tool's limitations. In 2018 Julia Dressel and Hany Farid showed that untrained people recruited online, given seven facts about each defendant, predicted reoffending about as accurately as COMPAS (67% against 65%), and that a logistic regression on two features, age and number of prior convictions, did as well as the tool's 137 inputs.

8 — Auditing the Lecture 8 income model

In Lecture 8 we trained XGBoost to predict whether a working Californian earns more than $50,000 a year, from the 2018 American Community Survey as packaged by Folktables (Ding et al., 2021). Folktables was built to replace the 1994 Census extract known as "Adult", which is small, old, and labelled with an income threshold fixed in 1994 dollars (Ding et al., 2021). Suppose such a model were used to decide who is offered a premium financial product. Here is the audit by sex, followed by two common remedies: deleting the sensitive attribute, and adjusting the threshold.

code — the Lecture 8 income model, audited by sex, without the sex column, and post-processed
# Auditing the Lecture 8 income model (XGBoost on the 2018 American Community Survey, California) by sex
import xgboost as xgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
from folktables import ACSIncome
acs = pd.read_csv('data/acs/2018/1-Year/psam_p06.csv', usecols=ACSIncome.features + ['PINCP', 'PWGTP'])
Xa, ya_, _ = ACSIncome.df_to_pandas(acs); ad = Xa.copy(); ad['y'] = ya_.iloc[:, 0].astype(int).values
use = list(ACSIncome.features); cat = ['COW', 'MAR', 'OCCP', 'POBP', 'RELP', 'SEX', 'RAC1P']
for c in cat: ad[c] = ad[c].astype(int).astype('category')
Atr, Ate = train_test_split(ad, test_size=0.3, stratify=ad['y'], random_state=0)        # the same split as Lecture 8
Afit, Aval = train_test_split(Atr, test_size=0.15, stratify=Atr['y'], random_state=1)
def fit_xgb(features, target='y'):
    return xgb.XGBClassifier(n_estimators=3000, learning_rate=0.05, max_depth=6, subsample=0.8, colsample_bytree=0.8, max_cat_to_onehot=1,
                             enable_categorical=True, tree_method='hist', early_stopping_rounds=100, random_state=0
                             ).fit(Afit[features], Afit[target].astype(int), eval_set=[(Aval[features], Aval[target].astype(int))], verbose=False)
female = (Ate.SEX == 2).values; ya = Ate.y.values                                        # ACS codes sex as 1 = male, 2 = female
def audit(name, p, thr_f=0.5, thr_m=0.5):
    pred = np.where(female, p > thr_f, p > thr_m).astype(int)
    print(f'{name}: accuracy {np.mean(pred == ya):.4f}')
    for g, s_ in [('  women', female), ('  men', ~female)]:
        r_ = rates(pred[s_], ya[s_])
        print(f'{g:7s} base rate {r_["base"]:5.1%}  predicted >$50K {r_["flagged"]:5.1%}  TPR {1 - r_["FNR"]:5.1%}  FPR {r_["FPR"]:5.1%}  '
              f'PPV {r_["PPV"]:5.1%}  mean score {p[s_].mean():5.1%}')
xa = fit_xgb(use); pa = xa.predict_proba(Ate[use])[:, 1]
audit('XGBoost, all features (Lecture 8)', pa)
no_sex = [c for c in use if c != 'SEX']; pb = fit_xgb(no_sex).predict_proba(Ate[no_sex])[:, 1]
audit('XGBoost without the sex column ("unawareness")', pb)
for d_ in (Afit, Aval): d_['is_f'] = (d_.SEX == 2)
ps_ = fit_xgb(no_sex, 'is_f').predict_proba(Ate[no_sex])[:, 1]                           # how much do the other columns reveal about sex?
print(f'  sex predicted from the other nine columns: AUC {roc_auc_score(female, ps_):.3f}')
tpr_m = np.mean(pa[~female & (ya == 1)] > 0.5)                          # equal opportunity: match women's TPR to men's
thr_f = np.quantile(pa[female & (ya == 1)], 1 - tpr_m)
audit(f'XGBoost with threshold {thr_f:.3f} for women (equal opportunity)', pa, thr_f=thr_f)
XGBoost, all features (Lecture 8): accuracy 0.8270
  women base rate 35.0%  predicted >$50K 34.0%  TPR 75.1%  FPR 11.9%  PPV 77.3%  mean score 35.0%
  men   base rate 46.4%  predicted >$50K 47.1%  TPR 81.2%  FPR 17.4%  PPV 80.2%  mean score 46.5%
XGBoost without the sex column ("unawareness"): accuracy 0.8268
  women base rate 35.0%  predicted >$50K 35.7%  TPR 77.3%  FPR 13.3%  PPV 75.8%  mean score 36.7%
  men   base rate 46.4%  predicted >$50K 45.4%  TPR 79.6%  FPR 15.8%  PPV 81.4%  mean score 45.0%
  sex predicted from the other nine columns: AUC 0.832
XGBoost with threshold 0.415 for women (equal opportunity): accuracy 0.8261
  women base rate 35.0%  predicted >$50K 38.6%  TPR 81.2%  FPR 15.5%  PPV 73.8%  mean score 35.0%
  men   base rate 46.4%  predicted >$50K 47.1%  TPR 81.2%  FPR 17.4%  PPV 80.2%  mean score 46.5%

The model (the same model as in Lecture 8, 82.7% accurate) predicts high income for 34.0% of women and 47.1% of men. The ratio of selection rates is 0.72, below the four-fifths rule's 0.8, so independence fails; but so do the base rates themselves (35.0% against 46.4%), because the data record a real income gap. Sufficiency nearly holds: the positive predictive values are close (77.3% and 80.2%), and the average score in each group matches its base rate (35.0% and 46.5%), as calibration requires (Theorem 13.4, step 1). Separation fails: the model finds 75.1% of high-earning women but 81.2% of high-earning men, and its false-positive rate is higher for men (17.4% against 11.9%). Theorem 13.3 says this pattern is forced by the different base rates once the predictive values are (nearly) equal.

The first remedy, "fairness through unawareness", deletes the sex column. It changes little (women's true-positive rate moves from 75.1% to 77.3%), because the other features carry much of the same information: a model given only the other nine columns tells women from men with AUC 0.832. What it does change is calibration within groups: without the sex column the model can no longer adjust for it, and women's average score (36.7%) now exceeds their base rate (35.0%) while men's (45.0%) falls below theirs (46.4%). Removing a protected attribute from the inputs rarely removes it from the model, and it can make the scores mean different things for different groups. The second remedy, post-processing (Hardt, Price and Srebro, 2016), keeps the model and chooses group-specific thresholds to equalise a criterion. Lowering the threshold for women from 0.5 to 0.415 equalises the true-positive rates at 81.2% (equal opportunity) at a cost of only 0.09 points of accuracy; but women's false-positive rate rises from 11.9% to 15.5% and their positive predictive value falls from 77.3% to 73.8%, so the scores now mean different things for the two groups. Equal opportunity was bought with sufficiency, as the theorems said it must be. And using a person's sex or race explicitly to set their threshold is itself unlawful in some settings: the US Civil Rights Act of 1991, for example, forbids adjusting employment test scores or cut-offs by race, sex or national origin. Fairness interventions are made inside a legal framework, not outside it.

spot check 13.5

A bank removes race from its credit model's inputs and finds that the approval rates by race barely change. What is the most likely reason?

How did the income model of Section 8 manage without the sex column?
"Unawareness" is not a fairness criterion that the theorems of Section 7 recognise, and in practice it is weak. It also prevents the audit itself: you cannot measure group error rates without recording group membership.
big picture — sentiment, and audits in law

Sentiment analysis is a large industry. Brand-monitoring services score millions of social-media posts a day for companies; trading firms and data vendors turn news and earnings-call sentiment into market signals, often with Loughran–McDonald-style dictionaries alongside fine-tuned language models; platforms use related classifiers to flag toxic content, where the same name and dialect biases matter even more (Sap et al., 2019, found that tweets in African-American English were up to twice as likely to be flagged as offensive). Fairness audits have moved from research into law. New York City's Local Law 144, enforced since July 2023, requires employers who use automated employment decision tools to commission an independent bias audit every year, publish the selection rates and impact ratios by sex and race or ethnicity (and their intersections), and notify candidates. The EU's AI Act (Regulation 2024/1689) classes systems used in employment, credit, education and access to essential services as high-risk and requires their providers to examine training data for possible biases; under the 2026 "Digital Omnibus" amendment (Regulation 2026/1744), those obligations apply from 2 December 2027.

9 — Causal notions of fairness

The three criteria of Section 6 are statistical: they are properties of the joint distribution of \(R\), \(Y\) and \(A\). Hardt, Price and Srebro (2016) pointed out their limit: two scenarios can produce exactly the same joint distribution while differing in whether the decision is intuitively fair, because they differ in why \(A\) and \(Y\) are related. Whether a disparity is unfair depends on the causal path that produces it. The 1973 graduate admissions at Berkeley are the classic example (Bickel, Hammel and O'Connell, 1975): women were admitted at a lower rate overall, but within departments their rates were similar or higher, because women applied more often to the departments that admitted fewer applicants. The path from sex to admission through choice of department was widely judged acceptable; a direct path would not have been. Answering such questions needs the causal models of Lectures 3, 4 and 8.

definition 13.3 — counterfactual fairness (Kusner, Loftus, Russell and Silva, 2017)

Given a structural causal model with background variables \(U\), a predictor \(\hat Y\) is counterfactually fair if, for every individual (every \(x\) and \(a\)) and every \(y\) and \(a'\),

$$P\big(\hat Y_{A\leftarrow a}(U)=y\mid X=x,A=a\big)=P\big(\hat Y_{A\leftarrow a'}(U)=y\mid X=x,A=a\big):$$

the prediction would have been the same had the individual's protected attribute been different, with everything not caused by it held fixed.

Computing it follows Pearl's three steps: infer the background variables from what we observe (abduction), set \(A\) to \(a'\) (action), and recompute the features and the prediction (prediction).

A predictor that uses only variables that are not descendants of \(A\) is automatically counterfactually fair.

Path-specific fairness relaxes the requirement to some causal paths from \(A\), treating others, like the choice of department, as legitimate (Kilbertus et al., 2017; Nabi and Shpitser, 2018).

code — counterfactual fairness in a world whose causal model we know
# Counterfactual fairness in a simulated world where we know the causal model (after Kusner et al., 2017)
from sklearn.linear_model import LinearRegression
r = np.random.default_rng(0); N = 20000
A = r.integers(0, 2, N)                           # group membership
K = r.normal(0, 1, N)                             # unobserved "knowledge", independent of A
e1, e2, e3 = r.normal(0, 0.6, (3, N))
def world(A):                                     # structural equations: the test and the grades are depressed for group 1
    X1 = K - 1.0 * A + e1; X2 = K - 0.5 * A + e2; return np.c_[X1, X2]
X = world(A); Yv = K + e3                         # the outcome depends only on K
Xcf = world(1 - A)                                # counterfactual: same people, same K and noise, group flipped
res = lambda X_, A_: X_ - np.c_[[X[A == a].mean(0) for a in (0, 1)]][A_]   # residuals of the features given the group
models = {'aware (features and group)': (np.c_[X, A], np.c_[Xcf, 1 - A]), 'unaware (features only)': (X, Xcf),
          'counterfactually fair (residuals)': (res(X, A), res(Xcf, 1 - A))}
print(f'{"predictor":36s} {"MSE":>6s} {"mean prediction, group 0 / 1":>30s} {"mean |change| if group flipped":>31s}')
for name, (Z, Zcf) in models.items():
    m = LinearRegression().fit(Z, Yv); yh, ycf = m.predict(Z), m.predict(Zcf)
    print(f'{name:36s} {np.mean((yh - Yv) ** 2):6.3f} {yh[A == 0].mean():+14.3f} / {yh[A == 1].mean():+.3f} {np.mean(np.abs(ycf - yh)):31.3f}')
predictor                               MSE   mean prediction, group 0 / 1  mean |change| if group flipped
aware (features and group)            0.511         +0.006 / -0.005                           0.000
unaware (features only)               0.594         +0.268 / -0.267                           0.525
counterfactually fair (residuals)     0.511         +0.000 / +0.000                           0.011

In the simulated world, the outcome depends only on an unobserved "knowledge" \(K\), which is the same in both groups; but group 1's test scores and grades are depressed (by 1.0 and 0.5) by barriers unrelated to knowledge. The unaware model, which uses the scores and grades but not the group, is the unfair one: it predicts 0.53 lower for group 1 than for group 0 on average, and flipping a person's group (holding their knowledge and luck fixed) would change their prediction by 0.53 on average. The aware linear model, given the group, learns to add back the penalty and is counterfactually fair, with lower error. The third predictor uses the residuals of the features after removing their group means, \(X-E[X\mid A]\), which in this additive model are functions of \(K\) and the noise alone: it is counterfactually fair by construction (the 0.011 is estimation error in the group means) and just as accurate. The example is idealised, since a real causal model is never known this precisely, but it makes the point that blindness to a protected attribute can produce exactly the unfairness it was meant to avoid.

causal — why fairness is ultimately a causal question

Every number in this lecture's audits depends on the label \(Y\), and labels are measurements. COMPAS's "reoffended" is a new arrest, and arrests depend on where police patrol; if one group is policed more heavily, the same behaviour produces more arrests, the base rates of Theorem 13.3 are partly artefacts of measurement, and a perfectly calibrated score inherits the distortion. Obermeyer, Powers, Vogeli and Mullainathan (2019) found the same mechanism in a widely used US health-care algorithm: it predicted future medical costs as a proxy for medical need, and because less money was spent on Black patients with the same illnesses, Black patients had to be considerably sicker than white patients to receive the same score. Changing the label to a direct measure of health would have raised the share of Black patients flagged for extra care from 17.7% to 46.5%. No statistical criterion computed from \((R,Y,A)\) can detect a biased \(Y\); asking what the label measures, and along which paths the protected attribute affects features, outcomes and measurements, is causal reasoning.

10 — Summary, exercises, projects, reading

summary

Next: Part IV begins. Lecture 14 drops the labels altogether and asks what can be learned from the structure of unlabelled data.

exercises
  1. A naive Bayes classifier is trained on four documents: positive "great fun", "great acting"; negative "boring plot", "great bore". With vocabulary {great, fun, acting, boring, plot, bore} and add-one smoothing, compute \(\theta_{yj}\) for "great" in both classes and the log-odds of the new document "great plot".

    Each class has 4 word tokens and \(V=6\), so \(\theta_{yj}=(c_{yj}+1)/10\). Positive: great 3/10, plot 1/10. Negative: great 2/10, plot 2/10. Priors are equal. Log-odds \(=\log\frac{0.3}{0.2}+\log\frac{0.1}{0.2}=0.405-0.693=-0.288\): slightly negative, because "plot" appeared only in a negative review.

  2. Show that if base rates differ between groups (\(Y\not\perp A\)), no score can satisfy both independence (\(R\perp A\)) and sufficiency (\(Y\perp A\mid R\)).

    Sufficiency gives \(P(Y=1\mid A=a)=\int P(Y=1\mid R=r)\,dP(r\mid A=a)\), with the integrand not depending on \(a\). Independence makes the distribution of \(R\) the same for every \(a\). Then \(P(Y=1\mid A=a)\) is the same for every \(a\), contradicting \(Y\not\perp A\). (In the language of conditional independence, \(R\perp A\) and \(Y\perp A\mid R\) together imply \(Y\perp A\).)

  3. Show that a binary decision satisfying both independence and separation in two groups with different base rates must be independent of the outcome, \(\hat Y\perp Y\) (a useless predictor).

    With common TPR and FPR, the selection rate in group \(a\) is \(\mathrm{TPR}\,p_a+\mathrm{FPR}\,(1-p_a)=\mathrm{FPR}+p_a(\mathrm{TPR}-\mathrm{FPR})\). Independence requires this to be the same for \(p_1\ne p_2\), so \(\mathrm{TPR}=\mathrm{FPR}\): the decision is positive with the same probability whatever the outcome.

  4. A WEAT has \(X=\{x_1,x_2\}\), \(Y=\{y_1,y_2\}\) with associations \(s(x_1)=0.10\), \(s(x_2)=0.06\), \(s(y_1)=-0.02\), \(s(y_2)=0.02\). Compute the effect size (sample standard deviation over all four words) and the exact one-sided permutation \(p\)-value.

    Means 0.08 and 0.00; the four values have mean 0.04 and sample standard deviation \(\sqrt{(0.0036+0.0004+0.0036+0.0004)/3}=0.0516\), so \(d=0.08/0.0516=1.55\). There are \(\binom42=6\) splits; the statistic \(S=\sum_Xs-\sum_Ys\) is 0.16 for the observed split and smaller for every other (the next largest is 0.08), so \(p=1/6\). With two words per set no result can be significant at 5%: the smallest possible \(p\) is \(1/6\).

  5. Using the COMPAS calibration table, explain why a single threshold applied to both groups gives the same positive predictive value only approximately, and why moving the white defendants' threshold down by one decile raises their false-positive rate.

    The PPV at threshold \(t\) is the average rearrest rate over deciles \(\ge t\), weighted by the number of defendants in each decile. Even if the rate in each decile were identical for the two groups, the weights differ (white defendants are concentrated in low deciles), so the averages differ: 65.0% against 59.5% at \(t=5\). Lowering the threshold adds a decile of defendants, among whom those not rearrested become new false positives, so the FPR can only rise.

  6. In the additive model \(X=K+\beta A+\varepsilon\) with \(K,\varepsilon\) independent of \(A\), show that \(\tilde X=X-E[X\mid A]\) is unchanged by the intervention \(A\leftarrow a'\) (holding \(K\) and \(\varepsilon\) fixed), so any predictor built from \(\tilde X\) is counterfactually fair. Why does this fail if \(\beta\) varies from person to person?

    \(E[X\mid A]=E[K]+\beta A+E[\varepsilon]\), so \(\tilde X=(K-E[K])+(\varepsilon-E[\varepsilon])\), which does not involve \(A\). If the effect \(\beta_i\) differs by person, \(X-E[X\mid A]=K+\varepsilon+(\beta_i-\bar\beta)A-\text{const}\) still depends on \(A\) through \((\beta_i-\bar\beta)A\), so flipping \(A\) changes it. Counterfactual fairness then needs the individual-level model, which observational data rarely identify.

student projects

Each project uses this lecture's methods and public data, and has a clear result.

1. A fairness audit of New York pretrial release decisions

question
Since New York's 2020 bail reform, how do release decisions and subsequent rearrest differ by race, sex and age, and what trade-offs between parity, error-rate balance and calibration would a simple risk model face on these data?
data
The New York State Unified Court System's public pretrial release data (case-level arraignment, release and rearrest records, published by the Office of Court Administration and the Division of Criminal Justice Services).
method
Fit a transparent risk model (logistic regression or a small tree ensemble) for rearrest; audit it by group as in Sections 6–8; show the impossibility trade-offs in real numbers with the threshold analysis of the widget; discuss rearrest as a measured, not true, label.
outcome
A reproducible audit of a live policy question in New York City, of interest to criminal-justice researchers and policy centres.

2. Name and dialect bias in sentiment models

question
Do lexicon, logistic and transformer sentiment models score otherwise identical sentences differently by the race or gender signalled by names, or by dialect?
data
The Equity Evaluation Corpus (Kiritchenko and Mohammad, 2018; 8,640 template sentences), plus tweets with estimated dialect (African-American English or not) from Blodgett, Green and O'Connor (2016).
method
Score with VADER, the IMDb logistic model of this lecture and an open pre-trained transformer; estimate group differences with permutation tests and confidence intervals, with enough names and templates for adequate power (Section 5's lesson).
outcome
A comparison across model families of a measurable harm, with a power analysis; a natural workshop paper.

3. What do Local Law 144 bias audits report?

question
What do the published bias audits of automated hiring tools used in New York City actually show, how consistent are their methods, and how often do impact ratios fall below the four-fifths threshold?
data
Audit summaries that employers must post publicly under Local Law 144, collected systematically from employers' career sites; the city's rules on audit methodology.
method
Build a dataset of reported selection rates and impact ratios by sex, race and their intersections; analyse completeness, sample sizes and the share below 0.8; compare with the definitions of Section 6.
outcome
An empirical study of an early algorithmic-audit law, relevant to policy makers in other jurisdictions.

Reading and sources

Data. IMDb reviews: A. L. Maas et al. (2011), CSV copy from github.com/Ankit152/IMDB-sentiment-analysis (the Stanford archive was not reachable from the build environment). Financial PhraseBank v1.0: Malo et al. (2014), CC BY-NC-SA 3.0, copy from github.com/maxwellsarpong/NLP-financial-text-processing-dataset. Lexicons: Harvard IV-4 and Loughran–McDonald master dictionary from the pysentiment2 package (PyPI); VADER lexicon from the vaderSentiment package (MIT licence). GloVe 6B 100-dimensional vectors (Pennington et al., 2014; Public Domain Dedication and License), from the gensim-data release on GitHub. COMPAS: ProPublica, github.com/propublica/compas-analysis. American Community Survey 2018 (US Census Bureau, public domain) via Folktables, as in Lecture 8. Pretrained models from the Hugging Face Hub: DistilBERT fine-tuned on SST-2 (distilbert/distilbert-base-uncased-finetuned-sst-2-english, Apache 2.0) and FinBERT (ProsusAI/finbert, CC BY-NC-SA 4.0).

numbers in the text not printed by a code box are computed in verify_L13.py, widget data in figs_L13.py; code-box outputs are pasted from actual runs (scikit-learn, XGBoost and JAX on two CPU cores)