Machine Learning · Lecture 17b

Lecture 17b
Copulas: Dependence Beyond Correlation

Part IV · Unsupervised learning  |  why correlation is not dependence, the probability integral transform, Sklar's theorem and the Fréchet–Hoeffding bounds, Kendall's tau, Spearman's rho and tail dependence, elliptical and Archimedean copulas, rank-based fitting, joint crashes in industry returns, synthetic tabular data, the nonparanormal graphical model  |  reading: Nelsen, An Introduction to Copulas (2006); McNeil, Frey & Embrechts, Quantitative Risk Management (2015), ch. 7; Genest, Ghoudi & Rivest (1995); Liu, Lafferty & Wasserman (2009)  |  mixture models are in Lecture 17a

Cartoon: a modest correlationTwo stick figures look at a chart of two share prices. One says the correlation is only 0.6, nothing to fear. The other points at a day when both prices fell off a cliff together and says: except on the days that matter. Correlation is 0.6.Nothing to fear. Except on the daysthat matter.
A correlation coefficient summarises dependence in one number; a copula describes where in the distribution the dependence lives
In this lecture
  1. Dependence beyond correlation
  2. Sklar's theorem: separating margins from dependence
  3. Measuring dependence with copulas: ranks and tails
  4. Copula families, fitting, and joint crashes
  5. Copulas in machine learning I: synthetic tabular data
  6. Copulas in machine learning II: the nonparanormal
  7. Summary, exercises, projects, reading

Lecture 14 estimated densities and Lecture 17a built them from mixtures. Both treated a multivariate distribution as one object. This lecture splits it in two. Every continuous joint distribution can be written as its one-dimensional margins, which say how each variable behaves on its own, glued together by a copula, which says how they move together and nothing else. The split is useful in practice. Margins are easy to estimate one at a time, even nonparametrically; dependence is where models go wrong, and a copula lets us model it separately, test it, and see where it lives: near the middle of the distribution, or in the tails, where joint crashes and joint floods happen.

By the end you should be able to state and use Sklar's theorem and the probability integral transform; compute Kendall's tau, Spearman's rho and tail dependence from a copula; choose between Gaussian, \(t\) and Archimedean copulas and fit them by maximum pseudo-likelihood on ranks; and use copulas in machine learning to generate synthetic tabular data and to estimate graphical models that are robust to unknown monotone transformations of the data.

1 — Dependence beyond correlation

Pearson's correlation measures linear association. It is the right summary for a bivariate normal distribution, where it determines the whole dependence structure, and a poor one almost everywhere else. Three facts make this concrete. It changes when one variable is put through an increasing transformation (a change of units from returns to cubed returns, or from income to log income), although the "amount of dependence" has not changed. It can be zero for variables that are perfectly dependent, such as \(X\) and \(X^2\) for a symmetric \(X\). And very different joint distributions share the same correlation, which matters most when the difference is in the tails.

The last point is the subject of this lecture's first experiment. We draw from five joint distributions that all have standard normal margins and all have the same Kendall rank correlation (Definition 17.8), and ask how often the two variables are extreme together.

code — five dependence structures with the same rank correlation, and a small copula library
# Same correlation, different dependence: five bivariate copulas with the same Kendall tau, on standard normal margins
import numpy as np, pandas as pd, time, warnings; warnings.filterwarnings('ignore')
from scipy import stats, optimize, special, integrate
Phi, Phinv = stats.norm.cdf, stats.norm.ppf
# --- a small copula library: samplers, log-densities, Kendall's tau and tail dependence for each family -------------------
def debye1(t):                                                  # D_1(t) = (1/t) int_0^t s / (e^s - 1) ds, for Frank's tau
    return integrate.quad(lambda s: s / np.expm1(s), 0, t)[0] / t if t != 0 else 1.0
FAM = {
 'Gaussian': dict(tau=lambda r: 2 / np.pi * np.arcsin(r), bounds=(-0.99, 0.99), lam=lambda r: (0.0, 0.0)),
 't (4 d.f.)': dict(tau=lambda r: 2 / np.pi * np.arcsin(r), bounds=(-0.99, 0.99),
                    lam=lambda r, nu=4: (2 * stats.t.cdf(-np.sqrt((nu + 1) * (1 - r) / (1 + r)), nu + 1),) * 2),
 'Clayton': dict(tau=lambda th: th / (th + 2), bounds=(0.01, 30), lam=lambda th: (2 ** (-1 / th), 0.0)),
 'Gumbel': dict(tau=lambda th: 1 - 1 / th, bounds=(1.001, 20), lam=lambda th: (0.0, 2 - 2 ** (1 / th))),
 'Frank': dict(tau=lambda th: 1 - 4 / th * (1 - debye1(th)), bounds=(0.01, 60), lam=lambda th: (0.0, 0.0)),
}
def logc(fam, th, u, v):                                        # log copula density at (u, v) in (0,1)^2
    if fam == 'Gaussian':
        x, y = Phinv(u), Phinv(v); return -0.5 * np.log(1 - th ** 2) - (th ** 2 * (x ** 2 + y ** 2) - 2 * th * x * y) / (2 * (1 - th ** 2))
    if fam.startswith('t'):
        nu = 4; x, y = stats.t.ppf(u, nu), stats.t.ppf(v, nu)
        q = (x ** 2 + y ** 2 - 2 * th * x * y) / (1 - th ** 2)
        lj = special.gammaln((nu + 2) / 2) - special.gammaln(nu / 2) - np.log(nu * np.pi) - 0.5 * np.log(1 - th ** 2) - (nu + 2) / 2 * np.log1p(q / nu)
        return lj - stats.t.logpdf(x, nu) - stats.t.logpdf(y, nu)
    if fam == 'Clayton':
        return np.log1p(th) - (th + 1) * (np.log(u) + np.log(v)) - (2 + 1 / th) * np.log(u ** -th + v ** -th - 1)
    if fam == 'Gumbel':
        a, b = -np.log(u), -np.log(v); A = a ** th + b ** th; A1 = A ** (1 / th)
        return -A1 - np.log(u * v) + (th - 1) * np.log(a * b) + (1 / th - 2) * np.log(A) + np.log(A1 + th - 1)
    if fam == 'Frank':
        e = -np.expm1(-th)
        return np.log(th * e) - th * (u + v) - 2 * np.log(np.abs(e - (-np.expm1(-th * u)) * (-np.expm1(-th * v))))
def sample(fam, th, n, g):                                      # n draws (U, V) from the copula
    u, w = g.random(n), g.random(n)
    if fam == 'Gaussian':
        z = g.multivariate_normal([0, 0], [[1, th], [th, 1]], n); return Phi(z[:, 0]), Phi(z[:, 1])
    if fam.startswith('t'):
        z = g.multivariate_normal([0, 0], [[1, th], [th, 1]], n) / np.sqrt(g.chisquare(4, n) / 4)[:, None]
        return stats.t.cdf(z[:, 0], 4), stats.t.cdf(z[:, 1], 4)
    if fam == 'Clayton':                                         # invert the conditional distribution C(v | u)
        return u, (u ** -th * (w ** (-th / (1 + th)) - 1) + 1) ** (-1 / th)
    if fam == 'Frank':
        return u, -np.log1p(w * -np.expm1(-th) / (w * (np.exp(-th * u) - 1) - np.exp(-th * u))) / th
    if fam == 'Gumbel':                                          # Marshall-Olkin: a positive stable frailty S, then exp(-(E/S)^(1/theta))
        al = 1 / th; U, E = np.pi * g.random(n), g.exponential(size=n)
        S = np.sin(al * U) / np.sin(U) ** (1 / al) * (np.sin((1 - al) * U) / E) ** ((1 - al) / al)
        return tuple(np.exp(-(g.exponential(size=n) / S) ** al) for _ in range(2))
def theta_for_tau(fam, tau):                                    # the parameter that gives a chosen Kendall's tau
    lo, hi = FAM[fam]['bounds']; return optimize.brentq(lambda t: FAM[fam]['tau'](t) - tau, lo, hi)
# --- the experiment ---------------------------------------------------------------------------------------------------------
g = np.random.default_rng(0); n = 100_000; tau = 0.5
print(f"{n:,} draws from each copula with Kendall's tau = {tau}, put on standard normal margins")
print("copula        parameter   Pearson r   Kendall tau   P(Y below its 1% quantile | X below its)   the same, above 99%")
for fam in FAM:
    th = theta_for_tau(fam, tau); u, v = sample(fam, th, n, g); x, y = Phinv(u), Phinv(v)
    qx, qy = np.quantile(x, [0.01, 0.99]), np.quantile(y, [0.01, 0.99])
    lo = np.mean(y[x < qx[0]] < qy[0]); hi = np.mean(y[x > qx[1]] > qy[1])
    print(f"{fam:12s} {th:10.3f} {np.corrcoef(x, y)[0, 1]:11.3f} {stats.kendalltau(x[:5000], y[:5000])[0]:13.3f} {lo:30.3f} {hi:28.3f}")
print("independent copula: either conditional probability would be 0.010")
100,000 draws from each copula with Kendall's tau = 0.5, put on standard normal margins
copula        parameter   Pearson r   Kendall tau   P(Y below its 1% quantile | X below its)   the same, above 99%
Gaussian          0.707       0.708         0.495                          0.258                        0.277
t (4 d.f.)        0.707       0.701         0.499                          0.419                        0.420
Clayton           2.000       0.682         0.495                          0.701                        0.036
Gumbel            2.000       0.701         0.493                          0.134                        0.590
Frank             5.736       0.659         0.510                          0.051                        0.056
independent copula: either conditional probability would be 0.010

All five samples have Kendall's \(\tau\) of 0.5, and their Pearson correlations are nearly the same too (0.659 to 0.708). Yet they behave very differently where it matters for risk. When \(X\) has one of its worst 1% of values, \(Y\) has one of its own worst 1% with probability 0.258 under the Gaussian copula, 0.419 under the \(t\) copula, 0.701 under Clayton, 0.134 under Gumbel and 0.051 under Frank, against 0.010 for independence. In the upper tail the order changes: Gumbel 0.590, Clayton 0.036. The Gaussian and Frank copulas do have joint extremes at the 1% level, but, as Section 3 shows, the probability tends to zero further out; the \(t\) copula keeps it positive in both tails, Clayton in the lower and Gumbel in the upper. One correlation coefficient cannot tell these apart.

A copula explorer

Choose a copula family and its strength, measured by Kendall's \(\tau\), and the margins to put on it. The scatter shows 1,500 draws. Uniform margins show the copula itself; normal or exponential margins show the same dependence in other clothes. Notice where the points crowd together: in the corners for some families, nowhere in particular for others.

2 — Sklar's theorem: separating margins from dependence

The construction rests on one transformation. Applying a variable's own distribution function to it produces a uniform variable, whatever the distribution was.

theorem 17.6 — the probability integral transform

Let \(X\) have a continuous distribution function \(F\). Then \(F(X)\) is uniformly distributed on \((0,1)\). Conversely, if \(U\) is uniform on \((0,1)\) and

$$F^{-}(u)=\inf\{x:F(x)\ge u\}$$

is the generalised inverse of any distribution function \(F\), then \(F^{-}(U)\) has distribution function \(F\).

For the converse, \(F^{-}(u)\le x\) holds exactly when \(u\le F(x)\), because \(F\) is non-decreasing and right-continuous. So

$$P\big(F^{-}(U)\le x\big)=P\big(U\le F(x)\big)=F(x).$$

For the first part, take \(u\in(0,1)\). By continuity there is an \(x\) with \(F(x)=u\), and \(F(X)\le u\) holds exactly when \(X\le F^{-}(u)\) up to an event of probability zero (where \(F\) is flat). Hence

$$P\big(F(X)\le u\big)=P\big(X\le F^{-}(u)\big)=F\big(F^{-}(u)\big)=u.\qquad\square$$

The converse is how computers sample from any distribution with a known quantile function, and it is how we shall put new margins on a copula.

definition 17.7 — copula

A \(d\)-dimensional copula is the distribution function of a random vector \((U_1,\dots,U_d)\) whose components are each uniformly distributed on \((0,1)\):

$$C(u_1,\dots,u_d)=P(U_1\le u_1,\dots,U_d\le u_d).$$

Three bivariate copulas are landmarks: the independence copula \(\Pi(u,v)=uv\); the comonotone copula \(M(u,v)=\min(u,v)\), the copula of \((U,U)\); and the countermonotone copula \(W(u,v)=\max(u+v-1,0)\), the copula of \((U,1-U)\). When \(C\) has a density, we write it \(c(u,v)=\partial^2C/\partial u\,\partial v\).

theorem 17.7 — Sklar's theorem (Sklar, 1959)

Let \(H\) be the joint distribution function of \((X_1,\dots,X_d)\), with margins \(F_1,\dots,F_d\). Then there is a copula \(C\) such that, for all \(x\),

$$H(x_1,\dots,x_d)=C\big(F_1(x_1),\dots,F_d(x_d)\big).$$

If the margins are continuous, \(C\) is unique: it is the distribution function of \(\big(F_1(X_1),\dots,F_d(X_d)\big)\). Conversely, for any copula \(C\) and any distribution functions \(F_1,\dots,F_d\), the formula defines a joint distribution with those margins. With densities, the joint density factorises into a copula part and a margin part:

$$h(x_1,\dots,x_d)=c\big(F_1(x_1),\dots,F_d(x_d)\big)\prod_{j=1}^df_j(x_j).$$

Let \(U_j=F_j(X_j)\). By Theorem 17.6 each \(U_j\) is uniform, so the distribution function \(C\) of \((U_1,\dots,U_d)\) is a copula. Because \(F_j\) is continuous and non-decreasing, \(X_j\le x_j\) holds exactly when \(F_j(X_j)\le F_j(x_j)\), up to an event of probability zero. Therefore

$$H(x)=P\big(F_1(X_1)\le F_1(x_1),\dots,F_d(X_d)\le F_d(x_d)\big)=C\big(F_1(x_1),\dots,F_d(x_d)\big).$$

Any other copula with this property agrees with \(C\) on \(\operatorname{Ran}F_1\times\dots\times\operatorname{Ran}F_d=(0,1)^d\), so \(C\) is unique. For the converse, let \((U_1,\dots,U_d)\sim C\) and \(X_j=F_j^{-}(U_j)\). By Theorem 17.6 each \(X_j\) has distribution \(F_j\), and, using \(F_j^-(u)\le x\iff u\le F_j(x)\),

$$P(X_1\le x_1,\dots,X_d\le x_d)=P\big(U_1\le F_1(x_1),\dots,U_d\le F_d(x_d)\big)=C\big(F_1(x_1),\dots,F_d(x_d)\big).$$

The density formula follows by differentiating once in each argument and using the chain rule. \(\square\)

The density formula is why the split pays off in estimation: the log-likelihood of a sample is the sum of one term for each margin and one term for the copula, so they can be fitted separately. Two simple facts complete the picture. Every copula lies between the two extreme cases, and the copula does not see changes of units.

theorem 17.8 — the Fréchet–Hoeffding bounds

Every bivariate copula satisfies

$$W(u,v)=\max(u+v-1,0)\;\le\;C(u,v)\;\le\;\min(u,v)=M(u,v).$$

The upper bound is attained when \(V\) is an increasing function of \(U\), the lower when it is a decreasing one.

\(C(u,v)=P(U\le u,V\le v)\) is at most \(P(U\le u)=u\) and at most \(P(V\le v)=v\). For the lower bound,

$$C(u,v)=P(U\le u)+P(V\le v)-P(U\le u\text{ or }V\le v)\ge u+v-1,$$

and \(C\ge0\). The copula of \((U,U)\) is \(P(U\le\min(u,v))=\min(u,v)\); that of \((U,1-U)\) is \(P(1-v\le U\le u)=\max(u+v-1,0)\). \(\square\)

proposition 17.9 — copulas ignore changes of scale

If \(g_1,\dots,g_d\) are strictly increasing, then \(\big(g_1(X_1),\dots,g_d(X_d)\big)\) has the same copula as \((X_1,\dots,X_d)\). In particular, anything computed from ranks alone is a property of the copula.

The distribution function of \(g_j(X_j)\) is \(F_j\circ g_j^{-1}\), and applying it to \(g_j(X_j)\) gives back \(F_j(X_j)\). So the vectors of probability integral transforms, whose distribution is the copula by Theorem 17.7, are identical. Ranks are preserved by increasing maps, so rank statistics are functions of the copula. \(\square\)

In data we do not know the margins, but we have a natural estimate of \(F_j(X_{ij})\): the rank of observation \(i\) among the \(n\) values of variable \(j\), scaled into the unit interval. The values

$$\hat U_{ij}=\frac{\operatorname{rank}(X_{ij})}{n+1}$$

are called pseudo-observations; dividing by \(n+1\) rather than \(n\) keeps them strictly inside \((0,1)\), where copula densities are finite. Here they are on real data.

data — daily returns of US industry portfolios

Kenneth French's data library at Dartmouth publishes daily returns of 30 value-weighted US industry portfolios, built from all NYSE, AMEX and NASDAQ stocks in CRSP and classified by SIC code, from July 1926 (file built from the September 2026 CRSP database). We use 9,254 trading days from January 1990 to September 2026 and two industries: Fin (banks, insurance, real estate, trading) and BusEq (computers, software, electronic equipment). The years 1990–2006 are used for fitting; 2007–2026, which contain the financial crisis of 2008 and the pandemic crash of 2020, are held out.

code — heavy-tailed margins, the probability integral transform and ranks
# Daily returns of two US industry portfolios: the margins are not normal, and ranks fix that (Kenneth French's data library)
import io
txt = open('data/30_Industry_Portfolios_Daily.csv').read()
blk = txt.split('Average Value Weighted Returns -- Daily')[1].split('Average Equal Weighted')[0]
ind = pd.read_csv(io.StringIO(blk.strip()), index_col=0); ind.columns = ind.columns.str.strip()
ind.index = pd.to_datetime(ind.index.astype(str).str.strip(), format='%Y%m%d'); ind = ind['1990':]
train, test = ind[:'2006'], ind['2007':]                        # fit on 1990-2006; the 2007-2026 period, with 2008 and 2020, is held out
print(f"{len(ind):,} trading days, {ind.index[0]:%d %b %Y} to {ind.index[-1]:%d %b %Y}; training {len(train):,}, test {len(test):,}")
pair = ['Fin', 'BusEq']                                          # financial firms; business equipment (computers, software, electronics)
X = train[pair].to_numpy()
for j, name in enumerate(pair):
    x = X[:, j]; sd = x.std()
    print(f"{name:6s} mean {x.mean():+.3f}%  sd {sd:.3f}%  skewness {stats.skew(x):+.2f}  excess kurtosis {stats.kurtosis(x):5.1f}  "
          f"days beyond 4 sd: {np.mean(np.abs(x - x.mean()) > 4 * sd):.4f} (normal: {2 * stats.norm.sf(4):.5f})")
# the probability integral transform with a fitted normal margin, and with the ranks
pseudo = lambda A: stats.rankdata(A, axis=0) / (len(A) + 1)   # pseudo-observations: ranks scaled into (0, 1)
U_norm = Phi((X - X.mean(0)) / X.std(0)); U_rank = pseudo(X)
for j, name in enumerate(pair):
    print(f"{name:6s} Kolmogorov-Smirnov distance from uniform: normal margin {stats.kstest(U_norm[:, j], 'uniform').statistic:.3f}"
          f" (p = {stats.kstest(U_norm[:, j], 'uniform').pvalue:.1e}), ranks {stats.kstest(U_rank[:, j], 'uniform').statistic:.4f}")
print(f"Pearson r {np.corrcoef(X.T)[0, 1]:.3f}; Kendall tau {stats.kendalltau(*X.T)[0]:.3f}; Spearman rho {stats.spearmanr(*X.T)[0]:.3f}")
print(f"the same rank correlations after cubing one series (a strictly increasing transform): Kendall {stats.kendalltau(X[:, 0] ** 3, X[:, 1])[0]:.3f}, "
      f"Spearman {stats.spearmanr(X[:, 0] ** 3, X[:, 1])[0]:.3f}; Pearson becomes {np.corrcoef(X[:, 0] ** 3, X[:, 1])[0, 1]:.3f}")
9,254 trading days, 02 Jan 1990 to 30 Sep 2026; training 4,287, test 4,967
Fin    mean +0.063%  sd 1.124%  skewness +0.13  excess kurtosis   4.0  days beyond 4 sd: 0.0047 (normal: 0.00006)
BusEq  mean +0.059%  sd 1.888%  skewness +0.36  excess kurtosis   5.6  days beyond 4 sd: 0.0056 (normal: 0.00006)
Fin    Kolmogorov-Smirnov distance from uniform: normal margin 0.065 (p = 2.2e-16), ranks 0.0036
BusEq  Kolmogorov-Smirnov distance from uniform: normal margin 0.069 (p = 5.7e-18), ranks 0.0023
Pearson r 0.584; Kendall tau 0.420; Spearman rho 0.571
the same rank correlations after cubing one series (a strictly increasing transform): Kendall 0.420, Spearman 0.571; Pearson becomes 0.357

Both margins are far from normal. Their excess kurtosis is 4.0 and 5.6 (0 for a normal distribution), and moves beyond four standard deviations happen on 0.47% and 0.56% of days, roughly 74 and 88 times as often as a normal distribution allows. So the probability integral transform with a fitted normal margin does not give uniform values: the Kolmogorov–Smirnov distance is 0.065 and 0.069, with \(p\)-values below \(10^{-15}\). The ranks are uniform by construction (distances 0.0036 and 0.0023, only the discreteness of \(i/(n+1)\)). Pearson's correlation is 0.584, Kendall's \(\tau\) 0.420 and Spearman's \(\rho\) 0.571. Cube one of the series, a strictly increasing change of units, and the two rank correlations do not move while Pearson's falls to 0.357: by Proposition 17.9 the copula, and everything computed from ranks, ignores the margins.

spot check 17.5

You fit a copula to the pseudo-observations of two variables, then discover that one variable was recorded in logarithms. What do you need to refit?

What does a strictly increasing transformation do to the ranks (Proposition 17.9)?
The log is strictly increasing, so the ranks of the variable do not change, and neither do the pseudo-observations or anything fitted to them. Only a model of that variable's own distribution needs refitting.

3 — Measuring dependence with copulas: ranks and tails

If dependence is the copula, a measure of dependence should be a property of the copula alone. Pearson's correlation is not (it changed when we cubed a series), but two older rank correlations are.

definition 17.8 — Kendall's tau and Spearman's rho

Let \((X_1,Y_1)\) and \((X_2,Y_2)\) be independent copies of \((X,Y)\). The pair is concordant if \((X_1-X_2)(Y_1-Y_2)>0\) and discordant if the product is negative. Kendall's tau is

$$\tau=P\big[(X_1-X_2)(Y_1-Y_2)\gt0\big]-P\big[(X_1-X_2)(Y_1-Y_2)\lt0\big],$$

and Spearman's rho is the Pearson correlation of the probability integral transforms,

$$\rho_S=\operatorname{corr}\big(F(X),G(Y)\big).$$

Their sample versions use the observed pairs and the ranks (Kendall, 1938; Spearman, 1904). Both lie in \([-1,1]\), equal \(1\) for the copula \(M\), \(-1\) for \(W\) and \(0\) for \(\Pi\).

proposition 17.10 — rank correlations are functionals of the copula

For continuous \((X,Y)\) with copula \(C\),

$$\tau=4\int_0^1\!\!\int_0^1C(u,v)\,dC(u,v)-1,\qquad\rho_S=12\int_0^1\!\!\int_0^1\big(C(u,v)-uv\big)\,du\,dv.$$

For the Gaussian copula with correlation parameter \(r\) (the copula of a bivariate normal with correlation \(r\)),

$$\tau=\frac2\pi\arcsin r,\qquad\rho_S=\frac6\pi\arcsin\frac r2.$$

By Proposition 17.9 we may work with \((U,V)=(F(X),G(Y))\). Concordance has probability \(2P(U_2\lt U_1,V_2\lt V_1)=2\,\mathbb E[C(U_1,V_1)]\), and discordance has probability one minus that, so \(\tau=4\,\mathbb E[C(U,V)]-1\), which is the first formula. For Spearman's rho, \(\operatorname{var}U=\operatorname{var}V=1/12\), and Hoeffding's covariance identity gives

$$\operatorname{cov}(U,V)=\int_0^1\!\!\int_0^1\big(C(u,v)-uv\big)\,du\,dv.$$

For the Gaussian copula take \((X,Y)\) bivariate normal with correlation \(r\). The differences \((X_1-X_2,\,Y_1-Y_2)\) are again bivariate normal with correlation \(r\), and for such a pair the probability that both have the same sign is \(\tfrac12+\tfrac1\pi\arcsin r\) (Sheppard, 1899). So \(\tau=2\big(\tfrac12+\tfrac1\pi\arcsin r\big)-1=\tfrac2\pi\arcsin r\). The formula for \(\rho_S\) follows in the same way from the correlation of \(\Phi(X)\) and \(\Phi(Y)\). \(\square\)

spot check 17.6

A Gaussian copula has correlation parameter \(r=0.5\). What is its Kendall's \(\tau\)? (Three decimals.)

Use \(\tau=(2/\pi)\arcsin r\), and \(\arcsin\tfrac12=\pi/6\).
\(\tau=(2/\pi)(\pi/6)=1/3\approx0.333\). Kendall's tau is smaller than the correlation parameter, as it is for any \(0\lt r\lt1\).

Rank correlations summarise the whole copula. Risk depends on its corners: the probability that both variables are extreme together, as the probability of the event shrinks.

definition 17.9 — tail dependence

The lower and upper tail-dependence coefficients of a bivariate copula are the limits, when they exist,

$$\lambda_L=\lim_{q\to0}P\big(V\le q\mid U\le q\big)=\lim_{q\to0}\frac{C(q,q)}q,$$ $$\lambda_U=\lim_{q\to1}P\big(V\gt q\mid U\gt q\big)=\lim_{q\to1}\frac{1-2q+C(q,q)}{1-q}.$$

A copula with \(\lambda_L\gt0\) makes joint extremes in the lower tail stay likely however extreme the threshold; with \(\lambda_L=0\) they become, relative to one extreme, ever rarer.

proposition 17.11 — tail dependence of common copulas

For the Gaussian copula with \(r\lt1\), \(\lambda_L=\lambda_U=0\). For the \(t\) copula with correlation \(r\) and \(\nu\) degrees of freedom,

$$\lambda_L=\lambda_U=2\,t_{\nu+1}\!\left(-\sqrt{\frac{(\nu+1)(1-r)}{1+r}}\right),$$

where \(t_{\nu+1}\) is the Student \(t\) distribution function. The Clayton copula (Definition 17.10) has \(\lambda_L=2^{-1/\theta}\) and \(\lambda_U=0\); the Gumbel copula has \(\lambda_L=0\) and \(\lambda_U=2-2^{1/\theta}\); the Frank copula has neither.

For Clayton, \(C(q,q)=(2q^{-\theta}-1)^{-1/\theta}\), so

$$\frac{C(q,q)}q=\big(2-q^{\theta}\big)^{-1/\theta}\to2^{-1/\theta}\qquad(q\to0).$$

For Gumbel, \(C(q,q)=\exp\!\big(-(2(-\ln q)^\theta)^{1/\theta}\big)=q^{2^{1/\theta}}\). Writing \(a=2^{1/\theta}\), l'Hôpital's rule gives

$$\lambda_U=\lim_{q\to1}\frac{1-2q+q^{a}}{1-q}=\lim_{q\to1}\frac{-2+a\,q^{a-1}}{-1}=2-a.$$

The Gaussian and \(t\) results need the asymptotics of the bivariate normal and \(t\) distributions; see McNeil, Frey and Embrechts (2015, section 7.2.4). \(\square\)

spot check 17.7

A Clayton copula has \(\theta=2\), which gives Kendall's \(\tau=0.5\). What is its lower tail-dependence coefficient? (Three decimals.)

\(\lambda_L=2^{-1/\theta}\).
\(2^{-1/2}=0.707\). In the first code box, the conditional probability at the 1% level was already close to this limit.

Tail dependence, as the threshold moves into the tail

For five copulas with the same Kendall's \(\tau\), the curves show \(P(V\le q\mid U\le q)\), estimated from 40,000 draws each, as \(q\) moves from 0.5 into the lower tail. Curves that bend down towards zero have no lower tail dependence; those that level off at a positive height do, and the dashed marks show the limits \(\lambda_L\) of Proposition 17.11.

4 — Copula families, fitting, and joint crashes

Two kinds of parametric copula cover most applications. Elliptical copulas are taken from elliptical distributions by Sklar's theorem: the Gaussian copula from the multivariate normal, the \(t\) copula from the multivariate \(t\). They extend to any dimension with a correlation matrix as parameter, and the \(t\) copula adds one parameter, \(\nu\), that controls joint extremes in both tails symmetrically. Archimedean copulas are built from a single function.

definition 17.10 — Archimedean copulas

Let \(\varphi:[0,1]\to[0,\infty]\) be continuous, strictly decreasing and convex with \(\varphi(1)=0\). The Archimedean copula with generator \(\varphi\) is

$$C(u,v)=\varphi^{-1}\big(\varphi(u)+\varphi(v)\big).$$

(For a generator with \(\varphi(0)\lt\infty\), \(\varphi^{-1}\) is replaced by the pseudo-inverse, equal to \(\varphi^{-1}\) on \([0,\varphi(0)]\) and to 0 beyond; the three families below have \(\varphi(0)=\infty\).) Three standard choices, each with one parameter \(\theta\):

Rotating a copula by 180 degrees, \((U,V)\mapsto(1-U,1-V)\), swaps its tails: the survival Gumbel copula has dependence in the lower tail.

How should a copula be fitted when the margins are unknown? The density factorisation of Sklar's theorem suggests fitting the margins first and then the copula to the transformed data, but a wrong margin then contaminates the copula. Replacing the margins by ranks avoids that.

definition 17.11 — maximum pseudo-likelihood

Given pseudo-observations \((\hat U_i,\hat V_i)\), the maximum pseudo-likelihood estimate of a copula parameter \(\theta\) is

$$\hat\theta=\arg\max_\theta\sum_{i=1}^n\log c\big(\hat U_i,\hat V_i;\theta\big).$$

It needs no model for the margins, and it is consistent and asymptotically normal (Genest, Ghoudi and Rivest, 1995). Families with different numbers of parameters are compared by AIC, \(2k-2\log L\), as in Lecture 17a.

We fit six families to the 1990–2006 returns of the two industries, and then ask the question a risk manager asks: in the held-out years, on a day when financial stocks had one of their worst days, how likely was business equipment to have one too?

code — six copulas fitted on 1990–2006, tested on 2007–2026
# Fit six copula families to the training years by maximum pseudo-likelihood; test them on 2007-2026
def logc_t(r, nu, u, v):
    x, y = stats.t.ppf(u, nu), stats.t.ppf(v, nu); q = (x ** 2 + y ** 2 - 2 * r * x * y) / (1 - r ** 2)
    lj = special.gammaln((nu + 2) / 2) - special.gammaln(nu / 2) - np.log(nu * np.pi) - 0.5 * np.log(1 - r ** 2) - (nu + 2) / 2 * np.log1p(q / nu)
    return lj - stats.t.logpdf(x, nu) - stats.t.logpdf(y, nu)
def loglik(fam, par, u, v):
    if fam == 't': return logc_t(par[0], par[1], u, v)
    if fam == 'survival Gumbel': return logc('Gumbel', par[0], 1 - u, 1 - v)       # Gumbel rotated by 180 degrees: a lower tail
    return logc(fam, par[0], u, v)
def fit(fam, u, v):
    if fam == 't':                                                # profile likelihood over the degrees of freedom
        best = None
        for nu in [2, 2.5, 3, 3.5, 4, 5, 6, 8, 10, 15, 30]:
            r = optimize.minimize_scalar(lambda r: -np.sum(logc_t(r, nu, u, v)), bounds=(-0.99, 0.99), method='bounded')
            if best is None or r.fun < best[1]: best = ((r.x, nu), r.fun)
        return best[0], -best[1]
    lo, hi = FAM['Gumbel' if fam == 'survival Gumbel' else fam]['bounds']
    r = optimize.minimize_scalar(lambda t: -np.sum(loglik(fam, (t,), u, v)), bounds=(lo, hi), method='bounded'); return (r.x,), -r.fun
def C_diag(fam, par, q):                                        # C(q, q): the probability that both are below their q-quantiles
    if fam == 'Gaussian': return stats.multivariate_normal([0, 0], [[1, par[0]], [par[0], 1]]).cdf([Phinv(q)] * 2)
    if fam == 't': return stats.multivariate_t([0, 0], [[1, par[0]], [par[0], 1]], df=par[1]).cdf([stats.t.ppf(q, par[1])] * 2)
    if fam == 'Clayton': return (2 * q ** -par[0] - 1) ** (-1 / par[0])
    if fam == 'Gumbel': return q ** (2 ** (1 / par[0]))
    if fam == 'survival Gumbel': return 2 * q - 1 + (1 - q) ** (2 ** (1 / par[0]))
    if fam == 'Frank': th = par[0]; return -np.log1p(np.expm1(-th * q) ** 2 / np.expm1(-th)) / th
def tails(fam, par):
    if fam == 't': return (2 * stats.t.cdf(-np.sqrt((par[1] + 1) * (1 - par[0]) / (1 + par[0])), par[1] + 1),) * 2
    if fam == 'survival Gumbel': return FAM['Gumbel']['lam'](par[0])[::-1]
    return FAM[fam]['lam'](par[0])
def tau_of(fam, par):
    return FAM['Gumbel' if fam == 'survival Gumbel' else 'Gaussian' if fam == 't' else fam]['tau'](par[0])
u, v = U_rank.T; Ut = pseudo(test[pair].to_numpy()); ut, vt = Ut.T
fams = ['Gaussian', 't', 'Clayton', 'Gumbel', 'survival Gumbel', 'Frank']; fits = {}
print(f"Fin and BusEq, {len(u):,} training days; empirical Kendall tau {stats.kendalltau(u, v)[0]:.3f} (test years: {stats.kendalltau(ut, vt)[0]:.3f})")
print("family            parameters          log-lik      AIC     implied tau   lambda_L   lambda_U   test log-lik per day")
for fam in fams:
    par, ll = fit(fam, u, v); fits[fam] = par; k = len(par)
    ps = ', '.join(f'{p:.3g}' for p in par); lL, lU = tails(fam, par)
    print(f"{fam:16s} {ps:16s} {ll:10.1f} {2 * k - 2 * ll:9.1f} {tau_of(fam, par):12.3f} {lL:10.3f} {lU:10.3f} {np.mean(loglik(fam, par, ut, vt)):17.4f}")
print("\nP(BusEq has one of its worst q of days | Fin does), test years 2007-2026, against each copula fitted on 1990-2006:")
print("q        observed  " + "  ".join(f"{f[:10]:>10s}" for f in fams))
for q in [0.05, 0.01]:
    obs = np.mean(vt[ut < q] < q)
    print(f"{q:<8} {obs:9.3f}  " + "  ".join(f"{C_diag(f, fits[f], q) / q:10.3f}" for f in fams))
Fin and BusEq, 4,287 training days; empirical Kendall tau 0.420 (test years: 0.469)
family            parameters          log-lik      AIC     implied tau   lambda_L   lambda_U   test log-lik per day
Gaussian         0.588                 906.7   -1811.4        0.400      0.000      0.000            0.2876
t                0.611, 3.5           1105.3   -2206.5        0.418      0.350      0.350            0.3405
Clayton          0.974                 747.0   -1492.0        0.328      0.491      0.000            0.2601
Gumbel           1.68                  980.6   -1959.2        0.406      0.000      0.491            0.2939
survival Gumbel  1.67                  941.9   -1881.8        0.401      0.485      0.000            0.3137
Frank            4.62                  921.3   -1840.6        0.432      0.000      0.000            0.2686

P(BusEq has one of its worst q of days | Fin does), test years 2007-2026, against each copula fitted on 1990-2006:
q        observed    Gaussian           t     Clayton      Gumbel  survival G       Frank
0.05         0.546       0.302       0.420       0.505       0.217       0.505       0.190
0.01         0.490       0.180       0.378       0.494       0.096       0.489       0.045

The \(t\) copula wins on every likelihood measure. Its AIC is −2206.5, against −1959.2 for the runner-up (Gumbel) and −1811.4 for the Gaussian; on the held-out years it also has the highest log-likelihood per day (0.3405, Gaussian 0.2876). Its fitted degrees of freedom, \(\nu=3.5\), are small, and the tail dependence they imply, \(\lambda=0.350\) in each tail, is far from the Gaussian copula's zero. The crash table shows what the Gaussian copula misses. In 2007–2026, on the days when financial stocks had one of their worst 1% of days, business equipment had one of its own worst 1% 49% of the time. The Gaussian copula fitted to 1990–2006 predicts 18%, the \(t\) copula 38%, and the two lower-tail families, Clayton and survival Gumbel, 49%. There is a trade-off. Clayton and survival Gumbel describe the lower tail well and the rest of the distribution less well (Clayton has the worst AIC of all, and survival Gumbel trails the \(t\) and Gumbel copulas), because they have no upper tail dependence; the \(t\) copula is the best single description but treats booms like crashes. Part of the miss is also the data's fault: Kendall's \(\tau\) rose from 0.420 in the training years to 0.469 in the test years. Dependence is not constant, and it tends to rise in crises, which is the case for models whose copula parameters change over time.

0.0050.010.020.050.10.20.500.20.40.60.8q (log scale)conditional probabilitydata, test years 2007–2026Gaussian copulat copulasurvival Gumbel copulaindependence, P = q
The conditional probability that BusEq is in its worst \(q\) of days given that Fin is, in the test years 2007–2026 (dots), against the predictions of three copulas fitted to 1990–2006. The dashed curve is independence, \(P=q\).

The fitted copulas against the data

The dots are 1,500 of the 4,287 training days as pseudo-observations (each industry's rank, scaled into \((0,1)\)). Choose a fitted copula to draw the same number of points from it, in colour, and zoom into the lower-left corner, where both industries have bad days.

history — from Fréchet's question to "the formula that killed Wall Street"

Wassily Hoeffding (1940) and Maurice Fréchet (1951) studied joint distributions with given margins and found the bounds of Theorem 17.8. Answering Fréchet's question of which joint distributions are possible, Abe Sklar (1959) proved the theorem that bears his name and coined the word "copula", Latin for a link or bond. For forty years copulas were mostly a topic in probability theory, until Paul Embrechts, Alexander McNeil and Daniel Straumann (1999, published 2002) showed risk managers how badly correlation can mislead them outside the normal world. In 2000 David X. Li proposed using a Gaussian copula to model the joint default times of companies, and the method became the industry standard for pricing collateralised debt obligations: a single correlation parameter summarised the dependence of hundreds of mortgages or loans. The Gaussian copula has no tail dependence (Proposition 17.11), so it treated simultaneous defaults as far rarer than they turned out to be in 2007–2008. Felix Salmon's 2009 article in Wired called it "the formula that killed Wall Street"; the model did not cause the crisis, but its users had relied on one number to describe a dependence that lived in the tails.

big picture — where copulas are used

Banks and insurers use copulas to combine the risks of business lines, assets or perils into one loss distribution, in capital models built for regulators. Hydrologists use them for floods, where the peak flow, the volume and the duration of a flood are dependent and the design of a dam depends on their joint extremes. In machine learning, Gaussian copulas generate synthetic tabular data (Section 5), model high-dimensional forecasts (Salinas et al., 2019, use a low-rank Gaussian copula process for thousands of related time series), estimate graphical models robustly (Section 6), and estimate mutual information, which equals the negative entropy of the copula (Ma and Sun, 2011). For many variables, vine copulas build a \(d\)-dimensional copula from \(d(d-1)/2\) bivariate ones arranged on a sequence of trees (Bedford and Cooke, 2002; Aas et al., 2009), which allows different tail behaviour for different pairs.

5 — Copulas in machine learning I: synthetic tabular data

Organisations often want to share data they cannot share: patient records, tax returns, customer histories. One response is to release synthetic data, records drawn from a model fitted to the real ones, which should be useful for analysis and model building while matching no real person. Sklar's theorem gives the simplest such model: keep each column's own distribution exactly, and glue the columns together with a Gaussian copula. This was the original model of the Synthetic Data Vault (Patki, Wedge and Veeramachaneni, 2016), and it is a standard baseline in later benchmarks of deep generative models for tables, such as CTGAN (Xu et al., 2019).

definition 17.12 — the Gaussian copula synthesiser

Given a table with \(n\) rows and \(d\) columns:

  1. replace each column by its pseudo-observations \(\hat U_{ij}\) and these by normal scores \(Z_{ij}=\Phi^{-1}(\hat U_{ij})\);
  2. estimate the correlation matrix \(\hat R\) of the normal scores;
  3. draw \(Z^*\sim\mathcal N(0,\hat R)\), set \(U^*_j=\Phi(Z^*_j)\), and map each \(U^*_j\) back through column \(j\)'s empirical quantile function (Theorem 17.6).

Each synthetic column then has the real column's distribution, and pairs of columns have, approximately, the real Spearman correlations.

The usual test of synthetic data is train on synthetic, test on real: fit a model to the synthetic records and measure its error on real held-out records. We use the California housing data of Lectures 3 and 8.

code — Gaussian-copula synthetic housing data, tested on real districts
# Synthetic tabular data from a Gaussian copula (the California housing data of Lectures 3 and 8): train on synthetic, test on real
from sklearn.model_selection import train_test_split
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.neighbors import NearestNeighbors
raw = pd.read_csv('data/housing.csv').dropna()
cal = pd.DataFrame({'MedInc': raw.median_income, 'HouseAge': raw.housing_median_age, 'AveRooms': raw.total_rooms / raw.households,
                    'AveBedrms': raw.total_bedrooms / raw.households, 'Population': raw.population, 'AveOccup': raw.population / raw.households,
                    'Latitude': raw.latitude, 'Longitude': raw.longitude, 'Value': raw.median_house_value / 1e5})
tr, te = train_test_split(cal, test_size=0.2, random_state=0)  # Lecture 3's split
R = tr.to_numpy(); n, d = R.shape; g = np.random.default_rng(0)
def inv_ecdf(Rcol, w):                                          # map uniforms back through the empirical quantiles of a column
    return np.quantile(Rcol, w, method='inverted_cdf')         # every synthetic value is a real value of that column
def gaussian_copula_synth(R, m, g):
    Z = Phinv(pseudo(R)); S = np.corrcoef(Z.T)                  # normal scores and their correlation: the fitted Gaussian copula
    W = Phi(g.multivariate_normal(np.zeros(R.shape[1]), S, m)) # draws from that copula
    return np.column_stack([inv_ecdf(R[:, j], W[:, j]) for j in range(R.shape[1])]), S
def independent_synth(R, m, g):                                 # the same margins, no dependence at all (the independence copula)
    return np.column_stack([inv_ecdf(R[:, j], g.random(m)) for j in range(R.shape[1])])
S_gc, Sig = gaussian_copula_synth(R, n, g); S_ind = independent_synth(R, n, g)
Xte, yte = te.to_numpy()[:, :-1], te.to_numpy()[:, -1]
def tstr(D):                                                     # train on (synthetic) D, test on the real test set
    m = HistGradientBoostingRegressor(max_iter=500, learning_rate=0.05, random_state=0).fit(D[:, :-1], D[:, -1])
    return np.sqrt(np.mean((m.predict(Xte) - yte) ** 2))
sp = lambda D: stats.spearmanr(D).statistic
nn = NearestNeighbors(n_neighbors=1).fit((R - R.mean(0)) / R.std(0))
dcr = lambda D: np.median(nn.kneighbors((D - R.mean(0)) / R.std(0))[0])        # median distance to the closest real training record
print(f"{n:,} real training districts, {len(te):,} real test districts; gradient boosting predicts the median house value ($100k)")
print("training data                      test RMSE   largest error in a Spearman correlation   median distance to closest real record")
print(f"{'real':34s} {tstr(R):9.3f} {0.0:24.3f} {dcr(te.to_numpy()):36.3f}  (real test districts)")
for name, D in [('Gaussian copula, empirical margins', S_gc), ('independent margins', S_ind)]:
    print(f"{name:34s} {tstr(D):9.3f} {np.abs(sp(D) - sp(R)).max():24.3f} {dcr(D):36.3f}")
k_lat, k_lon = cal.columns.get_loc('Latitude'), cal.columns.get_loc('Longitude')
def on_land(D, eps=0.05):                                       # share of synthetic districts within 0.05 degrees of some real district
    nb = NearestNeighbors(n_neighbors=1).fit(R[:, [k_lat, k_lon]]); return np.mean(nb.kneighbors(D[:, [k_lat, k_lon]])[0][:, 0] < eps)
print(f"share of districts within 0.05 degrees (about 5 km) of a real district: Gaussian copula {on_land(S_gc):.1%}, "
      f"independent {on_land(S_ind):.1%}, real test {on_land(te.to_numpy()):.1%}")
16,346 real training districts, 4,087 real test districts; gradient boosting predicts the median house value ($100k)
training data                      test RMSE   largest error in a Spearman correlation   median distance to closest real record
real                                   0.452                    0.000                                0.356  (real test districts)
Gaussian copula, empirical margins     0.672                    0.090                                0.540
independent margins                    1.159                    0.881                                0.955
share of districts within 0.05 degrees (about 5 km) of a real district: Gaussian copula 60.9%, independent 34.2%, real test 95.6%

Trained on real districts, gradient boosting has a test RMSE of 0.452 ($100,000s). Trained on the same number of synthetic districts from the Gaussian copula it gets 0.672, much better than the 1.159 of synthetic data with the right margins and no dependence, but clearly worse than real data. The pairwise rank correlations are kept (the largest error in a Spearman correlation is 0.090), so what is lost is dependence that a correlation matrix cannot express. Geography shows it plainly: only 60.9% of the Gaussian copula's districts lie within about 5 km of a real district, against 95.6% of real test districts, because California's coastline and its cities are not an ellipse in normal scores. Synthetic records are also no closer to real ones than real test records are (median distance to the closest training record 0.540 against 0.356), which is reassuring but is not a proof of privacy: an outlier copied from the training data would be invisible in a median.

Real against synthetic districts

Choose two columns of the housing data. The three panels show 1,500 real training districts, 1,500 synthetic districts from the Gaussian copula, and 1,500 from independent margins. Each column's distribution is the same in all three; only the copula differs. Longitude against latitude shows what a Gaussian copula cannot draw.

spot check 17.8

Why does the Gaussian copula synthesiser put many districts in the sea, although every synthetic latitude and every synthetic longitude is a real one?

Which part of the model, margins or copula, decides where the points fall in the plane?
Each coordinate is drawn from its real distribution, but the pair comes from a Gaussian copula, which can only tilt and stretch an ellipse in normal-score space. A coastline that bends, with cities in clusters, needs a richer copula, such as a vine, a mixture of copulas, or a deep generative model.

Two cautions apply to all synthetic data. Utility is measured on the questions one thought to ask: the TSTR error above says nothing about rare subgroups or about interactions that the regression model did not use. And distance to the closest real record is a weak privacy check; formal guarantees need methods designed for them, such as differentially private synthesisers, whose noise costs further utility.

6 — Copulas in machine learning II: the nonparanormal

Lecture 20a will estimate the conditional-independence graph of many variables with the graphical lasso, which assumes multivariate normal data. Real variables are skewed, bounded or heavy-tailed, and a monotone transformation that would make them normal is unknown. A Gaussian copula with arbitrary margins is exactly the model that is normal "up to unknown monotone transformations".

definition 17.13 — the nonparanormal (Liu, Lafferty and Wasserman, 2009)

A random vector \(X\in\mathbb R^d\) follows a nonparanormal distribution, \(X\sim\mathrm{NPN}(\Sigma,f)\), if there are strictly increasing functions \(f_1,\dots,f_d\) such that

$$\big(f_1(X_1),\dots,f_d(X_d)\big)\sim\mathcal N(0,\Sigma),\qquad\Sigma_{jj}=1.$$

Equivalently, \(X\) has a Gaussian copula with correlation matrix \(\Sigma\) and arbitrary continuous margins.

proposition 17.12 — the graph and the correlations from ranks

If \(X\sim\mathrm{NPN}(\Sigma,f)\), then \(X_j\) and \(X_k\) are conditionally independent given the other variables exactly when \((\Sigma^{-1})_{jk}=0\), and \(\Sigma\) is determined by the rank correlations:

$$\Sigma_{jk}=\sin\Big(\frac\pi2\,\tau_{jk}\Big)=2\sin\Big(\frac\pi6\,\rho_{S,jk}\Big).$$

Each \(f_j\) is invertible, so conditioning on \(X_k\) is the same as conditioning on \(f_k(X_k)\), and conditional independence of the \(X_j\) is conditional independence of the \(f_j(X_j)\). For a multivariate normal vector the latter holds exactly when the corresponding entry of the precision matrix is zero (Lecture 20a). The rank correlations of \(X\) are those of \(f(X)\) (Proposition 17.9), and for a normal vector Proposition 17.10 gives \(\tau=\tfrac2\pi\arcsin\Sigma_{jk}\) and \(\rho_S=\tfrac6\pi\arcsin(\Sigma_{jk}/2)\); inverting gives the formulas. \(\square\)

So the graphical lasso can be run on a correlation matrix estimated from ranks, either the correlation of the normal scores \(\Phi^{-1}(\hat U_{ij})\) or Kendall's \(\tau\) mapped through \(\sin(\pi\tau/2)\) (the SKEPTIC estimator of Liu et al., 2012). We test both on a known graph, observed through unknown distortions.

code — graphical lasso after monotone distortions: Pearson against the nonparanormal
# The nonparanormal (a Gaussian copula graphical model): recover a sparse graph after unknown monotone distortions of the variables
from sklearn.covariance import graphical_lasso
p, nobs = 30, 300; g = np.random.default_rng(1)
Theta = np.eye(p)                                               # a sparse precision matrix: a chain plus a few random edges
for j in range(p - 1): Theta[j, j + 1] = Theta[j + 1, j] = 0.4
for a, b in g.choice(p, (8, 2)):
    if abs(a - b) > 1: Theta[a, b] = Theta[b, a] = 0.3
Theta += np.eye(p) * max(0, 0.2 - np.linalg.eigvalsh(Theta).min()); Sigma = np.linalg.inv(Theta)
d_ = np.sqrt(np.diag(Sigma)); Sigma = Sigma / np.outer(d_, d_); true = (np.abs(np.linalg.inv(Sigma)) > 1e-8) & ~np.eye(p, dtype=bool)
Z = g.multivariate_normal(np.zeros(p), Sigma, nobs)
f = [lambda z: np.exp(2 * z), lambda z: z ** 5, lambda z: np.sign(z) * np.abs(z) ** 0.2, lambda z: 1 / (1 + np.exp(-4 * z)), lambda z: z]
Xd = np.column_stack([f[j % 5](Z[:, j]) for j in range(p)])     # what we observe: each variable bent by an unknown increasing function
def best_f1(S):                                                  # graphical lasso along a path of penalties; best F1 for the edges
    out = []
    for lam in np.geomspace(0.02, 0.6, 25):
        try: P = graphical_lasso(S, alpha=lam, max_iter=200)[1]
        except Exception: continue
        est = (np.abs(P) > 1e-6) & ~np.eye(p, dtype=bool); tp = np.sum(est & true) / 2
        out.append(2 * tp / ((est.sum() + true.sum()) / 2) if est.any() else 0.0)
    return max(out)
pearson = lambda X: np.corrcoef(X.T)
normal_scores = lambda X: np.corrcoef(Phinv(pseudo(X)).T)       # normal scores: a simplified form of Liu, Lafferty and Wasserman (2009)
def kendall_sin(X):                                              # SKEPTIC: Sigma_jk = sin(pi tau_jk / 2) (Liu et al., 2012)
    T = np.eye(X.shape[1])
    for a in range(X.shape[1]):
        for b in range(a + 1, X.shape[1]): T[a, b] = T[b, a] = np.sin(np.pi / 2 * stats.kendalltau(X[:, a], X[:, b])[0])
    w, V = np.linalg.eigh(T); return V @ np.diag(np.maximum(w, 1e-3)) @ V.T  # project to a positive definite matrix
print(f"{p} variables, {int(true.sum() / 2)} true edges, {nobs} observations; best F1 of the recovered edge set along the penalty path")
print(f"  graphical lasso on the Gaussian data before distortion (the ideal):   {best_f1(pearson(Z)):.3f}")
print(f"  graphical lasso on Pearson correlations of the distorted data:        {best_f1(pearson(Xd)):.3f}")
print(f"  nonparanormal, normal-score correlations:                             {best_f1(normal_scores(Xd)):.3f}")
print(f"  nonparanormal, Kendall's tau mapped by sin(pi tau / 2):                {best_f1(kendall_sin(Xd)):.3f}")
30 variables, 36 true edges, 300 observations; best F1 of the recovered edge set along the penalty path
  graphical lasso on the Gaussian data before distortion (the ideal):   0.958
  graphical lasso on Pearson correlations of the distorted data:        0.784
  nonparanormal, normal-score correlations:                             0.958
  nonparanormal, Kendall's tau mapped by sin(pi tau / 2):                0.958

On the undistorted Gaussian data the graphical lasso recovers the 36 edges with an F1 of 0.958 at its best penalty. After the distortions, Pearson correlations mislead it, and the best F1 drops to 0.784. Both nonparanormal estimators recover 0.958, the same as with the undistorted data. The invariance is exact: ranks are unchanged by increasing transformations, so the rank-based matrices are the same whether computed from the distorted or the undistorted data. That they also match the Pearson ideal to three decimals reflects how coarse F1 is on 36 edges. The comparison is generous to all methods, since each is scored at its own best penalty, and it is one simulated graph; but the conclusion, that a rank-based estimate costs almost nothing when the data are Gaussian and protects against distortions when they are not, matches Liu et al.'s theory and experiments.

Bayes — a copula likelihood that uses only ranks

A Bayesian treatment of a copula with unknown margins seems to need priors for the margins too. Hoff (2007) avoids them with the extended rank likelihood: the observed data imply only that the latent normal scores \(Z_{ij}\) are in the same order as the \(X_{ij}\) within each column, and the likelihood of the correlation matrix is the probability of that ordering. A Gibbs sampler alternates between drawing the latent scores, each constrained to an interval set by its neighbours in rank, and drawing the correlation matrix from its conditional posterior. The margins never need to be modelled, and the method works for ordinal and mixed discrete–continuous data, where ranks have ties; it is widely used for imputing missing values in surveys.

causal — independence lives in the copula

Whether \(X\) and \(Y\) are independent, or conditionally independent given \(Z\), is a property of the copula alone: independence means \(C=\Pi\). Constraint-based causal discovery, such as the PC algorithm of Lecture 20b, uses only such statements, so it can run on rank-based tests that are valid for any continuous margins; Harris and Drton (2013) proved that PC with Kendall or Spearman correlations recovers the Markov equivalence class of a nonparanormal model consistently, even in high dimensions. The methods that go beyond equivalence classes, such as LiNGAM or additive-noise models, need more than the copula: they rely on the shapes of the distributions, which the copula has deliberately thrown away.

7 — Summary, exercises, projects, reading

summary

Next: Lecture 18 turns from tables to signals, where the dependence of a series on its own past is the subject.

exercises
  1. Show that the Farlie–Gumbel–Morgenstern family \(C_\theta(u,v)=uv\big(1+\theta(1-u)(1-v)\big)\), \(|\theta|\le1\), consists of copulas with uniform margins, and compute its Spearman's rho and Kendall's tau. What do the answers say about the strength of dependence this family can represent?

    \(C_\theta(u,1)=u\) and \(C_\theta(1,v)=v\), and the density \(c=1+\theta(1-2u)(1-2v)\ge0\) when \(|\theta|\le1\). By Proposition 17.10,

    $$\rho_S=12\int_0^1\!\!\int_0^1\theta\,uv(1-u)(1-v)\,du\,dv=12\theta\Big(\frac16\Big)^2=\frac\theta3,$$

    and a similar calculation gives \(\tau=2\theta/9\). So \(|\rho_S|\le1/3\) and \(|\tau|\le2/9\): the family can only represent weak dependence, which is why it is used for perturbation arguments rather than for data.

  2. Let \((X,Y)\) have standard normal margins and a Clayton copula with \(\theta=2\). Is \((X,Y)\) bivariate normal? Explain using a property from this lecture.

    No. A bivariate normal distribution has a Gaussian copula, which by Sklar's theorem is the only copula giving that joint distribution with normal margins. The Clayton copula has lower tail dependence \(2^{-1/2}\approx0.71\) (Proposition 17.11) while every Gaussian copula with \(r\lt1\) has none. Normal margins are not enough for joint normality.

  3. Using Proposition 17.10, what Kendall's \(\tau\) corresponds to a Gaussian copula with \(r=0.9\), and what Spearman's \(\rho\)? Compare both with \(r\).

    \(\tau=\tfrac2\pi\arcsin0.9\approx0.713\) and \(\rho_S=\tfrac6\pi\arcsin0.45\approx0.891\). Spearman's rho is always close to the Pearson correlation under a Gaussian copula (within about 0.02), while Kendall's tau is noticeably smaller; the measures are on different scales and should not be compared across one another.

  4. For the \(t\) copula with \(r=0.6\), compute \(\lambda\) for \(\nu=4\) and \(\nu=30\) (you may use software for \(t_{\nu+1}\)). What happens as \(\nu\to\infty\)?

    \(\lambda=2t_5\big(-\sqrt{5\cdot0.4/1.6}\big)=2t_5(-1.118)\approx0.31\) for \(\nu=4\), and \(2t_{31}\big(-\sqrt{31\cdot0.4/1.6}\big)=2t_{31}(-2.78)\approx0.009\) for \(\nu=30\). As \(\nu\to\infty\) the \(t\) copula tends to the Gaussian copula and \(\lambda\to0\): heavy tails need small degrees of freedom.

  5. (Code.) Refit the \(t\) copula of Section 4 in rolling five-year windows from 1990 to 2026. Plot the fitted correlation and \(\nu\) over time. When do they change, and what does that imply for a risk model fitted once?

    Use the pseudo-observations within each window and the profile-likelihood fit of the code box. Expect dependence to rise in crises (for example 2000–2002, 2008–2009 and 2020), with the fitted correlation increasing and \(\nu\) tending to fall, so that joint extremes become more likely exactly when they matter; the test-period rise in Kendall's \(\tau\) in Section 4 points the same way. A model fitted on calm years under-predicts joint losses in the next crisis, as the test-period table showed; practice uses time-varying copulas or margins filtered by volatility models (GARCH) and refits frequently.

  6. (Code.) Replace the Gaussian copula in the synthesiser of Section 5 by a Gaussian mixture fitted to the normal scores (Lecture 17a), with 10 components. Does the share of districts on land, and the train-on-synthetic RMSE, improve?

    A Gaussian mixture in normal-score space is no longer a Gaussian copula, and it can bend around the coastline and concentrate on the cities; to keep the real margins, map each synthetic coordinate through the mixture's own marginal distribution function before the empirical quantile function. Expect the share of synthetic districts near real ones and the TSTR error both to move towards the real-data values, with more components helping until the mixture starts to memorise the training rows, which the distance-to-closest-record check will reveal.

student projects

Each project uses this lecture's methods on public data, and has a clear result.

1. Heavy rain and street flooding in New York City

question
How strongly are extreme daily rainfall in New York and the number of street-flooding complaints dependent in the upper tail, and how has that changed since 2010?
data
Daily precipitation at Central Park (NOAA GHCN-Daily station USW00094728) and 311 service requests with complaint type "Sewer" and descriptors "Street Flooding (SJ)" and "Sewer Backup (Use Comments) (SA)" from NYC Open Data, 2010 to the present.
method
Pseudo-observations on wet days; fit Gaussian, \(t\), Gumbel and survival Clayton copulas; estimate \(\lambda_U\) nonparametrically and from the fits; compare the first and second halves of the period.
outcome
An estimate of the probability of a top-1% flooding day given a top-1% rainfall day, with uncertainty, and a judgement of whether the city's exposure has changed.

2. Synthetic taxi trips: utility against privacy

question
How useful and how private are Gaussian-copula synthetic records of NYC taxi trips, compared with a deep generative model?
data
One month of NYC Taxi and Limousine Commission yellow-taxi trip records (pick-up and drop-off zones, times, distance, fare, tip).
method
Gaussian copula synthesiser (Definition 17.12) and CTGAN from the Synthetic Data Vault; train-on-synthetic-test-on-real for fare and tip prediction; distance to closest record and a membership-inference attack as privacy checks.
outcome
A utility–privacy comparison, and an account of which features (zone pairs, rush hours) each method fails to reproduce.

3. A vine copula for thirty industries

question
Does a vine copula, which allows different tail behaviour for each pair, forecast portfolio losses better than a single \(t\) copula?
data
The 30 daily industry portfolios of this lecture, 1990 to the present.
method
Filter each series by a GARCH(1,1) model; fit a \(t\) copula and an R-vine (the pyvinecopulib package) to the standardised residuals in rolling windows; backtest one-day 99% value-at-risk of an equal-weighted portfolio with Kupiec's test.
outcome
Exceedance rates and test results for both models, with the crisis years examined separately.

Reading and sources

Data. Industry portfolios: K. R. French, Data Library, "30 Industry Portfolios" (daily, value-weighted), mba.tuck.dartmouth.edu, Data Library, built from CRSP data. California housing: R. K. Pace, R. Barry, "Sparse spatial autoregressions", Statistics & Probability Letters 33(3), 291–297 (1997), in the copy from A. Géron's Hands-On Machine Learning repository, as in Lectures 3 and 8. The graph in Section 6 is simulated.

numbers in the text not printed by a code box are computed in verify_L17b.py, figures and widget data in figs_L17b.py; code-box outputs are pasted from actual runs (numpy, SciPy and scikit-learn on two CPU cores)