Chapter 6
Joint distributions

Lectures 15–16 · two unknowns at once: joint tables, independence, prediction from one quantity to another, covariance, correlation and sums

two things that move together may share a common cause

So far every model has had one unknown quantity: one count, one waiting time, one height. Most interesting questions involve two or more at once. How tall will a boy grow, given how tall his father is? If a geyser's eruption was long, how long until the next one? Does a second breathalyser reading tell us anything the first didn't? Are hair colour and eye colour connected? Each question asks how two random quantities behave together, and the tool for that is a joint distribution.

Joint distributions serve both of the course's goals. For prediction, they let us use one quantity to predict another: the father's height to predict the son's, the length of an eruption to predict the wait for the next. The best prediction turns out to be a conditional mean, and the way it shrinks towards the average is Galton's regression to the mean, seen at last through the mathematics. For inference, joint distributions describe how quantities are connected. Covariance and correlation measure the connection, and independence is the special case of no connection at all. But a connection in the data is not yet a cause, and this chapter keeps asking what produced the association.

Here is the route through the chapter.

Section 6.1 describes two random variables at once, with joint tables and joint densities.
Section 6.2 recovers each variable on its own (marginal distributions), and one given the other (conditional distributions).
Section 6.3 defines independence, the assumption behind most of the models so far.
Section 6.4 uses conditional expectation to predict one quantity from another.
Section 6.5 measures association with covariance and correlation.
Section 6.6 finds the mean and spread of sums and averages, which is the bridge to Chapters 7 to 9.

The calculus you need is double integrals over rectangles and simple regions (Section 0.4).

Think About It: Before You Start

A man is 6 ft 3 in tall, about 6 inches above the average for men. How tall would you predict his adult son will be: 6 inches above average, the average, or something in between? Write down a number before reading on. Example 6.9 answers it with Galton's data.

Something in between, and closer to the average than most people guess. Height is partly inherited, so the son of a tall father is likely to be tall. But the father's own height was partly luck: genes from his parents that he won't necessarily pass on, and a good childhood that his son may or may not share. The son gets only part of that advantage. In Galton's data, sons of fathers about 4 inches above average were only about 2 inches above average themselves. This is regression to the mean, and Section 6.4 shows exactly where it comes from.

Lecture 15 — Joint, marginal and conditional distributions

6.1 Joint distributions

Two variables in one table

When two random variables are discrete, their joint behaviour fits in a table: one row for each value of the first, one column for each value of the second, and the probability of each combination in the cell.

Definition: Joint PMF

The joint probability mass function of discrete random variables $X$ and $Y$ is $p (x, y) = Pr (X = x and Y = y) .$ It satisfies $p (x, y) \geq 0$ and $\sum_{x} \sum_{y} p (x,$ $y)$ $= 1$ . For any set $A$ of pairs, $Pr ((X, Y) \in A)$ $= \sum_{(x, y) \in A} p (x,$ $y)$ : add up the cells in $A$ .

The comma in $Pr (X = x,$ $Y$ $= y)$ means "and". This is the probability of the intersection of two events, exactly as in Chapter 3; the joint PMF just records all the intersections at once.

Example 6.1: Hair and Eye Colour ★ lecture example

In 1974 the statistician R. D. Snee reported the hair and eye colour of 592 statistics students at the University of Delaware. Pick one of these students at random, and let $H$ be their hair colour and $E$ their eye colour. (a) Find the joint PMF of $H$ and $E$ . (b) What is the probability that the student is blond with blue eyes? (c) What is the probability that their eyes are blue or green?

(a) From counts to probabilities. R's HairEyeColor table also splits the students by sex (a split added later, artificially, for teaching purposes); adding over sex recovers Snee's counts, and dividing by 592 gives the joint PMF:

he <- apply(HairEyeColor, c(1, 2), sum)   # add over the third dimension (sex)
he
round(he / sum(he), 3)                    # joint PMF

output

       Eye
Hair    Brown Blue Hazel Green
  Black    68   20    15     5
  Brown   119   84    54    29
  Red      26   17    14    14
  Blond     7   94    10    16
       Eye
Hair    Brown  Blue Hazel Green
  Black 0.115 0.034 0.025 0.008
  Brown 0.201 0.142 0.091 0.049
  Red   0.044 0.029 0.024 0.024
  Blond 0.012 0.159 0.017 0.027

hair \ eyes	brown	blue	hazel	green
black	0.115	0.034	0.025	0.008
brown	0.201	0.142	0.091	0.049
red	0.044	0.029	0.024	0.024
blond	0.012	0.159	0.017	0.027

All 16 entries are at least 0, and they add to 1 (up to rounding).

(b) Read one cell: $Pr (H = blond,$ $E$ $= blue)$ $= 94 / 592$ $\approx 0.159$ .

(c) Add the cells in the event. "Blue or green eyes" is the blue and green columns together, eight cells. The blue column holds $20$ $+ 84$ $+ 17$ $+ 94$ $= 215$ students and the green column $5$ $+ 29$ $+ 14$ $+ 16$ $= 64$ , so $Pr (E = blue or green)$ $= \frac{215 + 64}{592}$ $= \frac{279}{592}$ $\approx 0.471 .$

Picture. A tile plot shows the whole joint distribution at once: each tile's shade is its probability.

d <- as.data.frame(as.table(he / sum(he)))
names(d) <- c("hair", "eye", "p")
ggplot(d, aes(eye, hair, alpha = p)) + geom_tile(fill = "steelblue", colour = "white") +
  geom_text(aes(label = sprintf("%.3f", p)), alpha = 1) +
  scale_alpha(range = c(0.05, 0.6), guide = "none") +
  labs(x = "eye colour", y = "hair colour")

Blond hair goes with blue eyes far more often than with brown. The two variables seem connected, and Section 6.3 makes "connected" precise.

Data: R. D. Snee (1974), "Graphical display of two-way contingency tables", The American Statistician 28, 9–12; R's HairEyeColor.

Two continuous variables: joint densities

For two continuous random variables, probability is spread over the plane, and a joint density $f (x, y)$ describes how thickly. In Chapter 5, probability was an area under a curve; now it is a volume under a surface.

Definition: Joint Density

Continuous random variables $X$ and $Y$ have joint density $f (x, y)$ if $f (x, y) \geq 0$ , $\iint f (x,$ $y) d x d y$ $= 1$ , and for any region $A$ of the plane, $Pr ((X, Y) \in A)$ $= \iint_{A} f (x,$ $y) d x d y .$ For a rectangle this is $Pr (a \leq X \leq b,$ $c$ $\leq Y$ $\leq d)$ $= \int_{a}^{b} \int_{c}^{d} f (x,$ $y) d y d x$ .

As in one dimension, single points (and single lines) have probability zero, and $f (x, y)$ is not itself a probability: it is probability per unit area.

Example 6.2: The Shooting-Range Target ★ lecture example

A forensic team test-fires a weapon at a square target 1 metre on each side. Measured from the bottom-left corner, the horizontal and vertical positions of an impact, $X$ and $Y$ , have joint density $f (x,$ $y)$ $= \frac{6}{5} (x$ $+ y^{2}),$ $0$ $\leq x$ $\leq 1,$ $0$ $\leq y$ $\leq 1,$ and 0 elsewhere. (a) Check that $f$ is a valid joint density. (b) Where on the target are impacts most likely? (c) Find the probability that a shot lands in the top-right quarter-metre square, $X \geq 0.75$ and $Y \geq 0.75$ .

(a) Non-negative: $x + y^{2} \geq 0$ on the square. Total volume 1: integrate over $y$ first, holding $x$ fixed, then over $x$ : $\begin{matrix} \int_{0}^{1} \int_{0}^{1} \frac{6}{5} (x + y^{2}) d y d x & = \frac{6}{5} \int_{0}^{1} {[x y + \frac{y^{3}}{3}]}_{y = 0}^{y = 1} d x \\ = \frac{6}{5} \int_{0}^{1} (x + \frac{1}{3}) d x \\ = \frac{6}{5} (\frac{1}{2} + \frac{1}{3}) = \frac{6}{5} \cdot \frac{5}{6} = 1 . \end{matrix}$ $\int_{0}^{1} \int_{0}^{1} \frac{6}{5} (x$ $+ y^{2}) d y d x$ $= \frac{6}{5} \int_{0}^{1} {[x y + \frac{y^{3}}{3}]}_{y = 0}^{y = 1} d x$ $= \frac{6}{5} \int_{0}^{1} (x + \frac{1}{3}) d x$ $= \frac{6}{5} (\frac{1}{2} + \frac{1}{3})$ $= \frac{6}{5}$ $\cdot \frac{5}{6}$ $= 1 .$

(b) The density is largest where $x + y^{2}$ is largest: at the top-right corner $(1, 1)$ , where $f = 2.4$ , and smallest at the bottom-left corner, where $f = 0$ . Shots drift up and to the right, perhaps because of how the weapon was held.

(c) Integrate over the small square: $\begin{matrix} Pr (X \geq 0.75, Y \geq 0.75) \\ = \frac{6}{5} \int_{0.75}^{1} {[x y + \frac{y^{3}}{3}]}_{y = 0.75}^{y = 1} d x \\ = \frac{6}{5} \int_{0.75}^{1} (\frac{x}{4} + \frac{37}{192}) d x \\ = \frac{6}{5} (\frac{1 - {0.75}^{2}}{8} + \frac{37}{192} \cdot \frac{1}{4}) \approx 0.123 . \end{matrix}$ $Pr (X \geq 0.75,$ $Y$ $\geq 0.75)$ $= \frac{6}{5} \int_{0.75}^{1} {[x y + \frac{y^{3}}{3}]}_{y = 0.75}^{y = 1} d x$ $= \frac{6}{5} \int_{0.75}^{1} (\frac{x}{4} + \frac{37}{192}) d x$ $= \frac{6}{5} (\frac{1 - {0.75}^{2}}{8} + \frac{37}{192} \cdot \frac{1}{4})$ $\approx 0.123 .$ The top-right square is only $\frac{1}{16} = 6.25 %$ of the target's area but catches 12.3% of the shots. R can do the double integral numerically, as one integral inside another:

f <- function(x, y) 6/5 * (x + y^2)
inner <- function(x) sapply(x, function(xx) integrate(function(y) f(xx, y), 0.75, 1)$value)
integrate(inner, 0.75, 1)

output

0.1234375 with absolute error < 1.4e-15

Check Yourself

In Example 6.1, find the probability that a randomly chosen student has red hair and eyes that are not brown. The red-haired students' eye colours are 26 brown, 17 blue, 14 hazel and 14 green. Give 4 decimal places.

Pr =

A joint density has

f (0.5, 0.5) = 3

. What does that tell you?

Pr (X = 0.5,

Y

= 0.5)

= 3

, which is impossible, so

f

is invalid.Near

(0.5, 0.5)

, probability is concentrated at about 3 per unit area: a tiny square of area

0.01

there has probability about

0.03

Pr (X \leq 0.5,

Y

\leq 0.5)

= 3

Think About It: Whose Table Is It?

The hair and eye colour data came from 592 statistics students at one American university in the 1970s. Would you use this joint PMF to predict the eye colour of a blond person in Lagos, or in Stockholm? What kinds of question could these data answer?

No. The joint distribution describes this population: mostly young Americans of European descent, at one university, in one decade. Hair and eye colour vary enormously between populations, so in Lagos or Stockholm the table would look completely different. A probability model is always a model of some population, and predictions travel only as far as the population does. The data can answer questions about that group, and they can illustrate a real biological link: some of the same genes affect both hair and eye pigment. But to estimate how common blue eyes are anywhere else, you'd need data from there.

Key Takeaways

A joint PMF $p (x,$ $y)$ $= Pr (X = x,$ $Y$ $= y)$ lists the probability of every combination; probabilities of events are sums of cells.
A joint density $f (x, y)$ spreads probability over the plane; probabilities are volumes, $\iint_{A} f d x d y$ .
Integrate one variable at a time: hold $x$ fixed, integrate over $y$ , then integrate over $x$ .

6.2 Marginal and conditional distributions

Marginals: one variable at a time

The joint distribution contains everything about $X$ and $Y$ , including the distribution of each one on its own. To find $Pr (X = x)$ , add up every way it can happen: $X = x$ together with each possible value of $Y$ . In a table, that's a row sum, written in the margin, which is where the name comes from.

Definition: Marginal Distributions

If $X$ and $Y$ have joint PMF $p (x, y)$ , the marginal PMFs are $p_{X} (x)$ $= \sum_{y} p (x,$ $y),$ $p_{Y} (y)$ $= \sum_{x} p (x,$ $y) .$ If they have joint density $f (x, y)$ , the marginal densities are $f_{X} (x)$ $= \int_{- \infty}^{\infty} f (x,$ $y) d y,$ $f_{Y} (y)$ $= \int_{- \infty}^{\infty} f (x,$ $y) d x .$

This is the Law of Total Probability (Section 3.6) at work: the events ${Y = y}$ split the sample space into pieces, and $Pr (X = x)$ is the sum over the pieces.

Conditionals: one variable given the other

If we learn the value of $Y$ , our probabilities for $X$ change. Conditional probability (Section 3.5) says how: restrict attention to the row or column we now know we're in, and rescale it so it adds to 1.

Definition: Conditional Distributions

For discrete $X$ and $Y$ , the conditional PMF of $X$ given $Y = y$ (for any $y$ with $p_{Y} (y) > 0$ ) is $p_{X ∣ Y} (x ∣ y)$ $= Pr (X = x ∣ Y = y)$ $= \frac{p (x, y)}{p_{Y} (y)} .$ For continuous $X$ and $Y$ , the conditional density of $X$ given $Y = y$ (for $f_{Y} (y) > 0$ ) is $f_{X ∣ Y} (x ∣ y) = \frac{f (x, y)}{f_{Y} (y)} .$

For each fixed $y$ , the conditional distribution is a genuine distribution in $x$ : it's non-negative and adds (or integrates) to 1, because we divided by exactly the total of that slice. Geometrically, a conditional density is a slice of the joint density surface along the line $Y = y$ , rescaled to have area 1.

Example 6.3: What Colour Are a Blond Student's Eyes? ★ lecture example

Using Snee's students from Example 6.1: (a) find the marginal distributions of hair colour and of eye colour; (b) find the conditional distribution of eye colour given blond hair; (c) find $Pr (H = blond ∣ E = blue)$ . Compare (b) and (c) with the marginal probabilities.

(a) Row and column totals. The marginal of $H$ is the row sums, and the marginal of $E$ is the column sums, of the joint table:

round(rowSums(he) / sum(he), 3)          # marginal of hair colour
round(colSums(he) / sum(he), 3)          # marginal of eye colour

output

Black Brown   Red Blond 
0.182 0.483 0.120 0.215 
Brown  Blue Hazel Green 
0.372 0.363 0.157 0.108

hair	black	brown	red	blond
$p_{H}$	0.182	0.483	0.120	0.215

eyes	brown	blue	hazel	green
$p_{E}$	0.372	0.363	0.157	0.108

(b) Restrict to the blond row and rescale. There are 127 blond students, and their eye colours are 7 brown, 94 blue, 10 hazel and 16 green: $Pr (E = blue ∣ H = blond)$ $= \frac{94 / 592}{127 / 592}$ $= \frac{94}{127}$ $\approx 0.740,$ and similarly brown $7 / 127 \approx 0.055$ , hazel $0.079$ , green $0.126$ .

round(prop.table(he, 1), 3)              # each row divided by its total

output

       Eye
Hair    Brown  Blue Hazel Green
  Black 0.630 0.185 0.139 0.046
  Brown 0.416 0.294 0.189 0.101
  Red   0.366 0.239 0.197 0.197
  Blond 0.055 0.740 0.079 0.126

(c) Now restrict to a column. Of the 215 blue-eyed students, 94 are blond: $Pr (H = blond ∣ E = blue)$ $= \frac{94}{215}$ $\approx 0.437 .$

Comparing. Overall, 36% of students have blue eyes, but among blond students it's 74%. Overall 21.5% are blond, but among the blue-eyed it's 44%. Notice that $Pr (blue ∣ blond)$ $\neq Pr (blond ∣ blue)$ : conditioning in the two directions gives different answers, as Bayes' theorem (Section 3.6) warned. A bar chart of all four conditional distributions shows how much eye colour depends on hair colour:

d <- as.data.frame(as.table(prop.table(he, 1)))
names(d) <- c("hair", "eye", "p")
ggplot(d, aes(eye, p)) + geom_col(width = 0.6) + facet_wrap(~ hair, nrow = 1) +
  labs(x = "eye colour", y = "Pr(eye | hair)") +
  theme(axis.text.x = element_text(angle = 45, hjust = 1))

Joint, marginal and conditional, from one table

This is the hair and eye colour table from Example 6.1, as proportions of all 592 students. The margins give the marginal distributions: row sums for hair colour H, column sums for eye colour E. Choose a hair colour to condition on. The chart compares the conditional distribution of eye colour for that hair colour (orange) with the marginal distribution of eye colour (blue). If hair and eye colour were independent, the two sets of bars would match for every hair colour.

condition on hair colour

Example 6.4: The Target's Margins and Slices

For the shooting-range target of Example 6.2, with $f (x,$ $y)$ $= \frac{6}{5} (x$ $+ y^{2})$ on the unit square: (a) find the marginal densities of $X$ and $Y$ , and their means; (b) given that a shot landed at height $Y = 0.6$ , find the probability that it is in the left 40% of the target, $X \leq 0.4$ .

(a) Integrate out the other variable. For $X$ , integrate over $y$ from 0 to 1: $f_{X} (x)$ $= \int_{0}^{1} \frac{6}{5} (x$ $+ y^{2}) d y$ $= \frac{6}{5} (x + \frac{1}{3}),$ $0$ $\leq x$ $\leq 1 .$ For $Y$ , integrate over $x$ : $f_{Y} (y)$ $= \int_{0}^{1} \frac{6}{5} (x$ $+ y^{2}) d x$ $= \frac{6}{5} (\frac{1}{2} + y^{2}),$ $0$ $\leq y$ $\leq 1 .$ Then $E [X]$ $= \int_{0}^{1} x$ $\cdot \frac{6}{5} (x + \frac{1}{3}) d x$ $= \frac{6}{5} (\frac{1}{3} + \frac{1}{6})$ $= 0.6,$ and in the same way $E [Y]$ $= \frac{6}{5} (\frac{1}{4} + \frac{1}{4})$ $= 0.6$ . On average, shots land 0.6 m from the left and 0.6 m from the bottom.

(b) Slice and rescale. The conditional density of $X$ given $Y = 0.6$ is $f_{X ∣ Y} (x ∣ 0.6)$ $= \frac{\frac{6}{5} (x + 0.36)}{\frac{6}{5} (0.5 + 0.36)}$ $= \frac{x + 0.36}{0.86},$ $0$ $\leq x$ $\leq 1 .$ Then $\begin{matrix} Pr (X \leq 0.4 ∣ Y = 0.6) & = \int_{0}^{0.4} \frac{x + 0.36}{0.86} d x \\ = \frac{{0.4}^{2} / 2 + 0.36 \times 0.4}{0.86} = \frac{0.224}{0.86} \approx 0.260 . \end{matrix}$ $Pr (X \leq 0.4 ∣ Y = 0.6)$ $= \int_{0}^{0.4} \frac{x + 0.36}{0.86} d x$ $= \frac{{0.4}^{2} / 2 + 0.36 \times 0.4}{0.86}$ $= \frac{0.224}{0.86}$ $\approx 0.260 .$ The widget after Example 6.2 shows this slice: move its slider to $y = 0.6$ .

Example 6.5: The Dart's Horizontal Position ★ lecture example

Return to the dart of Example 5.2, landing uniformly at random on a round board of radius 1. Put the origin at the bullseye, so the landing point $(X, Y)$ has joint density $f (x, y) = \frac{1}{π}$ on the disc $x^{2} + y^{2} \leq 1$ . (a) Find the marginal density of $X$ , the dart's horizontal position. (b) What is the probability that the dart lands within half a foot of the vertical centre line, $| X | \leq 0.5$ ? (c) Given that the dart's height is $Y = y$ , what is the conditional distribution of $X$ ?

(a) Integrate out $y$ , minding the boundary. For a fixed $x$ between $- 1$ and 1, the disc extends vertically from $y$ $= - \sqrt{1 - x^{2}}$ to $y = \sqrt{1 - x^{2}}$ , and the density is 0 outside it. So $f_{X} (x)$ $= \int_{- \sqrt{1 - x^{2}}}^{\sqrt{1 - x^{2}}} \frac{1}{π} d y$ $= \frac{2}{π} \sqrt{1 - x^{2}},$ $- 1$ $\leq x$ $\leq 1 .$ This is a semicircle, scaled to have area 1. The dart's horizontal position is not uniform: positions near the centre line are more likely, because the board is tallest there.

(b) Integrate the marginal (R does it numerically): $Pr (| X | \leq 0.5)$ $= \int_{- 0.5}^{0.5} \frac{2}{π} \sqrt{1 - x^{2}} d x$ $\approx 0.609 .$

integrate(function(x) 2 / pi * sqrt(1 - x^2), -0.5, 0.5)

output

0.6089978 with absolute error < 6.8e-15

The vertical strip $| x | \leq 0.5$ is half the board's width but holds 61% of its area.

(c) Slice at height $y$ . For $| y | < 1$ , the marginal of $Y$ is $f_{Y} (y)$ $= \frac{2}{π} \sqrt{1 - y^{2}}$ by the same argument, so $f_{X ∣ Y} (x ∣ y)$ $= \frac{1 / π}{\frac{2}{π} \sqrt{1 - y^{2}}}$ $= \frac{1}{2 \sqrt{1 - y^{2}}},$ $| x |$ $\leq \sqrt{1 - y^{2}} .$ That's a uniform distribution on the chord of the disc at height $y$ . Learning the height changes what's possible for $x$ : at $y = 0.9$ , the dart must be within $\sqrt{0.19} \approx 0.44$ of the centre line.

Simulation check. Throw 20,000 random darts (keeping points in the square that fall inside the disc) and compare the histogram of $X$ with the semicircle:

set.seed(6)
x <- runif(20000, -1, 1); y <- runif(20000, -1, 1)
keep <- x^2 + y^2 <= 1
darts <- data.frame(x = x[keep], y = y[keep])
ggplot(darts, aes(x)) +
  geom_histogram(aes(y = after_stat(density)), binwidth = 0.1, boundary = 0, colour = "white") +
  stat_function(fun = function(x) 2 / pi * sqrt(1 - x^2), linewidth = 1) +
  labs(x = "horizontal position X", y = "density")
mean(abs(darts$x) <= 0.5)

output

[1] 0.6137667

Check Yourself

Using Snee's data, find $Pr (E = brown ∣ H = black)$ . There are 108 black-haired students, 68 of them with brown eyes. Give 3 decimal places.

Pr =

A joint PMF table has rows for

X

and columns for

Y

. To get the conditional distribution of

Y

given

X = 2

, you should: divide every cell in the table by

Pr (X = 2)

.divide the

Y

column totals by

Pr (X = 2)

.take the

X = 2

row and divide each cell in it by the row total.

Think About It: Who Is in the Denominator?

A news story says: "70% of people arrested for burglary in our city were unemployed." A politician concludes that unemployed people are very likely to commit burglary. Write both statements as conditional probabilities. Which one does the story actually report, and what extra information would you need to get the other?

The story reports $Pr (unemployed ∣ arrested for burglary)$ $= 0.7$ . The politician's claim is about $Pr (burglary ∣ unemployed)$ , which conditions in the other direction. To get it you'd need Bayes' theorem: the number of unemployed people in the city and the number of burglaries. If 20,000 people are unemployed and 300 burglary arrests were made, then about 210 arrests involved unemployed people, roughly 1% of the unemployed population. There is a second problem, too: arrests aren't burglaries. If police patrol poorer neighbourhoods more heavily, unemployed people may be more likely to be arrested for the same act. Which population is in the denominator, and how people got into it, decides what a conditional probability means.

Key Takeaways

Marginals: sum or integrate out the other variable, $p_{X} (x)$ $= \sum_{y} p (x,$ $y)$ or $f_{X} (x)$ $= \int f (x,$ $y) d y$ .
Conditionals: slice and rescale, $p_{X ∣ Y} (x ∣ y)$ $= p (x,$ $y) / p_{Y} (y)$ ; the same with densities.
Mind the boundaries of the region where $f > 0$ : they set the limits of integration.
$Pr (A ∣ B)$ $\neq Pr (B ∣ A)$ in general: always check which variable is being conditioned on.

6.3 Independence

Two events are independent if learning that one happened doesn't change the probability of the other (Section 3.5). Two random variables are independent if learning the value of one doesn't change the distribution of the other: every conditional distribution equals the marginal. That's equivalent to the joint distribution factorizing.

Definition: Independent Random Variables

Random variables $X$ and $Y$ are independent if $p (x, y) = p_{X} (x) p_{Y} (y) for all x, y$ (discrete), or $f (x, y) = f_{X} (x) f_{Y} (y) for all x, y$ (continuous). Equivalently, $Pr (X \in A,$ $Y$ $\in B)$ $= Pr (X \in A) Pr (Y \in B)$ for all sets $A$ and $B$ . More generally, $X_{1}, \dots, X_{n}$ are (mutually) independent if their joint PMF or density is the product of all $n$ marginals.

To show dependence, one counterexample is enough: a single pair $(x, y)$ where the joint isn't the product of the marginals. To show independence, the factorization must hold everywhere.

Hair and eye colour are dependent. $Pr (blond,$ $blue)$ $= 0.159$ , but $Pr (blond) Pr (blue)$ $= 0.215$ $\times 0.363$ $= 0.078$ . Blond, blue-eyed students are about twice as common as independence would predict.
The target is dependent. $f (x,$ $y)$ $= \frac{6}{5} (x$ $+ y^{2})$ isn't a product of a function of $x$ and a function of $y$ : for example, $f (0, 0) = 0$ , but $f_{X} (0) f_{Y} (0)$ $= \frac{6}{5}$ $\cdot \frac{1}{3}$ $\times \frac{6}{5}$ $\cdot \frac{1}{2}$ $= 0.24$ .
The dart is dependent. Near the corner $(0.9, 0.9)$ , outside the disc, the joint density is 0, but both marginals are positive at 0.9. A quick test: if the region where $f > 0$ isn't a rectangle (with sides parallel to the axes), the variables can't be independent, because knowing one restricts the range of the other.

Watch Out: Pairwise Isn't Mutual

Mutual independence of $X_{1}, \dots, X_{n}$ implies that every pair is independent, but not the other way round. Toss two fair coins, let $X$ and $Y$ be 1 for heads and 0 for tails, and let $Z = 1$ if the coins match. Each pair of $X$ , $Y$ , $Z$ is independent (check that $Pr (X = 1,$ $Z$ $= 1)$ $= \frac{1}{4}$ $= Pr (X = 1) Pr (Z = 1)$ , for instance). But all three together are not: once you know $X$ and $Y$ , you know $Z$ for certain. When a model says "independent", it almost always means mutually independent, which is the stronger assumption.

Example 6.6: Who Wins the Match? ★ lecture example

In English Premier League football, home teams have scored about 1.5 goals per game on average and away teams about 1.2. A simple model says the home goals $X \sim Pois (1.5)$ and the away goals $Y \sim Pois (1.2)$ are independent. Under this model: (a) find the probability that the match ends 1–1; (b) find the probabilities of a home win, a draw and an away win. (c) Is independence a reasonable assumption?

(a) Independence turns the joint PMF into a product: $Pr (X = 1,$ $Y$ $= 1)$ $= Pr (X = 1) Pr (Y = 1)$ $= 1.5 e^{- 1.5}$ $\times 1.2 e^{- 1.2}$ $\approx 0.335$ $\times 0.361$ $\approx 0.121 .$

(b) Add the cells in each event. A home win is every cell with $x > y$ , a draw is the diagonal $x = y$ , and an away win is $x < y$ . There are infinitely many cells, but those with more than 10 goals for either side have negligible probability, so an 11-by-11 table is plenty:

P <- outer(dpois(0:10, 1.5), dpois(0:10, 1.2))   # joint PMF: rows = home goals, columns = away
round(P[1:4, 1:4], 3)                            # the top-left corner of the table
c(home = sum(P[lower.tri(P)]), draw = sum(diag(P)), away = sum(P[upper.tri(P)]))

output

      [,1]  [,2]  [,3]  [,4]
[1,] 0.067 0.081 0.048 0.019
[2,] 0.101 0.121 0.073 0.029
[3,] 0.076 0.091 0.054 0.022
[4,] 0.038 0.045 0.027 0.011
     home      draw      away 
0.4414646 0.2548168 0.3037180

The model gives a home win 44%, a draw 25% and an away win 30%. That's close to the real long-run figures in the Premier League, where home teams have won a little under half of all matches and about a quarter have been drawn.

d <- expand.grid(home = 0:5, away = 0:5)
d$p <- dpois(d$home, 1.5) * dpois(d$away, 1.2)
d$result <- ifelse(d$home > d$away, "home win", ifelse(d$home == d$away, "draw", "away win"))
ggplot(d, aes(away, home, fill = result, alpha = p)) + geom_tile(colour = "white") +
  geom_text(aes(label = sprintf("%.3f", p)), alpha = 1, size = 3.8) +
  scale_fill_manual(values = c("orange", "grey60", "steelblue")) +
  scale_alpha(range = c(0.1, 0.6), guide = "none") +
  labs(x = "away goals", y = "home goals", fill = NULL)

(c) Critical thinking. Independence is a modelling assumption, and in football it's only roughly true. The score affects how teams play: a team that's losing attacks, which can produce goals at both ends, and teams that are level late on may settle for a draw. Statisticians who model football (most famously Mark Dixon and Stuart Coles, in 1997) found that independent Poisson models slightly underpredict low-scoring draws such as 0–0 and 1–1, and they added a small correction for this dependence. A model can be useful even when an assumption is a little wrong, as long as we check its predictions against data.

Here's a continuous example, where independence makes a double integral easy.

Example 6.7: Police or Fire First? ★ lecture example

A small town's emergency dispatcher receives police calls at random at 4 per hour and fire calls at random at 1 per hour, independently. Let $X$ be the time (in hours) until the first police call and $Y$ the time until the first fire call, so $X \sim Exp (4)$ and $Y \sim Exp (1)$ are independent (Section 5.4). (a) What is the probability that a police call comes first? (b) What is the distribution of the time $T$ $= min (X,$ $Y)$ until the first call of either kind? (c) What is the probability of a quiet half hour with no calls?

(a) Integrate the joint density over the region $x < y$ . By independence, $f (x,$ $y)$ $= 4 e^{- 4 x}$ $\cdot e^{- y}$ for $x, y > 0$ . For each $x$ , $y$ runs from $x$ to $\infty$ : $\begin{matrix} Pr (X < Y) & = \int_{0}^{\infty} \int_{x}^{\infty} 4 e^{- 4 x} e^{- y} d y d x = \int_{0}^{\infty} 4 e^{- 4 x} e^{- x} d x \\ = \int_{0}^{\infty} 4 e^{- 5 x} d x = \frac{4}{5} = 0.8 . \end{matrix}$ $Pr (X < Y)$ $= \int_{0}^{\infty} \int_{x}^{\infty} 4 e^{- 4 x} e^{- y} d y d x$ $= \int_{0}^{\infty} 4 e^{- 4 x} e^{- x} d x$ $= \int_{0}^{\infty} 4 e^{- 5 x} d x$ $= \frac{4}{5}$ $= 0.8 .$ In general, with rates $λ_{1}$ and $λ_{2}$ , $Pr (X < Y)$ $= \frac{λ_{1}}{λ_{1} + λ_{2}}$ : each type of call wins in proportion to its rate.

(b) The minimum is bigger than $t$ exactly when both are. By independence, $Pr (T > t)$ $= Pr (X > t) Pr (Y > t)$ $= e^{- 4 t} e^{- t}$ $= e^{- 5 t},$ so $T \sim Exp (5)$ : the time to the first call of either kind is exponential with the combined rate of 5 per hour, mean $\frac{1}{5}$ hour = 12 minutes.

set.seed(6)
x <- rexp(100000, rate = 4); y <- rexp(100000, rate = 1)
mean(x < y)                 # close to 0.8
mean(pmin(x, y)) * 60       # close to 12 minutes
mean(pmin(x, y) > 0.5)      # close to 0.082

output

[1] 0.79983
[1] 11.95805
[1] 0.08009

This is a basic fact about Poisson processes: two independent streams of random events merge into a single stream whose rate is the sum, and each event belongs to the first stream with probability $\frac{λ_{1}}{λ_{1} + λ_{2}}$ .

The calculation in (a) is a volume over a region. The contours below show the joint density $4 e^{- 4 x} e^{- y}$ , highest near the origin; the shaded region above the diagonal, where $x < y$ , holds 80% of the probability, because police calls are four times as frequent and so usually come first.

g <- expand.grid(x = seq(0, 1.2, length.out = 121), y = seq(0, 1.2, length.out = 121))
g$f <- 4 * exp(-4 * g$x) * exp(-g$y)
ggplot(g, aes(x, y)) +
  geom_polygon(data = data.frame(x = c(0, 1.2, 0), y = c(0, 1.2, 1.2)), fill = "orange", alpha = 0.25) +
  geom_contour(aes(z = f), colour = "steelblue", bins = 10) +
  geom_abline(slope = 1, intercept = 0, linetype = "dashed") +
  annotate("text", x = 0.74, y = 1.12, label = "police first (x < y): 0.8") +
  annotate("text", x = 0.9, y = 0.3, label = "fire first: 0.2") +
  coord_equal() + labs(x = "x: time to first police call (hours)", y = "y: time to first fire call (hours)")

Check Yourself

X

and

Y

have joint density

f (x,

y)

= 6 e^{- 2 x - 3 y}

for

x, y > 0

. Are they independent? Yes:

f (x,

y)

= 2 e^{- 2 x}

\cdot 3 e^{- 3 y}

, a product of two densities.No: the exponent contains both

x

and

y

.We can't tell without computing the covariance.

In Example 6.7, what is the probability that the first call is a fire call?

Pr =

Think About It: Independence Is a Claim About the World

Each of these models assumes independence. For each, say whether you believe it and what could create dependence: (a) the results of 10 coin tosses; (b) whether each of 10 houses on a street floods this year; (c) whether each of the 12 jurors on a jury votes "guilty"; (d) the DNA profiles of two people chosen at random from a city.

(a) Believable: coins have no memory and don't influence each other. (b) No: the houses share a river, a storm and a drainage system, so one flooding makes the others much more likely to flood. Insurers who treat neighbouring houses as independent badly underestimate the chance of many claims at once. (c) No: jurors deliberate together and influence each other, and they all saw the same evidence. (d) Roughly, for strangers, but not for relatives or people from small, closely related communities, who share DNA from common ancestors. That matters when a DNA match is reported as "1 in a billion": the calculation multiplies probabilities across many genetic markers, assuming independence, and it can be badly wrong for a suspect's brother. Sally Clark's case (Section 3.6) is the most famous example of a false independence assumption in court.

Key Takeaways

$X$ and $Y$ are independent when the joint PMF or density factorizes into the marginals; then every conditional distribution equals the marginal.
One cell where joint $\neq$ product of marginals proves dependence. A non-rectangular region where $f > 0$ means dependence.
With independence, joint probabilities are products, which makes many calculations easy. But independence is an assumption about the mechanism, and it should be checked.
Mutual independence implies pairwise independence, not the reverse.

Lecture 16 — Prediction, covariance and sums

6.4 Expected values and prediction

Expected values of functions of two variables

Expected values work just as in Chapters 4 and 5: weight each possible value by its probability. The only change is that the possible values are now pairs.

Theorem: Expected Value of a Function of X and Y

For any function $g$ , $E [g (X, Y)]$ $= \sum_{x} \sum_{y} g (x,$ $y) p (x,$ $y)$ in the discrete case, and $E [g (X, Y)]$ $= \iint g (x,$ $y) f (x,$ $y) d x d y$ in the continuous case (provided the sum or integral of $| g |$ times the probability is finite).

Two consequences are used constantly.

Theorem: Linearity, and Products of Independent Variables

For any random variables $X$ and $Y$ and constants $a$ , $b$ , $c$ : $E [a X$ $+ b Y$ $+ c]$ $= a E [X]$ $+ b E [Y]$ $+ c$ . No independence is needed.
If $X$ and $Y$ are independent, then $E [X Y]$ $= E [X] E [Y]$ , and more generally $E [g (X) h (Y)]$ $= E [g (X)] E [h (Y)]$ .

We give the discrete case; for densities, replace sums by integrals.

(1) With $g (x,$ $y)$ $= a x$ $+ b y$ $+ c$ , $\begin{matrix} E [a X + b Y + c] & = a \sum_{x} \sum_{y} x p (x, y) + b \sum_{x} \sum_{y} y p (x, y) + c \sum_{x} \sum_{y} p (x, y) \\ = a \sum_{x} x p_{X} (x) + b \sum_{y} y p_{Y} (y) + c = a E [X] + b E [Y] + c, \end{matrix}$ $E [a X$ $+ b Y$ $+ c]$ $= a \sum_{x} \sum_{y} x p (x,$ $y)$ $+ b \sum_{x} \sum_{y} y p (x,$ $y)$ $+ c \sum_{x} \sum_{y} p (x,$ $y)$ $= a \sum_{x} x p_{X} (x)$ $+ b \sum_{y} y p_{Y} (y)$ $+ c$ $= a E [X]$ $+ b E [Y]$ $+ c,$ because summing $p (x, y)$ over $y$ gives the marginal $p_{X} (x)$ , and summing over $x$ gives $p_{Y} (y)$ .

(2) If $p (x,$ $y)$ $= p_{X} (x) p_{Y} (y)$ , the double sum splits into a product of two single sums: $\begin{matrix} E [g (X) h (Y)] & = \sum_{x} g (x) p_{X} (x) \sum_{y} h (y) p_{Y} (y) \\ = \sum_{x} g (x) p_{X} (x) E [h (Y)] \\ = E [g (X)] E [h (Y)] . ◼ \end{matrix}$ $E [g (X) h (Y)]$ $= \sum_{x} g (x) p_{X} (x) \sum_{y} h (y) p_{Y} (y)$ $= \sum_{x} g (x) p_{X} (x) E [h (Y)]$ $= E [g (X)] E [h (Y)] . ◼$ ∎

For the football model of Example 6.6, linearity gives the expected total goals, $E [X$ $+ Y]$ $= 1.5$ $+ 1.2$ $= 2.7$ , and the expected home winning margin, $E [X$ $- Y]$ $= 0.3$ , without touching the joint table. Linearity would hold even if the goals were dependent. The product rule does need independence: it gives $E [X Y]$ $= 1.5$ $\times 1.2$ $= 1.8$ here, but for dependent variables $E [X Y]$ can be anything.

Conditional expectation: the best prediction

If we know $X = x$ , the distribution of $Y$ is its conditional distribution given $X = x$ , and that distribution has its own mean.

Definition: Conditional Expectation

The conditional expectation of $Y$ given $X = x$ is the mean of the conditional distribution: $E [Y ∣ X = x]$ $= \sum_{y} y p_{Y ∣ X} (y ∣ x)$ $or$ $\int y f_{Y ∣ X} (y ∣ x) d y .$ As $x$ varies, $E [Y ∣ X = x]$ is a function of $x$ , called the regression function of $Y$ on $X$ .

Why does it matter? Because it's the best possible prediction of $Y$ from $X$ , in the same sense that the mean was the best single guess in Chapters 2 and 4.

Theorem: The Conditional Mean Is the Best Predictor

Suppose we predict $Y$ by a number $c$ , after learning that $X = x$ , and the loss is the squared error ${(Y - c)}^{2}$ . The expected loss, $E [{(Y - c)}^{2} ∣ X = x]$ , is smallest when $c$ $= E [Y ∣ X = x]$ , and the smallest value is the conditional variance $Var [Y ∣ X = x]$ .

Write $m$ $= E [Y ∣ X = x]$ . Everything below is computed with the conditional distribution of $Y$ given $X = x$ , which is a genuine distribution, so the usual rules apply. Add and subtract $m$ : $\begin{matrix} E [{(Y - c)}^{2} ∣ X = x] & = E [{(Y - m)}^{2} ∣ X = x] + 2 (m - c) E [Y - m ∣ X = x] + {(m - c)}^{2} \\ = Var [Y ∣ X = x] + 0 + {(m - c)}^{2}, \end{matrix}$ $E [{(Y - c)}^{2} ∣ X = x]$ $= E [{(Y - m)}^{2} ∣ X = x]$ $+ 2 (m$ $- c) E [Y$ $- m ∣ X = x]$ $+ {(m - c)}^{2}$ $= Var [Y ∣ X = x]$ $+ 0$ $+ {(m - c)}^{2},$ since $E [Y$ $- m ∣ X = x]$ $= m$ $- m$ $= 0$ . Only the last term depends on $c$ , and it is smallest (zero) at $c = m$ . $◼$ ∎

So prediction from one variable to another is a matter of conditional means, and the conditional variance measures how far off the prediction will typically be. If we know nothing about $X$ , the best guess is the overall mean $E [Y]$ , and the two are linked by a useful rule.

Theorem: Law of Total Expectation

$E [Y]$ $= \sum_{x} E [Y ∣ X = x] p_{X} (x)$ $or$ $E [Y]$ $= \int E [Y ∣ X = x] f_{X} (x) d x .$ The overall mean is the weighted average of the conditional means, weighted by how likely each $x$ is.

In the discrete case, $\sum_{x} E [Y ∣ X = x] p_{X} (x)$ $= \sum_{x} \sum_{y} y \frac{p (x, y)}{p_{X} (x)} p_{X} (x)$ $= \sum_{y} y \sum_{x} p (x,$ $y)$ $= \sum_{y} y p_{Y} (y)$ $= E [Y] . ◼$ ∎

Example 6.8: When Will Old Faithful Erupt Again?

Of the 272 eruptions in R's faithful data (Example 2.1), 97 lasted 3 minutes or less and 175 lasted longer. After a short eruption the average wait for the next was 54.5 minutes (SD 5.8); after a long one it was 80.0 minutes (SD 6.0). (a) Find the overall average wait using the law of total expectation. (b) A park ranger posts a prediction of the next eruption. How much better is a prediction that uses the length of the last eruption than one that ignores it?

(a) Let $L$ indicate a long eruption, with $Pr (L)$ $= 175 / 272$ . Then $\begin{matrix} E [W] & = E [W ∣ short] Pr (short) + E [W ∣ long] Pr (long) \\ = 54.5 \times \frac{97}{272} + 80.0 \times \frac{175}{272} \approx 70.9 minutes, \end{matrix}$ $E [W]$ $= E [W ∣ short] Pr (short)$ $+ E [W ∣ long] Pr (long)$ $= 54.5$ $\times \frac{97}{272}$ $+ 80.0$ $\times \frac{175}{272}$ $\approx 70.9 minutes,$ which is the overall mean of all 272 waits.

(b) Compare typical prediction errors. Ignoring the last eruption, the best prediction is 70.9 minutes, and the typical error is the overall SD of the waits, about 13.6 minutes. Using it, the ranger predicts 54.5 or 80.0 minutes, and the typical error is the conditional SD, about 6 minutes. Knowing one variable has cut the prediction error by more than half.

long <- faithful$eruptions > 3
tapply(faithful$waiting, long, mean)          # conditional means
tapply(faithful$waiting, long, sd)            # conditional SDs
sd(faithful$waiting)                          # unconditional SD

output

   FALSE     TRUE 
54.49485 79.98857 
   FALSE     TRUE 
5.840098 5.994239 
[1] 13.59497

This is what Yellowstone's rangers actually do: the posted prediction depends on the duration of the previous eruption. A prediction based on a conditional mean is only as good as the conditional spread around it, which is why the posted time comes with a window of about 10 minutes either way.

Example 6.9: Tall Fathers, Slightly Less Tall Sons ★ lecture example

Galton's data (Example 5.1) include the heights of 481 sons and of their fathers. (a) Estimate $E [son’s height ∣ father’s height]$ by averaging sons' heights for fathers in bands of height. (b) Fathers at least 72 inches tall averaged 73.1 inches. How tall were their sons on average? (c) What about the sons of fathers at most 66 inches tall? Explain the pattern.

(a) Group the fathers and average the sons in each group. Each group mean estimates the conditional mean for fathers of that height:

galton <- read.csv("https://raw.githubusercontent.com/vincentarelbundock/Rdatasets/master/csv/HistData/GaltonFamilies.csv")
sons <- subset(galton, gender == "male")
band <- cut(sons$father, breaks = c(61, 64, 66, 68, 70, 72, 74, 79))
data.frame(n = as.vector(table(band)), son_mean = round(tapply(sons$childHeight, band, mean), 1))

output

          n son_mean
(61,64]   8     67.4
(64,66]  55     67.6
(66,68] 100     68.2
(68,70] 179     69.4
(70,72] 104     70.3
(72,74]  26     71.2
(74,79]   9     70.9

means <- data.frame(father = tapply(sons$father, band, mean),
                    son = tapply(sons$childHeight, band, mean))
ggplot(sons, aes(father, childHeight)) +
  geom_jitter(width = 0.1, height = 0.1, alpha = 0.4, colour = "steelblue") +
  geom_abline(slope = 1, intercept = 0, linetype = "dashed") +
  geom_smooth(method = "lm", se = FALSE, colour = "black", linewidth = 0.8) +
  geom_point(data = means, aes(father, son), colour = "orange", size = 4) +
  labs(x = "father's height (inches)", y = "son's height (inches)")

output

`geom_smooth()` using formula = 'y ~ x'

The conditional means rise with the father's height, roughly along a straight line, $\hat{son}$ $= 38.4$ $+ 0.45$ $\times father$ (Section 6.5 explains where the slope comes from).

(b) and (c) Compare distances from the average. The average father was 69.1 inches and the average son 69.2.

fathers	number	fathers' mean	sons' mean
72 in or more	55	73.1 (+4.0)	71.0 (+1.8)
66 in or less	63	65.3 (−3.8)	67.6 (−1.7)

The numbers in brackets are distances from the average father or son.

tall <- sons$father >= 72; short <- sons$father <= 66
c(mean(sons$father[tall]), mean(sons$childHeight[tall]))
c(mean(sons$father[short]), mean(sons$childHeight[short]))

output

[1] 73.11091 71.02545
[1] 65.31746 67.56825

Sons of tall fathers were tall, but less than half as far above average as their fathers; sons of short fathers were short, but closer to average. This is regression to the mean (Section 4.7), and Galton described it for human height using exactly these data, in 1886 (he had first seen it in the seeds of sweet peas). It doesn't mean that heights are converging and everyone will one day be the same height: the spread of the sons' heights (SD 2.6) is at least as large as the fathers' (2.3). The tall sons of the next generation come partly from fathers of average height who were lucky in the genes and environment they passed on.

Prediction. For a father 75 inches tall (6 ft 3 in, the man in "before you start"), the fitted line (with unrounded coefficients, $38.363$ $+ 0.4465$ $\times 75$ ) predicts a son of about 71.9 inches: about 2.7 inches above average, not 6.

Caveat. Brothers share a father, so the 481 sons aren't independent observations; the estimates are fine, but Chapter 8's margins of error would need adjusting.

Predicting a son's height from his father's

Each dot is one of Galton's 481 father–son pairs (jittered slightly so that tied heights don't hide each other). Move the slider to pick fathers of a given height (within half an inch either way). The highlighted sons show the conditional distribution of a son's height given his father's height, and the large orange dot is its mean: the best prediction under squared-error loss. Compare it with the dashed line, where a son would be exactly as tall as his father, and with the solid fitted line.

father's height = inches

Causal Inference: Why Do Sons Regress?

Regression to the mean isn't a force pulling people towards the average. It happens whenever two quantities are imperfectly related, and the DAG shows why for heights. A father's height doesn't cause his son's height. Both are caused by shared influences (genes, and the family's food, money and health) and by influences that aren't shared (the half of the son's genes that come from his mother, and his own childhood). Only the shared part carries over. A father who is very tall was probably helped by both the shared and the unshared influences, and the son inherits only the first.

Fathers' and sons' heights are correlated through shared causes, not because one height causes the other.

Galton himself read the regression in genetic terms, and used it to argue that "ability" was inherited (the history box after Example 5.6). The DAG shows why that inference was too quick: the data can't separate genes from family resources, because both arrows point the same way. Disentangling them took twin and adoption studies, decades later.

Check Yourself

Using the fitted line with unrounded coefficients, $\hat{son}$ $= 38.363$ $+ 0.4465$ $\times father$ , predict the height of a son whose father is 75 inches tall. Give one decimal place.

predicted height =

In Example 6.8, the overall SD of waiting times is 13.6 minutes and the SD within each group is about 6. What does that tell you? The conditional means must be wrong, since the SDs should be equal.Knowing the last eruption's length removes most of the uncertainty about the next wait.The waiting times are independent of the eruption lengths.

Think About It: The Sophomore Slump

In many sports, the "Rookie of the Year" usually does worse in their second season, and athletes on the cover of Sports Illustrated often have a bad run afterwards (the "cover jinx"). Explain both using regression to the mean. Then design a comparison that would tell you whether the cover really causes bad performance.

An athlete becomes Rookie of the Year, or makes the cover, after an exceptional period: real skill plus some good luck. Next season the skill remains, but the luck is fresh, and on average it's ordinary, so performance usually drops towards the athlete's true level. No jinx is needed. To test for a real cover effect, compare cover athletes with athletes who had equally exceptional runs but didn't make the cover (perhaps because another story took the slot that week). If both groups decline by the same amount, the decline is regression to the mean. This is the same logic as the police hot-spots example in Section 4.7: when units are selected because they were extreme, you need a comparison group selected the same way.

Key Takeaways

$E [g (X, Y)]$ $= \sum \sum g p$ or $\iint g f$ . Linearity $E [a X$ $+ b Y]$ $= a E [X]$ $+ b E [Y]$ always holds; $E [X Y]$ $= E [X] E [Y]$ needs independence.
The conditional mean $E [Y ∣ X = x]$ is the best prediction of $Y$ from $X$ under squared loss; the conditional variance is its expected squared error.
Law of total expectation: $E [Y]$ is the average of the conditional means, weighted by $p_{X}$ .
When two quantities are imperfectly related, predictions regress to the mean.

6.5 Covariance and correlation

Covariance

We often want a single number saying how strongly two quantities move together. The idea: if large values of $X$ go with large values of $Y$ , then $X - μ_{X}$ and $Y - μ_{Y}$ tend to have the same sign, and their product tends to be positive. If large $X$ goes with small $Y$ , the product tends to be negative.

Definition: Covariance

The covariance of $X$ and $Y$ , with means $μ_{X}$ and $μ_{Y}$ , is $Cov (X,$ $Y)$ $= E [(X - μ_{X}) (Y - μ_{Y})] .$ The shortcut formula is $Cov (X,$ $Y)$ $= E [X Y]$ $- μ_{X} μ_{Y}$ .

Expand the product and use linearity: $E [X Y$ $- μ_{Y} X$ $- μ_{X} Y$ $+ μ_{X} μ_{Y}]$ $= E [X Y]$ $- μ_{Y} μ_{X}$ $- μ_{X} μ_{Y}$ $+ μ_{X} μ_{Y}$ $= E [X Y]$ $- μ_{X} μ_{Y}$ . $◼$ ∎

Theorem: Properties of Covariance

For random variables $X$ , $Y$ , $Z$ and constants $a$ , $b$ , $c$ , $d$ :

$Cov (X,$ $Y)$ $= Cov (Y,$ $X)$ , and $Cov (X,$ $X)$ $= Var [X]$ .
$Cov (a X$ $+ b,$ $c Y$ $+ d)$ $= a c Cov (X,$ $Y)$ : adding constants changes nothing, and rescaling rescales.
$Cov (X$ $+ Y,$ $Z)$ $= Cov (X,$ $Z)$ $+ Cov (Y,$ $Z)$ .
If $X$ and $Y$ are independent, $Cov (X, Y) = 0$ .

(1) is clear from the definition. For (2), $a X + b$ has mean $a μ_{X} + b$ , so its deviation from its mean is $a (X - μ_{X})$ ; similarly for $c Y + d$ ; and $E [a (X - μ_{X}) \cdot c (Y - μ_{Y})]$ $= a c Cov (X,$ $Y)$ . For (3), the deviation of $X + Y$ from its mean is $(X$ $- μ_{X})$ $+ (Y$ $- μ_{Y})$ ; multiply by $(Z - μ_{Z})$ and use linearity. For (4), independence gives $E [X Y] = μ_{X} μ_{Y}$ (Section 6.4), so the shortcut formula gives 0. $◼$ ∎

Example 6.10: The First Die and the Total ★ lecture example

Roll two fair dice. Let $X$ be the first die and $S = X + Y$ the total. (a) Find $Cov (X, S)$ . (b) Explain why it's positive.

(a) Use the properties rather than a 36-cell table. By property 3, then properties 1 and 4 (the two dice are independent), $Cov (X,$ $S)$ $= Cov (X,$ $X$ $+ Y)$ $= Cov (X,$ $X)$ $+ Cov (X,$ $Y)$ $= Var [X]$ $+ 0$ $= \frac{35}{12}$ $\approx 2.92,$ using $Var [X]$ $= E [X^{2}]$ $- μ^{2}$ $= \frac{91}{6}$ $- \frac{49}{4}$ $= \frac{35}{12}$ for a fair die.

Check by brute force over the 36 equally likely outcomes:

x <- rep(1:6, times = 6); y <- rep(1:6, each = 6); s <- x + y
mean((x - mean(x)) * (s - mean(s)))      # population covariance over the 36 outcomes
35 / 12

output

[1] 2.916667
[1] 2.916667

(We use mean rather than R's cov(), because cov() divides by $n - 1$ , as the sample variance does, and here we want the exact probability-weighted average.)

(b) A high first roll pushes the total up, so the two tend to be above (or below) average together. The covariance is exactly the first die's own variance, because that's the part of the total the first die contributes; the second die adds variation that is unrelated to $X$ .

Correlation

Covariance has awkward units (inches times inches, minutes times minutes), and its size depends on the units: measure heights in centimetres instead of inches and the covariance grows by a factor of ${2.54}^{2}$ . Dividing by both standard deviations gives a pure number.

Definition: Correlation

The correlation of $X$ and $Y$ (with positive standard deviations) is $ρ = Corr (X, Y) = \frac{Cov (X, Y)}{SD [X] SD [Y]} .$ It is the covariance of the standardized variables $\frac{X - μ_{X}}{σ_{X}}$ and $\frac{Y - μ_{Y}}{σ_{Y}}$ .

Theorem: Correlation Is Between −1 and 1

$- 1 \leq ρ \leq 1$ , and $ρ = \pm 1$ exactly when $Y$ is a linear function of $X$ , $Y = a + b X$ (with $b > 0$ for $+ 1$ and $b < 0$ for $- 1$ ), with probability 1.

Let $Z_{X}$ and $Z_{Y}$ be the standardized variables, each with variance 1. A variance is never negative, so, using the rules of Section 6.6 (which follow from the covariance properties above), $0$ $\leq Var [Z_{X}$ $+ Z_{Y}]$ $= 1$ $+ 1$ $+ 2 ρ,$ $0$ $\leq Var [Z_{X}$ $- Z_{Y}]$ $= 1$ $+ 1$ $- 2 ρ .$ The first gives $ρ \geq - 1$ and the second $ρ \leq 1$ . If $ρ = 1$ , then $Var [Z_{X}$ $- Z_{Y}]$ $= 0$ , so $Z_{Y} = Z_{X}$ with probability 1 (a random variable with zero variance is constant, and this one has mean 0), which makes $Y$ a linear function of $X$ with positive slope. The case $ρ = - 1$ is the same with $Z_{X} + Z_{Y}$ . Conversely, if $Y = a + b X$ , then $Cov (X,$ $Y)$ $= b σ_{X}^{2}$ and $σ_{Y} = | b | σ_{X}$ , so $ρ$ $= b / | b |$ $= \pm 1$ . $◼$ ∎

For the dice, $SD [X]$ $= \sqrt{35 / 12}$ and $SD [S]$ $= \sqrt{35 / 6}$ (the variance of the sum of two independent dice, Section 6.6), so $ρ$ $= \frac{35 / 12}{\sqrt{35 / 12} \sqrt{35 / 6}}$ $= \frac{1}{\sqrt{2}}$ $\approx 0.71 .$

What do different correlations look like? Each panel shows 200 simulated points with the stated correlation. The last panel is a perfect curved relationship with correlation 0: correlation sees only straight-line trends.

set.seed(6)
make <- function(rho) { x <- rnorm(200); y <- rho * x + sqrt(1 - rho^2) * rnorm(200)
  data.frame(panel = sprintf("r = %+.1f", rho), x, y) }
d <- do.call(rbind, lapply(c(-0.9, -0.5, 0, 0.5, 0.9), make))
x <- seq(-2, 2, length.out = 200)
d <- rbind(d, data.frame(panel = "curve, r = 0", x, y = x^2 - mean(x^2)))
d$panel <- factor(d$panel, levels = unique(d$panel))
ggplot(d, aes(x, y)) + geom_point(alpha = 0.5, size = 1, colour = "steelblue") +
  facet_wrap(~ panel, ncol = 3, scales = "free") + labs(x = "X", y = "Y") +
  theme(axis.text = element_blank())

Correlation and the best straight-line prediction

Correlation measures how well a straight line predicts one variable from another. That's the precise sense in which it measures linear association.

Theorem: The Best Linear Predictor

Among all predictions of $Y$ of the form $a + b X$ , the expected squared error $E [{(Y - a - b X)}^{2}]$ is smallest when $b$ $= \frac{Cov (X, Y)}{Var [X]}$ $= ρ \frac{σ_{Y}}{σ_{X}},$ $a$ $= μ_{Y}$ $- b μ_{X},$ and the smallest expected squared error is $σ_{Y}^{2} (1 - ρ^{2})$ . In standard units the line is simply ${\hat{z}}_{Y} = ρ z_{X}$ .

For any $b$ , the expected squared error is $Var [Y$ $- b X]$ $+ (E [Y - b X]$ $- a)^{2}$ (as in the proof of Section 6.4's theorem, with $Y - b X$ in place of $Y$ ). So the best $a$ is $E [Y$ $- b X]$ $= μ_{Y}$ $- b μ_{X}$ , leaving $Var [Y$ $- b X]$ $= σ_{Y}^{2}$ $- 2 b Cov (X,$ $Y)$ $+ b^{2} σ_{X}^{2} .$ This quadratic in $b$ is smallest at $b$ $= Cov (X,$ $Y) / σ_{X}^{2}$ , where it equals $σ_{Y}^{2}$ $- Cov {(X, Y)}^{2} / σ_{X}^{2}$ $= σ_{Y}^{2} (1$ $- ρ^{2})$ . $◼$ ∎

Three things follow.

Regression to the mean, exactly. In standard units, the predicted $Y$ is $ρ$ times the standardized $X$ . Unless $| ρ | = 1$ , the prediction is closer to the mean (in SDs) than the predictor was. Galton's fathers and sons have $ρ \approx 0.39$ , so a father 2 SDs above average predicts a son only about 0.8 SD above average.
$ρ^{2}$ is the fraction of variance explained. Using $X$ cuts the prediction error's variance from $σ_{Y}^{2}$ to $σ_{Y}^{2} (1 - ρ^{2})$ . With $ρ = 0.39$ , that's only a 15% reduction; with $ρ = 0.9$ , it's 81%.
If the true conditional mean is a straight line (as it is for a bivariate Gaussian, Section 6.7), the best linear predictor is the conditional mean, the best predictor of all.

For data $(x_{1},$ $y_{1}),$ $\dots,$ $(x_{n},$ $y_{n})$ , replace each population quantity by its sample version. The sample correlation is $r$ $= \frac{1}{n - 1} \sum_{i = 1}^{n} (\frac{x_{i} - \overline{x}}{s_{x}}) (\frac{y_{i} - \overline{y}}{s_{y}}),$ the average product of standardized values (in R, cor(x, y)), and the fitted line has slope $r s_{y} / s_{x}$ and passes through $(\overline{x}, \overline{y})$ . This is the least-squares line (in R, lm(y ~ x)), which Gauss used for Ceres (Section 5.3).

Example 6.11: Predicting Old Faithful from a Straight Line ★ lecture example

For the 272 eruptions in faithful, the eruption lengths have mean 3.49 minutes and SD 1.14, and the waits until the next eruption have mean 70.9 minutes and SD 13.6. The sample correlation is $r = 0.90$ . (a) Find the least-squares line for predicting the wait from the eruption length. (b) Predict the wait after a 2-minute eruption and after a 4.5-minute eruption. (c) How big is the typical prediction error?

(a) Slope and intercept from the summaries: $b$ $= r \frac{s_{y}}{s_{x}}$ $= 0.90$ $\times \frac{13.6}{1.14}$ $\approx 10.7,$ $a$ $= \overline{y}$ $- b \overline{x}$ $\approx 33.5 .$ These use R's unrounded summaries ( $r = 0.9008$ , $\overline{x} = 3.488$ , $\overline{y} = 70.897$ ), which give slope 10.73 and intercept 33.47, as coef(fit) below shows. (The rounded summaries printed above give 10.74 and 33.4, which is close enough for prediction.) So $\hat{wait}$ $= 33.5$ $+ 10.7$ $\times length$ : each extra minute of eruption adds about 11 minutes to the expected wait.

(b) After 2 minutes: $33.5$ $+ 10.7$ $\times 2$ $\approx 54.9$ minutes. After 4.5 minutes: $33.47$ $+ 10.73$ $\times 4.5$ $\approx 81.8$ minutes.

(c) The typical error is $s_{y} \sqrt{1 - r^{2}}$ $= 13.6 \sqrt{1 - 0.81}$ $\approx 5.9$ minutes, the same size as the within-group SDs in Example 6.8.

cor(faithful$eruptions, faithful$waiting)
fit <- lm(waiting ~ eruptions, data = faithful)
coef(fit)
predict(fit, data.frame(eruptions = c(2, 4.5)))
sd(faithful$waiting) * sqrt(1 - cor(faithful$eruptions, faithful$waiting)^2)

output

[1] 0.9008112
(Intercept)   eruptions 
   33.47440    10.72964 
       1        2 
54.93368 81.75778 
[1] 5.903088

ggplot(faithful, aes(eruptions, waiting)) +
  geom_point(alpha = 0.6, colour = "steelblue") +
  geom_smooth(method = "lm", se = FALSE, colour = "black") +
  labs(x = "eruption length (minutes)", y = "wait until next (min)")

output

`geom_smooth()` using formula = 'y ~ x'

Critical thinking. The line fits well, but the picture shows two clusters, and the line passes through the gap between them, where there are few eruptions. Predictions for a 3-minute eruption rest on very little data. A correlation of 0.90 summarizes the relationship in one number, and the scatterplot shows what the number leaves out.

Uncorrelated isn't independent

Independence implies zero covariance, but zero covariance does not imply independence. Covariance only detects linear tendencies: whether $Y$ tends to rise or fall as $X$ rises. A relationship can be strong and still have no linear trend.

Example 6.12: The Dart, Once More ★ lecture example

For the dart of Example 6.5, uniform on the disc $x^{2} + y^{2} \leq 1$ : (a) show that $Cov (X, Y) = 0$ ; (b) explain why $X$ and $Y$ are nonetheless dependent.

(a) Symmetry. The disc and the density are unchanged if we replace $x$ by $- x$ . So $X$ is symmetric about 0, giving $E [X] = 0$ , and the product $X Y$ is equally likely to be $x y$ and $- x y$ , giving $E [X Y] = 0$ . Hence $Cov (X,$ $Y)$ $= 0$ $- 0$ $\times E [Y]$ $= 0$ .

(b) Example 6.5 showed that the conditional distribution of $X$ given $Y = y$ is uniform on $[- \sqrt{1 - y^{2}}, \sqrt{1 - y^{2}}]$ , which depends on $y$ . Knowing that the dart is near the top tells you it must be near the centre line. The spread of $X$ depends on $Y$ , though its average doesn't, and covariance only sees the average.

cov(darts$x, darts$y)                                  # close to 0
c(sd(darts$x[abs(darts$y) < 0.2]), sd(darts$x[abs(darts$y) > 0.8]))   # spread depends on y

output

[1] -0.001576221
[1] 0.5720822 0.2759127

What a correlation looks like

Choose the true correlation ρ and draw a sample. Each point contributes the product of its standardized x and y to the covariance: points in the shaded quadrants (both above or both below average) push it up, the others push it down. Then try the other shapes: a perfect curve with zero correlation, and a single outlier that manufactures a correlation out of nothing.

data true ρ = sample size n =

Watch Out: What a Correlation Can't Tell You

It only measures straight-line association. $ρ = 0$ is consistent with a strong curved relationship, like the dart's, or $Y = X^{2}$ with $X$ symmetric about 0.
It is sensitive to outliers. A single extreme point can create or destroy a correlation (try it in the widget).
It depends on the range. Among NBA players, height and points scored are only weakly correlated, because everyone is tall. Restricting the range of $X$ weakens correlations.
It says nothing about cause. See the causal box below.

Causal Inference: Storks Deliver Babies (R = 0.62)

In 2000 the physicist Robert Matthews published a short paper with a deliberately absurd finding. Across 17 European countries, the number of breeding pairs of storks was strongly correlated with the number of births: $r = 0.62$ , too strong to be a fluke by the standards of Chapter 9. Do storks bring babies?

The explanation is a common cause. Large countries, such as Poland and Turkey, have more land for storks and more people to have babies. Land area is a confounder (Chapter 1), and in the DAG there is no arrow between storks and births at all:

A common cause makes storks and births move together, with no arrow between them.

Comparing countries of similar size, or comparing births per person with storks per square kilometre, would most likely weaken or remove the correlation. Real cases are less obvious. Ice-cream sales and drownings rise together (summer causes both); cities with more police have more crime (more crime leads cities to hire police, which is reverse causation); and selecting a group on a collider can create correlation from nothing (Section 3.5). A correlation is a fact about the data. Whether it reflects a cause is a question about the mechanism that produced the data, and the DAG is where to start.

Source: R. Matthews (2000), "Storks deliver babies (p = 0.008)", Teaching Statistics 22, 36–38.

History: Correlation Was Born in a Police Bureau

In the 1880s the Paris police adopted a system of criminal identification invented by Alphonse Bertillon, a clerk at the Prefecture of Police. Each arrested person was measured: head length, head breadth, the length of the left middle finger, the left foot, the forearm and more. The measurements were filed so that a repeat offender using a false name could be identified. Francis Galton studied the system in 1888 and noticed a flaw. The measurements weren't independent: a person with long forearms usually has long feet and is tall. Recording several related measurements added less information than Bertillon assumed, just as a second correlated breathalyser reading does (Section 6.6).

Galton realised that he needed a measure of how closely two quantities were "co-related", and in his 1888 paper "Co-relations and their measurement, chiefly from anthropometric data" he introduced the correlation coefficient, building on his work on heredity (the history box after Example 5.6). Karl Pearson gave it the formula we use today in 1896, which is why it is often called the Pearson correlation. Bertillon's system was soon overtaken by fingerprints, another method Galton helped to establish, in his book Finger Prints (1892).

The word itself has drawn criticism. In Bernoulli's Fallacy (2021), Aubrey Clayton includes "correlation" in his list of terms that statistics would do better without. One reason the word is risky: it sounds like a general measure of connection between two things, when the number measures only how well a straight line fits, and it invites causal stories it can't support. These notes keep the standard word, because every paper and program uses it (R's cor()). But read every correlation as "how well does a straight line predict one from the other, in these data?", and nothing more.

Check Yourself

Two variables have correlation

r = - 0.8

. Which statement is correct? Increasing

X

will cause

Y

to decrease.80% of the points lie on a straight line.A straight line with negative slope predicts

Y

from

X

fairly well: it removes

{0.8}^{2} = 64 %

of the variance of

Y

For Galton's fathers and sons, the covariance of heights is 2.37 square inches, the fathers' SD is 2.31 inches and the sons' SD is 2.62 inches. Find the correlation, to 2 decimal places.

r =

Think About It: More Police, More Crime?

Across US cities, the number of police officers per person is positively correlated with the crime rate. A commentator concludes that police cause crime. Give two other explanations, and say what kind of data would help you choose between them.

Reverse causation: cities with high crime hire more police, so crime causes police rather than the other way round. Confounding: larger, denser cities have both more crime and more police per person, and so may poorer cities with more crime and more federal policing grants. To find the effect of police on crime you need variation in police numbers that isn't caused by crime: for example, comparing the same city before and after a hiring increase that was decided for unrelated reasons (an election promise or a budget formula), with similar cities that didn't change as a comparison. Economists have used exactly these kinds of "natural experiments", and most find that more police lead to less crime, the opposite of the raw correlation.

Key Takeaways

$Cov (X,$ $Y)$ $= E [(X - μ_{X}) (Y - μ_{Y})]$ $= E [X Y]$ $- μ_{X} μ_{Y}$ . Independent ⇒ $Cov = 0$ , but not conversely.
$ρ$ $= Cov (X,$ $Y) / (σ_{X} σ_{Y})$ lies in $[- 1, 1]$ , and $\pm 1$ only for exact straight lines.
The best straight-line prediction is ${\hat{z}}_{Y} = ρ z_{X}$ : slope $ρ σ_{Y} / σ_{X}$ , error variance $σ_{Y}^{2} (1 - ρ^{2})$ . This is regression to the mean.
Correlation measures linear association in the data; it is not evidence of cause by itself.

6.6 Sums and averages

Totals and averages are everywhere: the total of an insurer's claims, the average of repeated measurements, the number of successes in $n$ trials. Their means are easy. Their spreads depend on covariances.

Theorem: Variance of a Sum

For any random variables $X$ and $Y$ and constants $a$ and $b$ , $Var [a X$ $+ b Y]$ $= a^{2} Var [X]$ $+ b^{2} Var [Y]$ $+ 2 a b Cov (X,$ $Y) .$ In particular, if $X$ and $Y$ are independent, $Var [X$ $+ Y]$ $= Var [X]$ $+ Var [Y]$ and also $Var [X$ $- Y]$ $= Var [X]$ $+ Var [Y]$ . More generally, $Var [\sum_{i = 1}^{n} X_{i}]$ $= \sum_{i = 1}^{n} Var [X_{i}]$ $+ 2 \sum_{i < j} Cov (X_{i},$ $X_{j}) .$

$Var [W]$ $= Cov (W,$ $W)$ , so by the properties of covariance (Section 6.5), $\begin{matrix} Var [a X + b Y] & = Cov (a X + b Y, a X + b Y) \\ = a^{2} Cov (X, X) + a b Cov (X, Y) + b a Cov (Y, X) + b^{2} Cov (Y, Y) \\ = a^{2} Var [X] + 2 a b Cov (X, Y) + b^{2} Var [Y] . \end{matrix}$ $Var [a X$ $+ b Y]$ $= Cov (a X$ $+ b Y,$ $a X$ $+ b Y)$ $= a^{2} Cov (X,$ $X)$ $+ a b Cov (X,$ $Y)$ $+ b a Cov (Y,$ $X)$ $+ b^{2} Cov (Y,$ $Y)$ $= a^{2} Var [X]$ $+ 2 a b Cov (X,$ $Y)$ $+ b^{2} Var [Y] .$ If $X$ and $Y$ are independent the covariance is 0; with $a = 1$ and $b = - 1$ this gives $Var [X$ $- Y]$ $= Var [X]$ $+ Var [Y]$ . The $n$ -variable version expands $Cov (\sum X_{i}, \sum X_{j})$ in the same way. $◼$ ∎

This is the result promised in Section 4.3. Two consequences will carry the rest of the course.

Subtracting adds uncertainty. The difference between two independent measurements is more variable than either.
Averaging reduces it. If $X_{1}, \dots, X_{n}$ are independent, each with variance $σ^{2}$ , their average $\overline{X}$ has $Var [\overline{X}]$ $= \frac{1}{n^{2}}$ $\cdot n σ^{2}$ $= \frac{σ^{2}}{n},$ $SD [\overline{X}]$ $= \frac{σ}{\sqrt{n}} .$ Four times as much data halves the spread of an average. Chapter 7 builds on this.

Sums also change shape. The total of one die is flat; of two dice, a triangle; of five dice, already close to a bell. Each new independent piece smooths the distribution further, a first glimpse of the Central Limit Theorem of Chapter 7.

total_pmf <- function(k) { p <- rep(1/6, 6); out <- 1
  for (i in 1:k) out <- convolve(out, rev(p), type = "open")
  data.frame(dice = paste(k, ifelse(k == 1, "die", "dice")), total = k:(6 * k), p = out) }
d <- do.call(rbind, lapply(c(1, 2, 3, 5), total_pmf))
d$dice <- factor(d$dice, levels = unique(d$dice))
ggplot(d, aes(total, p)) + geom_col(width = 0.8) + facet_wrap(~ dice, nrow = 1, scales = "free") +
  scale_x_continuous(breaks = function(l) pretty(l, n = 3)) +
  labs(x = "total", y = "probability")

Example 6.13: Is a Second Breathalyser Reading Worth Taking? ★ lecture example

A breathalyser's readings scatter around the true alcohol level with SD $σ = 0.005$ g/dL (Example 5.7). Two readings are taken and averaged. (a) If the two errors are independent, what is the SD of the average? (b) What if the errors have correlation 0.5, because the same device and the same breath sample affect both? (c) For a driver whose true level is 0.075, find the probability that the average is 0.08 or more in each case, assuming Gaussian errors.

(a) The average is $\frac{1}{2} R_{1} + \frac{1}{2} R_{2}$ . With independent errors, $Var [\frac{1}{2} R_{1} + \frac{1}{2} R_{2}]$ $= \frac{1}{4} σ^{2}$ $+ \frac{1}{4} σ^{2}$ $= \frac{σ^{2}}{2},$ $SD$ $= \frac{0.005}{\sqrt{2}}$ $\approx 0.0035 .$

(b) With correlation $ρ$ , the covariance is $ρ σ^{2}$ , so $\begin{matrix} Var [\frac{1}{2} R_{1} + \frac{1}{2} R_{2}] & = \frac{1}{4} σ^{2} + \frac{1}{4} σ^{2} + 2 \cdot \frac{1}{2} \cdot \frac{1}{2} ρ σ^{2} \\ = σ^{2} \frac{1 + ρ}{2}, \end{matrix}$ $Var [\frac{1}{2} R_{1} + \frac{1}{2} R_{2}]$ $= \frac{1}{4} σ^{2}$ $+ \frac{1}{4} σ^{2}$ $+ 2$ $\cdot \frac{1}{2}$ $\cdot \frac{1}{2} ρ σ^{2}$ $= σ^{2} \frac{1 + ρ}{2},$ and with $ρ = 0.5$ , $SD$ $= 0.005 \sqrt{0.75}$ $\approx 0.0043$ . The second reading helps much less.

independent: $Pr (Z \geq \frac{0.005}{0.005 / \sqrt{2}})$ $= Pr (Z \geq 1.41)$ $\approx 0.079$ ;
correlated: $Pr (Z \geq \frac{0.005}{0.005 \sqrt{0.75}})$ $= Pr (Z \geq 1.15)$ $\approx 0.124$ .

sd_avg <- 0.005 * sqrt((1 + c(0, 0.5)) / 2)
sd_avg
1 - pnorm(0.08, mean = 0.075, sd = sd_avg)

output

[1] 0.003535534 0.004330127
[1] 0.0786496 0.1241065

A single reading wrongly puts this driver over the limit 16% of the time (Example 5.7). Averaging two independent readings cuts that to 8%; with correlated errors, only to 12%. (Example 5.7's rule, using the lower of two readings, is stricter still: 2.5% with independent errors.) Laboratories check for correlated errors by using two different methods, such as a breath test and a blood test, rather than two readings from the same device.

Does a second reading help?

A breathalyser reading has standard deviation σ = 0.005 g/dL around the true level. We take two readings and average them. If the two readings' errors are independent, averaging cuts the standard deviation by a factor of √2. But if the errors are correlated (the same device, the same breath, the same minute), the second reading carries less new information. Move the correlation ρ between the two errors and watch the simulated pairs (left) and the standard deviation of the average (right).

correlation ρ =

Example 6.14: Where the Binomial Variance Comes from

In Section 4.4 we stated that $X \sim Bin (n, p)$ has variance $n p (1 - p)$ . Prove it.

Write $X$ $= I_{1}$ $+ \dots$ $+ I_{n}$ , where $I_{k} = 1$ if trial $k$ is a success and 0 otherwise. Each indicator has $E [I_{k}] = p$ and $E [I_{k}^{2}] = p$ (since $I_{k}^{2} = I_{k}$ ), so $Var [I_{k}]$ $= p$ $- p^{2}$ $= p (1$ $- p)$ . The trials are independent, so all the covariances are 0, and $Var [X]$ $= \sum_{k = 1}^{n} Var [I_{k}]$ $= n p (1$ $- p) .$ By linearity, $E [X] = n p$ , with no calculation over the binomial PMF at all. Writing a count as a sum of indicators is one of the most useful tricks in probability.

Example 6.15: How Many Shared Birthdays? ★ lecture example

In a class of 30 students, how many pairs of students share a birthday, on average? (Ignore February 29 and assume all 365 days are equally likely.)

Count with indicators. There are $(\binom{30}{2}) = 435$ pairs of students. For each pair, let $I = 1$ if the two share a birthday, which has probability $\frac{1}{365}$ . The number of shared-birthday pairs is the sum of these 435 indicators, so by linearity $E [number of pairs]$ $= 435$ $\times \frac{1}{365}$ $\approx 1.19 .$ Notice what we didn't need. The indicators are not independent: if Ana shares a birthday with Ben, and Ben with Carl, then Ana shares one with Carl. But linearity of expectation holds regardless. (The variance would need the covariances.) This average of 1.19 pairs fits with the birthday problem (Section 3.4): with 30 people, the probability of at least one shared birthday is about 0.71.

set.seed(6)
pairs <- replicate(10000, { b <- sample(1:365, 30, replace = TRUE); sum(choose(table(b), 2)) })
mean(pairs)          # close to 1.19

output

[1] 1.1759

Sums of independent named distributions

Sometimes the sum of independent random variables has a recognizable distribution.

Theorem: Sums of Independent Poisson and Gaussian Random Variables

If $X \sim Pois (λ)$ and $Y \sim Pois (μ)$ are independent, then $X$ $+ Y$ $\sim Pois (λ$ $+ μ)$ .
If $X \sim N (μ_{1}, σ_{1}^{2})$ and $Y \sim N (μ_{2}, σ_{2}^{2})$ are independent, then $X$ $+ Y$ $\sim N (μ_{1}$ $+ μ_{2},$ $σ_{1}^{2}$ $+ σ_{2}^{2})$ .

(1) The event $X + Y = n$ splits into the disjoint events $X$ $= k,$ $Y$ $= n$ $- k$ for $k = 0, \dots, n$ . Using independence and the binomial theorem, $\begin{matrix} Pr (X + Y = n) & = \sum_{k = 0}^{n} \frac{e^{- λ} λ^{k}}{k!} \cdot \frac{e^{- μ} μ^{n - k}}{(n - k)!} = \frac{e^{- (λ + μ)}}{n!} \sum_{k = 0}^{n} (\binom{n}{k}) λ^{k} μ^{n - k} \\ = \frac{e^{- (λ + μ)} {(λ + μ)}^{n}}{n!} . \end{matrix}$ $Pr (X$ $+ Y = n)$ $= \sum_{k = 0}^{n} \frac{e^{- λ} λ^{k}}{k!}$ $\cdot \frac{e^{- μ} μ^{n - k}}{(n - k)!}$ $= \frac{e^{- (λ + μ)}}{n!} \sum_{k = 0}^{n} (\binom{n}{k}) λ^{k} μ^{n - k}$ $= \frac{e^{- (λ + μ)} {(λ + μ)}^{n}}{n!} .$

(2) The mean and variance follow from linearity and the variance of a sum. That the sum is Gaussian again can be proved with moment generating functions: if $X$ and $Y$ are independent, then $E [e^{t (X + Y)}]$ $= E [e^{t X}] E [e^{t Y}]$ by the product rule of Section 6.4, and the product of two Gaussian MGFs is a Gaussian MGF (Exercise 14). $◼$ ∎

In the football model of Example 6.6, the total number of goals is therefore $Pois (2.7)$ , and the chance of a 0–0 draw is $e^{- 2.7} \approx 0.067$ , matching the top-left cell of the joint table.

Bayesian Inference: What Does a Reading of 0.081 Really Tell Us?

Example 5.7 predicted readings from a known alcohol level. A court has the reverse problem: given a reading of 0.081, how likely is it that the driver's true level $B$ was over 0.08? Bayes' theorem for random variables answers it with a joint distribution: the posterior density of $B$ given the reading $r$ is $f (b ∣ r)$ $= \frac{f (b) f (r ∣ b)}{f (r)}$ $\propto f (b) f (r ∣ b),$ prior times likelihood, exactly as in Section 3.6.

Suppose that, among drivers who are stopped and tested, true levels are roughly Gaussian with mean 0.06 and SD 0.02 (an illustrative prior), and that readings are Gaussian around the true level with SD 0.005. When both pieces are Gaussian, the posterior is Gaussian too, and its mean is a weighted average of the prior mean and the reading, each weighted by its precision (1/variance): $posterior mean$ $= \frac{0.06 / {0.02}^{2} + 0.081 / {0.005}^{2}}{1 / {0.02}^{2} + 1 / {0.005}^{2}}$ $\approx 0.0798,$ $posterior SD$ $= \frac{1}{\sqrt{1 / {0.02}^{2} + 1 / {0.005}^{2}}}$ $\approx 0.0049 .$

prec <- c(prior = 1 / 0.02^2, reading = 1 / 0.005^2)
post_mean <- sum(prec * c(0.06, 0.081)) / sum(prec)
post_sd <- 1 / sqrt(sum(prec))
c(post_mean, post_sd, 1 - pnorm(0.08, post_mean, post_sd))

output

[1] 0.079764706 0.004850713 0.480656043

So $Pr (B > 0.08 ∣ reading 0.081)$ $\approx 0.48$ : given everything we know, it's about as likely as not that this driver was under the limit. The estimate of the true level is pulled slightly below the reading, towards the typical tested driver, because readings just over the limit are more often lucky high readings from drivers just under it than the other way round. This is regression to the mean once more, now as a Bayesian calculation, and it's the reason most legal systems allow a margin for measurement error.

Check Yourself

$X$ and $Y$ are independent with SDs 3 and 4. What is $SD [X - Y]$ ?

SD =

A lab instrument has measurement SD 0.08 mg. How many independent repeat measurements must be averaged to bring the SD of the average down to 0.02 mg?

n =

Think About It: Ten Witnesses, or One?

Ten eyewitnesses each estimate a getaway car's speed. If their errors were independent, the average of ten estimates would have one-third ( $1 / \sqrt{10} \approx 0.32$ ) of the spread of a single estimate. Name two reasons why the errors may not be independent, and say what that does to the value of the extra witnesses.

Witnesses who talked to each other before giving statements tend to converge on a shared story, and witnesses who all stood at the same spot share the same viewing angle and the same illusions of speed. Leading questions from the same investigator can push them all the same way. Positively correlated errors mean the average is less precise than $σ / \sqrt{10}$ , and if the correlation is high, ten witnesses may be little better than one, even though their agreement looks impressive. This is why police guidelines say witnesses should be separated before they give statements: it protects the independence that makes several accounts more valuable than one. The "wisdom of crowds" works only when the crowd's errors are independent.

Key Takeaways

$Var [a X$ $+ b Y]$ $= a^{2} Var [X]$ $+ b^{2} Var [Y]$ $+ 2 a b Cov (X,$ $Y)$ ; for independent variables the covariance term vanishes.
The average of $n$ independent measurements with SD $σ$ has SD $σ / \sqrt{n}$ . Positive correlation erodes the benefit.
Write counts as sums of indicators; linearity of expectation needs no independence.
Independent Poissons add to a Poisson; independent Gaussians add to a Gaussian.

Prediction and inference in this chapter

Prediction. The best prediction of one quantity from another is a conditional mean: a son's height given his father's, the wait for Old Faithful given the last eruption. The conditional spread is the size of the error to expect. The best straight-line prediction has slope $ρ σ_{Y} / σ_{X}$ , and in standard units it is always closer to the average than the predictor, which is regression to the mean. And averages of independent measurements predict more precisely than single ones, by a factor of $\sqrt{n}$ .
Inference. Joint distributions describe how quantities are connected. Independence (or its failure) is a claim about the mechanism, and we checked it in tables, in densities and in football scores. Correlation measures linear association, but the storks, the police and Galton's own reading of his data all showed that association alone can't tell us why two things move together; for that we need the DAGs of Chapter 1 and the right comparison groups. The Bayesian box turned a breathalyser reading into a probability about the driver's true level, by combining a joint distribution of truth and measurement.
What comes next. The sample mean $\overline{X}$ is itself a random variable: a sum of independent pieces, divided by $n$ . This section gave its mean and its SD. Chapter 7 finds its whole distribution, which turns out to be approximately Gaussian (the Central Limit Theorem), and that is the key to the margins of error and tests in Chapters 8 and 9.

6.7 Interludes

Interlude (Optional): Galton's Ellipse: The Bivariate Gaussian

When Galton drew contour lines through his table of parents' and children's heights, joining cells with equal counts, he found that they formed ellipses, tilted along the direction of the correlation. He asked the Cambridge mathematician J. D. Hamilton Dickson to find the surface that would produce them, and in 1886 Dickson gave him the bivariate Gaussian density. In standard units, with correlation $ρ$ , it is $f (x,$ $y)$ $= \frac{1}{2 π \sqrt{1 - ρ^{2}}} e^{- q (x, y) / 2},$ $q (x,$ $y)$ $= \frac{x^{2} - 2 ρ x y + y^{2}}{1 - ρ^{2}} .$ Its contours are ellipses, its marginals are standard Gaussians, and every conditional distribution is Gaussian too: given $X = x$ , the variable $Y$ is Gaussian with mean $ρ x$ and variance $1 - ρ^{2}$ . So for bivariate Gaussian data the regression function is exactly a straight line, the best linear predictor of Section 6.5 is the best predictor of all, and zero correlation does mean independence (set $ρ = 0$ and the density factorizes). It is the one family where correlation tells the whole story.

Galton's own data show the ellipses. The curves below enclose about 50% and 90% of a bivariate Gaussian fitted to the fathers' and sons' heights; their tilt is the correlation of 0.39.

galton <- read.csv("https://raw.githubusercontent.com/vincentarelbundock/Rdatasets/master/csv/HistData/GaltonFamilies.csv")
sons <- subset(galton, gender == "male")
ggplot(sons, aes(father, childHeight)) +
  geom_jitter(width = 0.1, height = 0.1, alpha = 0.35, colour = "steelblue") +
  stat_ellipse(type = "norm", level = 0.5, linewidth = 1) +
  stat_ellipse(type = "norm", level = 0.9, linewidth = 1, linetype = "dashed") +
  labs(x = "father's height (inches)", y = "son's height (inches)")

The same idea extends to any number of variables, as the multivariate Gaussian, described by a vector of means and a matrix of covariances. It is the workhorse of modern statistics and machine learning, from weather forecasting to the "latent spaces" of image generators.

Why it matters: when you see correlation used as if it were the measure of dependence, the analyst is (often silently) assuming a bivariate Gaussian. For data like the dart's, that assumption fails.

Interlude (Optional): Markov Chains: Dependence Through Time

Many sequences of random variables are dependent in a simple way: tomorrow depends on today, but not on the more distant past. Suppose that if today is rainy, tomorrow is rainy with probability 0.6, and if today is dry, tomorrow is rainy with probability 0.2. This is a Markov chain, named after the Russian mathematician Andrey Markov, who in 1913 used one to describe how vowels and consonants follow each other in Pushkin's novel in verse Eugene Onegin.

What fraction of days are rainy in the long run? If that fraction is $π$ , it must reproduce itself from one day to the next (by the Law of Total Probability): $π$ $= 0.6 π$ $+ 0.2 (1$ $- π)$ $\Rightarrow$ $0.6 π$ $= 0.2$ $\Rightarrow$ $π$ $= \frac{1}{3} .$ Whatever the weather today, after a few weeks the chance of rain settles at $\frac{1}{3}$ . Markov chains are behind Google's original PageRank (a random surfer clicking links), the text predictors on your phone, models of how diseases progress, and the simulation methods that make modern Bayesian statistics possible.

Why it matters: independence is the exception. Most real data, from daily ozone (Chapter 1) to the eruptions of Old Faithful, carry memory, and the models of later courses handle it with chains like this one.

6.8 Exercises

Each exercise has a hint and an answer. For continuous problems, sketch the region where the density is positive before setting up any integral.

Exercises

A joint table. Discrete random variables $X$ and $Y$ have the joint PMF below.

Y = 1 Y = 2 Y = 3 Y = 4

$X = 1$ 0.10 0.05 0.03 0.01

$X = 2$ 0.07 0.16 0.18 0.09

$X = 3$ 0.12 0.09 0.04 0.06

(a) Find $Pr (X = 2,$ $Y$ $= 4)$ and $Pr (X = 1,$ $Y$ $= 1)$ . (b) Find the marginal distributions of $X$ and $Y$ . (c) Find $Pr (X = 2 ∣ Y = 4)$ and $Pr (Y \leq 3 ∣ X = 3)$ . (d) Find $Cov (X, Y)$ . (e) Are $X$ and $Y$ independent?

Row sums give $p_{X}$ ; column sums give $p_{Y}$ . For (d), compute $E [X Y]$ by multiplying each cell by $x y$ and adding; then subtract $E [X] E [Y]$ . For (e), find one cell that isn't the product of its margins.

(a) 0.09 and 0.10. (b) $p_{X}$ : 0.19, 0.50, 0.31; $p_{Y}$ : 0.29, 0.30, 0.25, 0.16. (c) $0.09 / 0.16$ $= 0.5625$ ; $(0.12$ $+ 0.09$ $+ 0.04) / 0.31$ $= 0.25 / 0.31$ $\approx 0.806$ . (d) $E [X] = 2.12$ , $E [Y] = 2.28$ , $E [X Y] = 4.89$ , so $Cov (X,$ $Y)$ $= 4.89$ $- 2.12$ $\times 2.28$ $\approx 0.056$ . (e) No: $Pr (X = 1,$ $Y$ $= 1)$ $= 0.10$ , but $p_{X} (1) p_{Y} (1)$ $= 0.19$ $\times 0.29$ $\approx 0.055$ .
A joint density. $X$ and $Y$ have joint density $f (x,$ $y)$ $= x$ $+ \frac{3}{2} y^{2}$ for $0 \leq x, y \leq 1$ . (a) Verify that $f$ is a valid joint density. (b) Find the marginal densities. (c) Find $E [X]$ and $E [Y]$ . (d) Find $Corr (X, Y)$ .

For (d) you need $E [X^{2}]$ , $E [Y^{2}]$ (from the marginals) and $E [X Y]$ $= \int_{0}^{1} \int_{0}^{1} x y (x$ $+ 1.5 y^{2}) d y d x$ .

(a) $f \geq 0$ , and $\int_{0}^{1} \int_{0}^{1} (x + \frac{3}{2} y^{2}) d y d x$ $= \frac{1}{2}$ $+ \frac{1}{2}$ $= 1$ . (b) $f_{X} (x) = x + \frac{1}{2}$ and $f_{Y} (y)$ $= \frac{1}{2}$ $+ \frac{3}{2} y^{2}$ , each on $[0, 1]$ . (c) $E [X]$ $= \frac{1}{3}$ $+ \frac{1}{4}$ $= \frac{7}{12}$ and $E [Y]$ $= \frac{1}{4}$ $+ \frac{3}{8}$ $= \frac{5}{8}$ . (d) $E [X^{2}] = \frac{5}{12}$ , $E [Y^{2}] = \frac{7}{15}$ and $E [X Y]$ $= \frac{1}{6}$ $+ \frac{3}{16}$ $= \frac{17}{48}$ , so $Cov (X,$ $Y)$ $= \frac{17}{48}$ $- \frac{35}{96}$ $= - \frac{1}{96}$ , $Var [X] = \frac{11}{144}$ , $Var [Y] = \frac{73}{960}$ , and $ρ \approx - 0.137$ .
Correlation from a small table. Find $Corr (X, Y)$ for the joint PMF below. Are $X$ and $Y$ independent?

Y = 0 Y = 3

$X = 0$ 0.15 0.18

$X = 1$ 0.20 0.30

$X = 2$ 0.07 0.10

Marginals: $p_{X}$ $= (0.33,$ $0.50,$ $0.17)$ , $p_{Y}$ $= (0.42,$ $0.58)$ . Only the $Y = 3$ column contributes to $E [X Y]$ .

$E [X] = 0.84$ , $E [Y] = 1.74$ , $E [X Y]$ $= 3 (0.30$ $+ 2 \times 0.10)$ $= 1.50$ , so $Cov$ $= 1.50$ $- 0.84$ $\times 1.74$ $= 0.0384$ . With $Var [X]$ $= 1.18$ $- {0.84}^{2}$ $= 0.4744$ and $Var [Y]$ $= 5.22$ $- {1.74}^{2}$ $= 2.1924$ , $ρ \approx 0.038$ : very nearly uncorrelated. But they aren't independent: $Pr (X = 0,$ $Y$ $= 0)$ $= 0.15$ , while $0.33$ $\times 0.42$ $\approx 0.139$ .
Does the fund beat the market? Many hedge funds advertise returns that are uncorrelated with the stock market. Over 10 weeks, the market's weekly returns (%) were 1, 5, 3, 10, 15, 2, 3, 7, 11, 10, and a fund's were 2, 7, 7, 11, 12, 8, 6, 8, 10, 14. (a) Draw a scatterplot. (b) Compute the sample correlation. (c) Is the advertisement believable?

Enter the two vectors in R and use plot(x, y) and cor(x, y).

(b) $r \approx 0.83$ . (c) No: the fund rose and fell with the market. With only 10 weeks the estimate is uncertain, but a correlation this high would be very unlikely if the true correlation were 0 (Chapter 9 makes this precise).
Linearity without independence. Twenty people throw their hats into a pile and each takes one at random. Let $N$ be the number who get their own hat. (a) Using indicators, find $E [N]$ . (b) Are the indicators independent? (c) Compare with Secret Santa (Example 4.7).

Person $k$ gets their own hat with probability $\frac{1}{20}$ .

(a) $E [N]$ $= 20$ $\times \frac{1}{20}$ $= 1$ , whatever the number of people. (b) No: if 19 people have their own hats, so does the 20th. (c) It's the same problem; there, about 37% of games had nobody drawing their own name, and the average was 1.
Which comes first? Three components of a machine fail at random, independently, with exponential lifetimes of mean 2, 4 and 8 years. (a) What is the distribution of the time until the first failure? (b) What is its mean? (c) What is the probability that the 2-year component fails first?

The rates are $\frac{1}{2}$ , $\frac{1}{4}$ and $\frac{1}{8}$ per year. Extend Example 6.7 to three variables.

(a) $Exp (\frac{1}{2} + \frac{1}{4} + \frac{1}{8})$ $= Exp (\frac{7}{8})$ . (b) $\frac{8}{7} \approx 1.14$ years. (c) $\frac{1 / 2}{7 / 8}$ $= \frac{4}{7}$ $\approx 0.571$ .
Independent or not? For each joint density, decide whether $X$ and $Y$ are independent, and explain. (a) $f (x, y) = 4 x y$ on $0 \leq x, y \leq 1$ . (b) $f (x, y) = 8 x y$ on $0 \leq x \leq y \leq 1$ . (c) $f (x,$ $y)$ $= \frac{1}{2 π} e^{- (x^{2} + y^{2}) / 2}$ for all $x, y$ .

Does the density factor into a function of $x$ times a function of $y$ , and is the region a rectangle?

(a) Independent: $4 x y = 2 x \cdot 2 y$ on a square. (b) Dependent: the formula factors, but the region $x \leq y$ isn't a rectangle; learning $Y = 0.3$ forces $X \leq 0.3$ . (c) Independent: $\frac{1}{2 π} e^{- x^{2} / 2} e^{- y^{2} / 2}$ is the product of two standard Gaussian densities.
Pairwise but not mutual. Toss two fair coins. Let $X = 1$ if the first is heads, $Y = 1$ if the second is heads, and $Z = 1$ if they match (each is 0 otherwise). (a) Show that $X$ and $Z$ are independent. (b) Show that $X$ , $Y$ and $Z$ are not mutually independent.

List the four equally likely outcomes with the values of $X$ , $Y$ , $Z$ .

(a) $Pr (X = 1,$ $Z$ $= 1)$ $= Pr (HH)$ $= \frac{1}{4}$ $= \frac{1}{2}$ $\times \frac{1}{2}$ ; the other three cells work the same way. (b) $Pr (X = 1,$ $Y$ $= 1,$ $Z$ $= 0)$ $= 0$ , but the product of the marginals is $\frac{1}{8}$ .
Galton's prediction. Using the line $\hat{son}$ $= 38.363$ $+ 0.4465$ $\times father$ for Galton's data: (a) predict the son's height for fathers of 65, 69.1 and 75 inches; (b) of the 15 sons whose fathers were at least 74 inches tall, 12 were shorter than their fathers. Is that surprising? (c) Would you use this line to predict the height of a son born in 2025? Why or why not?

For (b), think about regression to the mean. For (c), think about whether the mechanism has stayed the same since the 1880s.

(a) About 67.4, 69.2 and 71.9 inches. (b) Not surprising: the prediction for these fathers is about 72 inches, well below their own heights, so most sons should be shorter. (c) No: average heights have risen by several inches since Galton's time, because of better nutrition and health. The relationship within families may be similar, but the intercept has changed. Predictions assume the mechanism that produced the data still holds.
Diversification. Using R's EuStockMarkets (daily closing prices of four European stock indices, 1991–1998), compute daily percentage returns with r <- diff(log(EuStockMarkets)) * 100. (a) Find the SDs of the DAX (Germany) and FTSE (UK) returns and their correlation. (b) Use the variance-of-a-sum formula to find the SD of a portfolio half in each, and check it in R. (c) What would the portfolio's SD be if the two were independent?

The portfolio return is $\frac{1}{2} R_{DAX}$ $+ \frac{1}{2} R_{FTSE}$ . Use sd(r[, "DAX"]), cor(r[, "DAX"], r[, "FTSE"]).

(a) SDs about 1.03% and 0.80%; correlation about 0.64. (b) The variance is $\frac{1}{4} ({1.03}^{2}$ $+ {0.80}^{2})$ plus the covariance term $\frac{1}{2}$ $\times 0.64$ $\times 1.03$ $\times 0.80$ , about 0.69 in total (0.686 with unrounded inputs), so the SD is about $0.83 %$ , matching sd((r[, "DAX"] + r[, "FTSE"]) / 2). That's below the DAX's SD but slightly above the FTSE's: with correlation 0.64, mixing helps only a little. (c) $\frac{1}{2} \sqrt{{1.03}^{2} + {0.80}^{2}}$ $\approx 0.65 %$ . Diversification works best when assets are weakly correlated.
Total expectation at the dispatch desk. On a night shift, a dispatcher receives $N \sim Pois (6)$ calls, and each call takes on average 15 minutes to handle, independently of how many calls there are. What is the expected total handling time? Use the law of total expectation.

Given $N = n$ , the expected total is $15 n$ . Average that over the distribution of $N$ .

$E [total]$ $= \sum_{n} 15 n Pr (N = n)$ $= 15 E [N]$ $= 90$ minutes.
Spot the flaw. For each claim, say what is wrong. (a) "The correlation between ice-cream sales and drownings is 0.8, so banning ice cream would save lives." (b) "Hours studied and exam score have correlation 0, so studying doesn't matter." (c) "Shoe size and reading ability are strongly correlated in elementary schools, so bigger feet help children read."

Think about confounders, curved relationships and what was held fixed.

(a) Summer heat is a common cause of both; correlation isn't causation. (b) Zero correlation rules out only a straight-line trend; the relationship could be curved (a little study helps a lot, cramming all night hurts), or other variables could mask it (weaker students study more). (c) Age is a confounder: older children have bigger feet and read better.
The Gaussian integral. Chapter 5 promised a proof that the standard Gaussian density integrates to 1. Let $I$ $= \int_{- \infty}^{\infty} e^{- x^{2} / 2} d x$ . (a) Write $I^{2}$ as a double integral over the plane. (b) Change to polar coordinates ( $d x d y = r d r d θ$ ) and evaluate it. (c) Conclude that $\frac{1}{\sqrt{2 π}} e^{- x^{2} / 2}$ is a valid density.

$I^{2}$ $= \int e^{- x^{2} / 2} d x \int e^{- y^{2} / 2} d y$ $= \iint e^{- (x^{2} + y^{2}) / 2} d x d y$ , and $x^{2} + y^{2} = r^{2}$ . The substitution $u = r^{2} / 2$ handles the $r$ integral.

$I^{2}$ $= \int_{0}^{2 π} \int_{0}^{\infty} e^{- r^{2} / 2} r d r d θ$ $= 2 π [- e^{- r^{2} / 2}]_{0}^{\infty} = 2 π$ , so $I = \sqrt{2 π}$ , and dividing by $\sqrt{2 π}$ gives total area 1. Notice that the proof uses exactly the factorization of the joint density in Exercise 7(c).
Gaussian sums by MGFs. The moment generating function of $N (μ, σ^{2})$ is $M (t)$ $= e^{μ t + σ^{2} t^{2} / 2}$ . Use it to show that the sum of independent $N (μ_{1}, σ_{1}^{2})$ and $N (μ_{2}, σ_{2}^{2})$ random variables is Gaussian, and find its parameters.

For independent $X$ and $Y$ , $M_{X + Y} (t)$ $= M_{X} (t) M_{Y} (t)$ . An MGF determines the distribution.

$M_{X + Y} (t)$ $= e^{μ_{1} t + σ_{1}^{2} t^{2} / 2} e^{μ_{2} t + σ_{2}^{2} t^{2} / 2}$ $= e^{(μ_{1} + μ_{2}) t + (σ_{1}^{2} + σ_{2}^{2}) t^{2} / 2}$ , the MGF of $N (μ_{1}$ $+ μ_{2},$ $σ_{1}^{2}$ $+ σ_{2}^{2})$ .

	Y = 1	Y = 2	Y = 3	Y = 4
$X = 1$	0.10	0.05	0.03	0.01
$X = 2$	0.07	0.16	0.18	0.09
$X = 3$	0.12	0.09	0.04	0.06

	Y = 0	Y = 3
$X = 0$	0.15	0.18
$X = 1$	0.20	0.30
$X = 2$	0.07	0.10

Chapter 6Joint distributions

Think About It: Before You Start

Lecture 15 — Joint, marginal and conditional distributions

6.1 Joint distributions

Two variables in one table

Definition: Joint PMF

Example 6.1: Hair and Eye Colour ★ lecture example

Two continuous variables: joint densities

Definition: Joint Density

Example 6.2: The Shooting-Range Target ★ lecture example

Check Yourself

Think About It: Whose Table Is It?

Key Takeaways

6.2 Marginal and conditional distributions

Marginals: one variable at a time

Definition: Marginal Distributions

Conditionals: one variable given the other

Definition: Conditional Distributions

Example 6.3: What Colour Are a Blond Student's Eyes? ★ lecture example

Example 6.4: The Target's Margins and Slices

Example 6.5: The Dart's Horizontal Position ★ lecture example

Check Yourself

Think About It: Who Is in the Denominator?

Key Takeaways

6.3 Independence

Definition: Independent Random Variables

Watch Out: Pairwise Isn't Mutual

Example 6.6: Who Wins the Match? ★ lecture example

Example 6.7: Police or Fire First? ★ lecture example

Check Yourself

Think About It: Independence Is a Claim About the World

Key Takeaways

Lecture 16 — Prediction, covariance and sums

6.4 Expected values and prediction

Expected values of functions of two variables

Theorem: Expected Value of a Function of X and Y

Theorem: Linearity, and Products of Independent Variables

Conditional expectation: the best prediction

Definition: Conditional Expectation

Theorem: The Conditional Mean Is the Best Predictor

Theorem: Law of Total Expectation

Example 6.8: When Will Old Faithful Erupt Again?

Example 6.9: Tall Fathers, Slightly Less Tall Sons ★ lecture example

Causal Inference: Why Do Sons Regress?

Check Yourself

Think About It: The Sophomore Slump

Key Takeaways

6.5 Covariance and correlation

Covariance

Definition: Covariance

Theorem: Properties of Covariance

Example 6.10: The First Die and the Total ★ lecture example

Correlation

Definition: Correlation

Theorem: Correlation Is Between −1 and 1

Correlation and the best straight-line prediction

Theorem: The Best Linear Predictor

Example 6.11: Predicting Old Faithful from a Straight Line ★ lecture example

Uncorrelated isn't independent

Example 6.12: The Dart, Once More ★ lecture example

Watch Out: What a Correlation Can't Tell You

Causal Inference: Storks Deliver Babies (R = 0.62)

History: Correlation Was Born in a Police Bureau

Check Yourself

Think About It: More Police, More Crime?

Key Takeaways

6.6 Sums and averages

Theorem: Variance of a Sum

Example 6.13: Is a Second Breathalyser Reading Worth Taking? ★ lecture example

Example 6.14: Where the Binomial Variance Comes from

Example 6.15: How Many Shared Birthdays? ★ lecture example

Sums of independent named distributions

Theorem: Sums of Independent Poisson and Gaussian Random Variables

Bayesian Inference: What Does a Reading of 0.081 Really Tell Us?

Check Yourself

Think About It: Ten Witnesses, or One?

Key Takeaways

Prediction and inference in this chapter

6.7 Interludes

Interlude (Optional): Galton's Ellipse: The Bivariate Gaussian

Chapter 6
Joint distributions