Chapter 9
Hypothesis testing

Lectures 23–25 · weighing evidence against a claim: null hypotheses, p-values, tests for means and proportions, errors and power, and testing in a world of many tests

a hypothesis test works like a trial: assume the null, and reject it only if the evidence is overwhelming

In 1710, the Scottish physician John Arbuthnot studied 82 years of London's christening records, from 1629 to 1710. In every single year more boys had been christened than girls. If boys and girls were equally likely, he reasoned, each year would be like a coin toss, and the chance of boys winning all 82 tosses would be ${(\frac{1}{2})}^{82}$ , about $2 \times 10^{- 25}$ , a number with 24 zeros after the decimal point. So chance couldn't be the explanation. (Arbuthnot concluded that Divine Providence was at work, ensuring enough men to replace those lost to war and danger. Today we know that slightly more boys than girls are born almost everywhere, about 105 for every 100.) Arbuthnot's argument is often called the first significance test: assume a simple explanation, compute how surprising the data would be if it were true, and abandon it if the answer is "absurdly surprising".

This chapter makes that logic precise. Chapter 8 asked what values of a parameter are plausible. A hypothesis test asks whether one particular value, or one particular claim, is plausible: Is this die fair? Does the new drug do anything? Did smoking mothers have lighter babies? Has the filling machine drifted?

Testing is a tool of inference: it asks whether the data give evidence against a claim about the mechanism that produced them. It's also a tool for decisions: shut down the production line or not, approve the drug or not, launch the new version of a website or not. Those decisions are made under uncertainty, so they can be wrong in two different ways, and Section 9.5 counts the cost of each. Finally, in a world where thousands of tests are run every day, some "discoveries" are bound to be false alarms. Section 9.7 asks how many, and connects testing back to Bayes' theorem and to the causal questions of Chapter 1.

Here is the route through the chapter.

Sections 9.1 and 9.2 set out the logic of a test and the p-value.
Sections 9.3 and 9.4 give tests for means and proportions.
Section 9.5 counts the two kinds of error, and the power to detect a real effect.
Section 9.6 connects tests with confidence intervals, and statistical with practical significance.
Section 9.7 asks what happens when many tests are run, and why replication matters.

Think About It: Before You Start

A friend claims she can tell whether a coin is fair just by flipping it. She flips a coin 10 times, gets 8 heads, and announces: "This coin is biased towards heads." Is 8 heads out of 10 good evidence? What would you need to know to decide? Would 80 heads out of 100 be different?

With a fair coin, 8 or more heads in 10 flips happens with probability $Pr (X \geq 8)$ $= \frac{45 + 10 + 1}{1024}$ $\approx 0.055$ , about 1 time in 18: unusual, but not shocking. If she also would have called the coin biased after 8 tails, the relevant probability doubles to about 0.11. So 8 out of 10 is weak evidence. But 80 heads in 100 flips is a different matter: that's 6 standard deviations above 50, which essentially never happens with a fair coin. The same proportion gives very different evidence depending on the sample size, which is the central lesson of Chapter 7, applied to testing.

Lecture 23 — The logic of a test, p-values and tests for a mean

9.1 The logic of a test

A hypothesis test is a structured argument by contradiction, with a probabilistic twist. A mathematical proof by contradiction assumes a claim and derives something impossible. A test assumes a claim and finds that the data would be very unlikely.

Definition: Null and Alternative Hypotheses

The null hypothesis $H_{0}$ is the default claim about a parameter, usually "no effect", "no difference" or "nothing unusual": for example $H_{0} : μ = μ_{0}$ . The alternative hypothesis $H_{1}$ is the claim we're looking for evidence for. It can be

two-sided: $H_{1} : μ \neq μ_{0}$ ;
right-sided: $H_{1} : μ > μ_{0}$ ; or
left-sided: $H_{1} : μ < μ_{0}$ .

Hypotheses are statements about population parameters, never about the sample, and the choice of alternative should be made before looking at the data.

Definition: Test Statistic and Null Distribution

A test statistic $T$ is a statistic that measures how far the data are from what $H_{0}$ predicts. Its sampling distribution, computed assuming $H_{0}$ is true, is the null distribution. Most test statistics have the form $T$ $= \frac{estimate - value claimed by H_{0}}{standard error}$ $= \frac{signal}{noise} .$

The courtroom is the best analogy, and it's worth keeping in mind throughout the chapter.

courtroom	hypothesis test
the defendant is presumed innocent	assume $H_{0}$ is true
the prosecution presents evidence	collect data and compute $T$
guilty only "beyond reasonable doubt"	reject $H_{0}$ only if the data would be very unlikely under it
"not guilty" doesn't mean "innocent"	"fail to reject $H_{0}$ " doesn't mean $H_{0}$ is true
convicting an innocent person	a Type I error (Section 9.5)
acquitting a guilty person	a Type II error

Example 9.1: Boys and Girls in 1710 ★ lecture example

Arbuthnot found more male than female christenings in each of 82 consecutive years. (a) State the null hypothesis he was testing. (b) How probable was his observation under it? (c) His null was rejected decisively, but was his conclusion justified?

(a) $H_{0}$ : each year, more boys than girls is a 50–50 event, independently of other years. In parameter terms, $H_{0} : p = \frac{1}{2}$ , where $p$ is the probability that boys outnumber girls in a year.

(b) Under $H_{0}$ , the number of "boy years" in 82 is $Bin (82, \frac{1}{2})$ , and $Pr (all 82 years)$ $= {(\frac{1}{2})}^{82}$ $\approx 2$ $\times 10^{- 25} .$

0.5^82
binom.test(82, 82, p = 0.5, alternative = "greater")$p.value

output

[1] 2.067952e-25
[1] 2.067952e-25

(c) Rejecting $H_{0}$ was right: boys really are slightly more likely than girls at birth. But the test says only that $H_{0}$ is false, not why. Arbuthnot's conclusion (Divine Providence) is one of countless alternatives consistent with the data. A test can rule out a simple explanation; it can't by itself establish any particular other one. This is the same gap between data and mechanism that ran through Chapter 1.

Arbuthnot's data survive. In every year the ratio of boys to girls christened was above 1 (the dashed line), usually by a few percent:

arb <- read.csv("https://raw.githubusercontent.com/vincentarelbundock/Rdatasets/master/csv/HistData/Arbuthnot.csv")
ggplot(arb, aes(Year, Males / Females)) + geom_line(colour = "steelblue") + geom_point(size = 1) +
  geom_hline(yintercept = 1, linetype = "dashed") +
  labs(x = "year", y = "boys per girl")

Example 9.2: Has the Machine Drifted? ★ lecture example

Each vial from a filling machine should contain 500 mg of a drug. The fills vary with a known SD of $σ = 8$ mg. An engineer weighs a random sample of 20 vials. Without doing a formal test yet, compare two results: (a) $\overline{x} = 501$ mg; (b) $\overline{x} = 523$ mg. What is at stake if she decides wrongly?

If the machine is on target, $\overline{X}$ $\sim N (500,$ $8^{2} / 20)$ , with standard error $8 / \sqrt{20} \approx 1.79$ mg.

(a) $\overline{x} = 501$ is $\frac{1}{1.79} \approx 0.56$ standard errors above 500, entirely ordinary for a machine on target.

(b) $\overline{x} = 523$ is $\frac{23}{1.79} \approx 12.9$ standard errors above 500. A calibrated machine would essentially never produce it.

Both means exceed 500, but only the second is surprising: what matters is the deviation relative to the standard error. That ratio is the test statistic.

What's at stake. Stopping the line when nothing is wrong (a false alarm) wastes money and time. Missing a real drift sends patients doses that are too high. The two errors have very different costs, and Section 9.5 shows how to balance them.

History: Popper and the Black Swan

The philosopher Karl Popper (1902–1994) pointed out an asymmetry in how evidence works. No number of white swans can prove that all swans are white, but a single black swan proves that not all are, and black swans (Cygnus atratus) were indeed found in Australia. Popper argued that a scientific claim must be falsifiable: it must say something that observation could contradict.

The null hypothesis is a falsifiable claim, set up to be knocked down. There's one important difference from Popper's logic. A black swan logically refutes "all swans are white", but in statistics nothing is logically impossible under $H_{0}$ ; extreme data are only improbable. We reject $H_{0}$ when keeping it would require believing in a very unlikely coincidence. And, as with the swans, failing to find evidence against $H_{0}$ never proves it: not seeing a black swan this afternoon doesn't mean they don't exist.

Check Yourself

A researcher wants evidence that a new teaching method raises the mean exam score above the current 70. Which hypotheses should she use?

H_{0} : \overline{x} = 70

H_{1} : \overline{x} > 70

H_{0} : μ = 70

H_{1} : μ > 70

H_{0} : μ > 70

H_{1} : μ = 70

Think About It: Who Carries the Burden of Proof?

A company wants to sell a new food additive. Should the null hypothesis be "the additive is safe" or "the additive is harmful"? Who benefits from each choice, and what does each mean for a small, inconclusive study?

The null is the claim that stands unless the evidence overturns it. With $H_{0}$ : "safe", a small, noisy study that finds nothing lets the additive onto the market: failure to reject favours the company. With $H_{0}$ : "harmful" (or "not shown to be safe"), the company must produce strong evidence of safety, and an inconclusive study keeps the additive off the market. Regulators such as the US Food and Drug Administration generally put the burden on the manufacturer to demonstrate safety for new drugs and food additives. Which hypothesis is the null isn't a mathematical question: it's a decision about who bears the risk of error, and it should be made openly.

Key Takeaways

A test assumes $H_{0}$ and asks whether the data would be surprising under it.
$H_{0}$ and $H_{1}$ are about parameters; choose one- or two-sided before seeing the data.
A test statistic is usually (estimate − null value)/standard error: signal over noise.
Rejecting $H_{0}$ doesn't establish any particular alternative explanation.

9.2 p-values

Definition: p-Value

The p-value is the probability, computed assuming $H_{0}$ is true, of getting a test statistic at least as extreme as the one observed, in the direction(s) of the alternative. For an observed value $t$ :

right-sided: $p$ $= Pr (T \geq t ∣ H_{0})$ ;
left-sided: $p$ $= Pr (T \leq t ∣ H_{0})$ ;
two-sided: $p$ $= Pr (| T | \geq | t | ∣ H_{0})$ .

A small p-value means the data would be unusual if $H_{0}$ were true, which is evidence against $H_{0}$ .

To turn a p-value into a decision, choose a significance level $α$ in advance (0.05 is conventional, but not sacred). If $p \leq α$ , reject $H_{0}$ : the result is statistically significant at level $α$ . If $p > α$ , fail to reject $H_{0}$ . Equivalently, reject when the test statistic falls in the rejection region, the set of values beyond the critical value (such as $| z | > 1.96$ for a two-sided test at the 5% level).

Test statistics, rejection regions and p-values

The curve is the null distribution: how the test statistic behaves if H₀ is true. The red bars along the axis mark the rejection region: the statistic lands there with probability α if H₀ is true. The orange area is the p-value: the probability, under H₀, of a statistic at least as extreme as the one observed. Load one of the chapter's examples, or move the observed statistic yourself. Notice that we reject exactly when the observed statistic is in the red region, which is exactly when the p-value is at most α.

load an example null distribution degrees of freedom =

alternative significance level α = observed statistic =

Watch Out: What a p-Value Is Not

A p-value is $Pr (data at least this extreme ∣ H_{0})$ . It is not:

the probability that $H_{0}$ is true (that's $Pr (H_{0} ∣ data)$ , the reverse conditional: Section 3.6's prosecutor's fallacy again);
the probability that the result happened "by chance";
a measure of how large or important the effect is (Section 9.6);
and $1 - p$ is not the probability that $H_{1}$ is true.

A large p-value doesn't show that $H_{0}$ is true either: the study may simply have been too small to detect the effect (Section 9.5).

Example 9.3: Evidence for Precognition? ★ lecture example

In 2011 the psychologist Daryl Bem published nine experiments in a leading journal, claiming statistically significant evidence that people can sense future events. In a simplified version of one experiment, subjects guess which of two curtains hides a picture, with the picture's position chosen at random after the guess. With no precognition, each guess is right with probability $\frac{1}{2}$ . Suppose 53 of 100 guesses are right. (a) Test $H_{0} : p = 0.5$ against $H_{1} : p > 0.5$ . (b) If 20 independent experiments like this were run, and precognition doesn't exist, how many would you expect to reach $p < 0.05$ ? (c) What does this suggest about a published significant result?

(a) Under $H_{0}$ , the number correct is $Bin (100, 0.5)$ , with mean 50 and SD 5, so $z$ $= \frac{53 - 50}{5}$ $= 0.6$ , and the one-sided p-value is $Pr (Z \geq 0.6)$ $\approx 0.27$ . The exact binomial p-value, $Pr (X \geq 53)$ , is 0.31. Either way, 53 correct is entirely consistent with guessing: we fail to reject $H_{0}$ .

1 - pnorm(0.6)                                               # Gaussian approximation
binom.test(53, 100, p = 0.5, alternative = "greater")$p.value  # exact

output

[1] 0.2742531
[1] 0.3086497

(b) If $H_{0}$ is true, a p-value falls below 0.05 with probability (at most) 0.05 (that's what the threshold means), so among 20 experiments we'd expect $20 \times 0.05 = 1$ "significant" result by chance alone.

(c) A single significant result, especially when it's surprising and when several analyses or experiments were tried, is weak evidence. Bem's paper helped trigger psychology's "replication crisis" (Section 9.7): when other researchers repeated his experiments carefully, they found no effect. A p-value controls the false-alarm rate of one pre-planned test, not of a research programme that tries many.

Run the same study again and again

Each run draws a fresh sample of size n from a Gaussian population and carries out a two-sided t-test of H₀: μ = 0. The true effect is the real mean, measured in standard deviations. With no effect, p-values are spread evenly between 0 and 1, and a fraction α of them fall below α purely by chance. With a real effect they pile up near 0, and the fraction below α is the power. Watch how much a single p-value jumps around between identical studies.

true effect = σ sample size n = α =

The widget shows something important: even for identical studies of a real effect, the p-value varies enormously from one study to the next. A p-value is a statistic, with its own sampling distribution, and when $H_{0}$ is true (and the test statistic is continuous) it's uniformly distributed between 0 and 1.

Check Yourself

A two-sided $z$ test gives $z = 2.0$ . Find the p-value, to 4 decimal places.

p =

A study reports

p = 0.01

for the effect of a supplement. Which statement is correct? There is a 1% chance that the supplement has no effect.There is a 99% chance that the supplement works.If the supplement had no effect, data at least this extreme would occur about 1% of the time.

Think About It: No Evidence of an Effect

A trial of a blood-pressure drug with 10 patients finds $p = 0.30$ . The lead investigator writes: "The drug has no effect on blood pressure." What's wrong with this conclusion, and how should it be worded?

Failing to reject $H_{0}$ means the data are consistent with no effect, not that they show no effect. With only 10 patients, the trial may have had little chance of detecting even a useful reduction (low power, Section 9.5). The correct statement is "we found no statistically significant evidence of an effect", ideally with a confidence interval for the effect, such as "the reduction was 4 mmHg, 95% CI −4 to 12 mmHg", which shows that the data are consistent both with no effect and with a large one. "Absence of evidence is not evidence of absence."

Key Takeaways

p-value $= Pr (a statistic at least as extreme as observed ∣ H_{0})$ .
Reject $H_{0}$ if $p \leq α$ , chosen in advance; otherwise "fail to reject", never "accept".
A p-value is not the probability that $H_{0}$ is true, and it doesn't measure effect size.
p-values vary a lot between identical studies; under $H_{0}$ they are uniform on $[0, 1]$ .

9.3 Tests for a mean

The tests for a mean mirror the confidence intervals of Section 8.4: the same three cases, the same standard errors, the same critical values.

Theorem: Tests for a Population Mean

To test $H_{0} : μ = μ_{0}$ with a random sample of size $n$ :

Gaussian population, σ known: $Z = \frac{\overline{X} - μ_{0}}{σ / \sqrt{n}}$ has a standard Gaussian null distribution.
Gaussian population, σ unknown: $T = \frac{\overline{X} - μ_{0}}{S / \sqrt{n}}$ has a $t$ null distribution with $n - 1$ degrees of freedom (the one-sample $t$ test).
Any population, large $n$ : $T$ is approximately standard Gaussian (by the CLT), and the $t$ test is approximately valid.

The p-value is the tail area beyond the observed statistic, in the direction(s) of $H_{1}$ . In R, t.test(x, mu = mu0) carries out case 2 (and, approximately, case 3).

Example 9.4: Testing the Vials ★ lecture example

Return to the filling machine of Example 9.2 ( $σ = 8$ mg, $n = 20$ ). This time the sample mean is $\overline{x} = 503.5$ mg. Test $H_{0} : μ = 500$ against $H_{1} : μ \neq 500$ at the 5% level.

Test statistic: $z = \frac{503.5 - 500}{8 / \sqrt{20}} = \frac{3.5}{1.789} \approx 1.957 .$

p-value (two-sided): $2 Pr (Z \geq 1.957)$ $\approx 0.0504$ .

z <- (503.5 - 500) / (8 / sqrt(20))
c(z = z, p = 2 * (1 - pnorm(z)))

output

         z          p 
1.95655948 0.05039928

Decision. $p = 0.0504$ is just above 0.05, so at the 5% level we fail to reject $H_{0}$ ; equivalently, $| z |$ $= 1.957$ $< 1.96$ .

Critical thinking. A result of $p = 0.0504$ and one of $p = 0.0496$ are, for all practical purposes, the same strength of evidence, yet one is "significant" and the other isn't. The 5% line is a convention, not a cliff. A sensible engineer would report the p-value (or better, the confidence interval for $μ$ : $503.5$ $\pm 3.51$ $= (499.99,$ $507.01)$ mg, which just contains 500, as it must when $p > 0.05$ ), weigh the cost of a drifting machine, and perhaps weigh a few more vials.

The p-value is an area: the probability in the two tails beyond the observed $z$ , under the null distribution.

z <- seq(-4, 4, by = 0.01); d <- data.frame(z, f = dnorm(z))
ggplot(d, aes(z, f)) +
  geom_area(data = subset(d, z >= 1.957), fill = "orange", alpha = 0.7) +
  geom_area(data = subset(d, z <= -1.957), fill = "orange", alpha = 0.7) +
  geom_line(linewidth = 1) +
  geom_vline(xintercept = c(-1.96, 1.96), linetype = "dashed") +
  annotate("text", x = 3.05, y = 0.09, label = "p/2 = 0.025") + annotate("text", x = -3.05, y = 0.09, label = "p/2 = 0.025") +
  labs(x = "z (if H0 is true)", y = "density")

Example 9.5: Student's Sleep Data, Tested ★ lecture example

In Example 8.11, the 10 patients in Cushny and Peebles's trial slept on average $\overline{d} = 1.58$ hours more on the second drug than on the first (SD 1.23). Test whether the mean difference is zero, at the 5% level.

$H_{0} : μ_{d} = 0$ (the drugs are equally effective) against $H_{1} : μ_{d} \neq 0$ . With $n = 10$ and σ unknown, use the $t$ test with 9 degrees of freedom: $t = \frac{1.58 - 0}{1.23 / \sqrt{10}} = \frac{1.58}{0.389} \approx 4.06 .$ The two-sided p-value is $2 Pr (T_{9} \geq 4.06)$ $\approx 0.003$ .

d <- with(sleep, extra[group == 2] - extra[group == 1])
t.test(d, mu = 0)

output


    One Sample t-test

data:  d
t = 4.0621, df = 9, p-value = 0.002833
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 0.7001142 2.4598858
sample estimates:
mean of x 
     1.58

Reject $H_{0}$ : the second drug gives more sleep. Notice that R's output also gives the 95% confidence interval from Example 8.11, (0.70, 2.46), which excludes 0, exactly as it must when $p < 0.05$ (Section 9.6). The interval says more than the test: not just "there's a difference", but "about 0.7 to 2.5 hours".

Comparing two groups

Many questions compare two independent groups: smokers and non-smokers, treatment and control. The same signal-over-noise idea applies, using Section 6.6's rule that variances of independent quantities add.

Theorem: The Two-Sample (Welch) T Test

For independent random samples of sizes $n_{1}$ and $n_{2}$ from two populations with means $μ_{1}$ and $μ_{2}$ , to test $H_{0} : μ_{1} = μ_{2}$ use $T = \frac{{\overline{X}}_{1} - {\overline{X}}_{2}}{\sqrt{\frac{S_{1}^{2}}{n_{1}} + \frac{S_{2}^{2}}{n_{2}}}} .$ Under $H_{0}$ , $T$ is approximately $t$ -distributed, with degrees of freedom given by a formula that R computes (the Welch approximation); for large samples it's approximately standard Gaussian. In R: t.test(y ~ group).

The denominator is the standard error of ${\overline{X}}_{1} - {\overline{X}}_{2}$ : each mean has variance $σ_{i}^{2} / n_{i}$ , and the variance of a difference of independent quantities is the sum of their variances.

Example 9.6: Smoking and Birth Weight ★ lecture example

In the Baystate Medical Center data of Chapter 1 (MASS::birthwt), the 74 babies of mothers who smoked during pregnancy weighed on average 2,772 g (SD 660 g), and the 115 babies of non-smokers 3,056 g (SD 753 g). (a) Test whether the mean birth weights differ. (b) Find a 95% confidence interval for the difference. (c) Does this show that smoking causes lower birth weight?

(a) The difference in means is $3056$ $- 2772$ $= 284$ g, with standard error $\sqrt{\frac{753^{2}}{115} + \frac{660^{2}}{74}}$ $\approx \sqrt{4931 + 5886}$ $\approx 104 g,$ so $t$ $= 284 / 104$ $\approx 2.73$ .

t.test(bwt ~ smoke, data = MASS::birthwt)

output


    Welch Two Sample t-test

data:  bwt by smoke
t = 2.7299, df = 170.1, p-value = 0.007003
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
  78.57486 488.97860
sample estimates:
mean in group 0 mean in group 1 
       3055.696        2771.919

R gives $t = 2.73$ with 170 degrees of freedom, and $p = 0.007$ . Reject $H_{0}$ at the 5% level: the mean birth weights differ.

(b) R's 95% confidence interval for $μ_{non-smokers}$ $- μ_{smokers}$ is (79, 489) g: smokers' babies were on average between about 80 and 490 g lighter.

bw <- MASS::birthwt
bw$mother <- factor(bw$smoke, labels = c("non-smoker", "smoker"))
ggplot(bw, aes(mother, bwt)) + geom_boxplot(width = 0.5, outlier.shape = NA) +
  geom_jitter(width = 0.12, alpha = 0.4, colour = "steelblue") +
  labs(x = NULL, y = "birth weight (g)")

(c) Not by itself. The test shows that the difference is unlikely to be a fluke of sampling, not why it exists. These are observational data: mothers who smoked may differ in income, diet, age or healthcare (the confounders of Chapter 1's DAG). Chapter 1 discussed exactly this example. The conclusion that smoking causes lower birth weight is now firmly established, but from many lines of evidence, including dose–response relationships, biological mechanisms and trials of programmes that help mothers quit. Statistical significance answers "could this be chance?"; it doesn't answer "what caused it?".

Check Yourself

For a two-sided $t$ test at the 5% level with a sample of 10, what critical value must $| t |$ exceed?

critical value =

Think About It: One-Sided or Two-Sided?

A researcher planned a two-sided test, got $p = 0.08$ , then noticed that the effect went in the direction she'd expected and switched to a one-sided test, getting $p = 0.04$ . Is this legitimate?

No. A one-sided p-value is half the two-sided one when the effect goes in the predicted direction, so switching after seeing the data always makes a result look stronger, and it doubles the real false-alarm rate. The alternative must be chosen before the data are seen, based on the scientific question: one-sided only when an effect in the other direction would be treated exactly like no effect (for example, a quality check where only underfilling matters). Pre-registering the analysis plan (Section 9.7) is the cure for this temptation, and for many others like it.

Key Takeaways

One-sample tests for a mean: $z$ $= (\overline{x}$ $- μ_{0}) / (σ / \sqrt{n})$ if σ is known; otherwise $t$ $= (\overline{x}$ $- μ_{0}) / (s / \sqrt{n})$ on $n - 1$ degrees of freedom.
Two independent groups: Welch's $t$ $= ({\overline{x}}_{1}$ $- {\overline{x}}_{2}) / \sqrt{s_{1}^{2} / n_{1} + s_{2}^{2} / n_{2}}$ . Paired data: test the differences.
A significant difference in observational data doesn't establish cause.

Lecture 24 — Tests for a proportion, errors and power

9.4 Tests for a proportion

Theorem: Tests for a Proportion

To test $H_{0} : p = p_{0}$ with $x$ successes in $n$ independent trials:

Large sample (both $n p_{0}$ and $n (1 - p_{0})$ at least 10): under $H_{0}$ , $Z = \frac{\hat{p} - p_{0}}{\sqrt{p_{0} (1 - p_{0}) / n}}$ is approximately standard Gaussian. Note that the standard error uses the null value $p_{0}$ , since the null distribution is computed assuming $H_{0}$ .
Exact: the p-value is a binomial tail probability computed with $p_{0}$ . In R: binom.test(x, n, p = p0).

The large-sample test is exactly the test that the Wilson interval of Section 8.5 inverts, which is one reason the Wilson interval is the better one.

Example 9.7: Is This Die Loaded? ★ lecture example

In Example 7.11, a die showed 120 sixes in 600 rolls. Test, at the 5% level, whether it favours six: $H_{0} : p = \frac{1}{6}$ against $H_{1} : p > \frac{1}{6}$ .

$\hat{p}$ $= 120 / 600$ $= 0.2$ , and under $H_{0}$ the standard error is $\sqrt{\frac{1}{6} \cdot \frac{5}{6} / 600}$ $\approx 0.0152$ , so $z$ $= \frac{0.2 - 0.1667}{0.0152}$ $\approx 2.19,$ $p$ $= Pr (Z \geq 2.19)$ $\approx 0.014 .$ The exact binomial test gives $p$ $= Pr (X \geq 120)$ $\approx 0.018$ (Example 7.11 approximated it as 0.016 using a continuity correction).

z <- (120/600 - 1/6) / sqrt(1/6 * 5/6 / 600)
c(z = z, p = 1 - pnorm(z))
binom.test(120, 600, p = 1/6, alternative = "greater")$p.value

output

         z          p 
2.19089023 0.01422987 
[1] 0.01801076

Reject $H_{0}$ at the 5% level: there is evidence that the die favours six. But notice what we haven't learned: how much it favours six. The 95% Wilson interval for $p$ is about (0.17, 0.23), which runs from "barely loaded" to "noticeably loaded". And if a casino tested 100 dice and this was the most extreme, a p-value of 0.014 would be unremarkable (Section 9.7).

k <- 70:135
ggplot(data.frame(k, p = dbinom(k, 600, 1/6), tail = k >= 120), aes(k, p, fill = tail)) +
  geom_col(width = 0.9) + scale_fill_manual(values = c("grey60", "orange"), guide = "none") +
  labs(x = "sixes in 600 rolls of a fair die", y = "probability")

Example 9.8: Was Kerrich's Coin Fair?

Kerrich tossed a coin 10,000 times and got 5,067 heads (Example 7.5). Test $H_{0} : p = 0.5$ against a two-sided alternative.

$z$ $= \frac{0.5067 - 0.5}{\sqrt{0.25 / 10000}}$ $= \frac{0.0067}{0.005}$ $= 1.34$ , and the two-sided p-value is $2 Pr (Z \geq 1.34)$ $\approx 0.18$ . Fail to reject $H_{0}$ : the data are consistent with a fair coin.

binom.test(5067, 10000, p = 0.5)

output


    Exact binomial test

data:  5067 and 10000
number of successes = 5067, number of trials = 10000, p-value
= 0.1835
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.4968503 0.5165458
sample estimates:
probability of success 
                0.5067

The exact test gives 0.18 too, and R's 95% confidence interval for $p$ is (0.497, 0.517). The coin might have a tiny bias, anywhere from about 0.3 percentage points towards tails to about 1.7 points towards heads, but 10,000 tosses can't tell. Real coins are very close to fair, though researchers who filmed coin tosses with high-speed cameras (Persi Diaconis and colleagues, 2007), and later a team who tossed coins about 350,000 times by hand (2023), found a slight tendency for a coin to land the same way up as it started, about 51% of the time.

Check Yourself

In 400 trials a forensic test gives 220 positives. Test $H_{0} : p = 0.5$ : compute the $z$ statistic.

z =

Key Takeaways

Proportion test: $z$ $= (\hat{p}$ $- p_{0}) / \sqrt{p_{0} (1 - p_{0}) / n}$ , using $p_{0}$ in the standard error; exact version binom.test.
A significant test tells you the null is implausible; a confidence interval tells you how big the effect might be.

9.5 Errors and power

Because a test is a decision made from random data, it can be wrong in two ways.

Definition: Type I and Type II Errors, and Power

	trueH₀ true	falseH₀ false
reject $H_{0}$	Type I error (false alarm)	correct (detection)
fail to reject $H_{0}$	correct	Type II error (missed effect)

$α$ $= Pr (reject H_{0} ∣ H_{0} true)$ , the significance level, is the Type I error rate.
$β$ , the Type II error rate, is $Pr (fail to reject H_{0})$ when a particular alternative is true.
The power of a test against an alternative is $1 - β$ : the probability of detecting the effect if it's really there.

Choosing $α$ fixes the false-alarm rate. The Type II error rate depends on four things: $α$ , the sample size $n$ , the noise $σ$ , and how big the true effect is. Reducing $α$ (being stricter) increases $β$ , for a fixed $n$ : you convict fewer innocent people but acquit more guilty ones. The only way to reduce both is more data, or less noise.

Example 9.9: How Likely Is the Engineer to Catch a Drift? ★ lecture example

The vial test of Example 9.4 uses $n = 20$ , $σ = 8$ mg, and a two-sided test at the 5% level. (a) If the machine has really drifted to $μ = 505$ mg, what is the power? (b) What if the engineer weighs 40 vials?

(a) Rejection region. We reject when $| \overline{x} - 500 |$ $> 1.96$ $\times 1.789$ $= 3.51$ mg, that is, when $\overline{x} > 503.51$ or $\overline{x} < 496.49$ .

Probability of landing there if $μ = 505$ . Now $\overline{X}$ $\sim N (505,$ ${1.789}^{2})$ , so $\begin{matrix} power & = Pr (\overline{X} > 503.51) + Pr (\overline{X} < 496.49) \\ = Pr (Z > \frac{503.51 - 505}{1.789}) + Pr (Z < \frac{496.49 - 505}{1.789}) \\ = Pr (Z > - 0.83) + Pr (Z < - 4.76) \approx 0.798 . \end{matrix}$ $power$ $= Pr (\overline{X} > 503.51)$ $+ Pr (\overline{X} < 496.49)$ $= Pr (Z > \frac{503.51 - 505}{1.789})$ $+ Pr (Z < \frac{496.49 - 505}{1.789})$ $= Pr (Z > - 0.83)$ $+ Pr (Z < - 4.76)$ $\approx 0.798 .$ An 80% chance of catching a 5 mg drift, and so a 20% chance ( $β$ ) of missing it.

(b) With $n = 40$ , the standard error falls to $8 / \sqrt{40} = 1.26$ and the power rises to about 0.98.

power_z <- function(mu1, n, sigma = 8, mu0 = 500, alpha = 0.05) {
  se <- sigma / sqrt(n); zc <- qnorm(1 - alpha / 2)
  1 - pnorm(zc - (mu1 - mu0) / se) + pnorm(-zc - (mu1 - mu0) / se) }
power_z(505, n = c(20, 40))

output

[1] 0.7981762 0.9768629

mu <- seq(490, 510, by = 0.1)
d <- rbind(data.frame(mu, power = power_z(mu, 20), n = "20 vials"),
           data.frame(mu, power = power_z(mu, 40), n = "40 vials"))
ggplot(d, aes(mu, power, linetype = n)) + geom_line(linewidth = 1) +
  geom_hline(yintercept = 0.05, colour = "grey60", linetype = "dotted") +
  annotate("text", x = 490, y = 0.12, label = "α = 0.05", hjust = 0, size = 3.9, colour = "grey40") +
  scale_y_continuous(limits = c(0, 1)) +
  labs(x = "true mean fill (mg)", y = "Pr(reject H0)", linetype = NULL)

The power curve shows the whole story: at the null value the "power" is just $α = 0.05$ , and it rises as the true mean moves away from 500, faster for the larger sample.

α, β and power

The vial-filling test of Examples 9.4 and 9.9: H₀: μ = 500 mg, with σ known. The solid blue curve is the distribution of the sample mean if H₀ is true; the dashed orange curve is its distribution if the machine has really drifted to μ₁. We reject when the sample mean lands in the red region. Red area under the blue curve is α (false alarm). The shaded orange area under the orange curve is β (a missed drift). Everything else under the orange curve is the power. The bottom panel shows the power for every possible true mean.

true mean μ₁ = mg sample size n = σ = mg

α = alternative

Planning a study: how many observations?

Before collecting data, a researcher should ask: if there really is an effect of the size I care about, how likely is my study to detect it? A study with low power is usually a waste of time and money, and of the participants' goodwill.

Theorem: Sample Size for a Z Test

To detect a difference $δ = | μ_{1} - μ_{0} |$ with power $1 - β$ , using a two-sided test at level $α$ with known σ, approximately $n = {(\frac{(z_{α / 2} + z_{β}) σ}{δ})}^{2},$ where $z_{β}$ is the upper- $β$ point of the standard Gaussian (0.84 for 80% power, 1.28 for 90%).

Ignore the far rejection region, which contributes almost nothing. Power $\approx Pr (Z > z_{α / 2} - \frac{δ}{σ / \sqrt{n}})$ . For this to equal $1 - β$ , we need $z_{α / 2}$ $- \frac{δ \sqrt{n}}{σ}$ $= - z_{β}$ , which rearranges to the formula. $◼$ ∎

Example 9.10: How Many Vials?

How many vials must the engineer weigh to have 90% power to detect a 5 mg drift ( $σ = 8$ , two-sided, $α = 0.05$ )?

$n$ $= {(\frac{(1.96 + 1.28) \times 8}{5})}^{2}$ $\approx 26.9,$ so 27 vials. For 80% power, ${(1.96 + 0.84)}^{2}$ $\times 64 / 25$ $\approx 20.1$ , so about 20 vials, matching Example 9.9. Notice the cost of certainty: going from 80% to 90% power needs a third more data, and halving the drift to be detected needs four times as much.

Check Yourself

A study has power 0.3 to detect the effect its authors care about. What does that mean? If the effect is real, the study has only a 30% chance of a significant result.There's a 30% chance that the effect is real.The false-alarm rate is 30%.

Think About It: Blackstone's Ratio

The English jurist William Blackstone wrote in the 1760s that "it is better that ten guilty persons escape than that one innocent suffer." Translate this into the language of Type I and Type II errors. What does it imply about the "significance level" of a criminal trial? Is a lower α always better?

With $H_{0}$ : "innocent", convicting an innocent person is a Type I error and acquitting a guilty one is a Type II error. Blackstone says a Type I error is at least ten times as costly as a Type II error, so the threshold for conviction should be strict (a small α), even at the price of more guilty people going free (a larger β). But α can't be pushed to zero without making conviction impossible, and acquitting the guilty has real costs to victims and to public safety. The right balance depends on the costs of each error, which is a moral and political judgment, not a statistical one. The same is true in medicine (approving a harmful drug vs. rejecting a useful one) and in forensic science (declaring a false match vs. missing a true one).

Key Takeaways

Type I error = false alarm (probability α); Type II error = missed effect (probability β); power = $1 - β$ .
Power grows with the sample size, the size of the effect and α, and falls with the noise σ.
Plan the sample size before collecting data: $n$ $\approx ((z_{α / 2} + z_{β}) σ / δ)^{2}$ .

Lecture 25 — Significance, many tests and replication

9.6 Tests and intervals; statistical and practical significance

Tests and confidence intervals are two views of one calculation

A two-sided $t$ test at level $α$ rejects $H_{0} : μ = μ_{0}$ exactly when $μ_{0}$ lies outside the $100 (1 - α) %$ $t$ confidence interval for $μ$ .

The test fails to reject when $| \frac{\overline{x} - μ_{0}}{s / \sqrt{n}} |$ $\leq t_{n - 1, α / 2}$ , that is, when $\overline{x}$ $- t_{n - 1, α / 2} \frac{s}{\sqrt{n}}$ $\leq μ_{0}$ $\leq \overline{x}$ $+ t_{n - 1, α / 2} \frac{s}{\sqrt{n}}$ : exactly when $μ_{0}$ is inside the interval. $◼$ ∎

So a 95% confidence interval is the set of null values that a 5%-level test would not reject. In Example 9.5, the interval (0.70, 2.46) excludes 0, so the test rejects $μ_{d} = 0$ . The interval is usually more informative than the test, because it also shows how big the effect might be.

Statistical significance is not practical significance

With enough data, any difference from $H_{0}$ , however tiny, becomes statistically significant, because the standard error shrinks like $1 / \sqrt{n}$ while the effect stays the same size.

Example 9.11: A Significant Trivial Effect ★ lecture example

An online test-prep company tests 1,000,000 students and finds that their average score on an IQ-style test, scaled to have mean 100 and SD 15 in the population, is 100.08. Test $H_{0} : μ = 100$ . Is the company's course worth paying for?

$z$ $= \frac{100.08 - 100}{15 / \sqrt{1,000,000}}$ $= \frac{0.08}{0.015}$ $\approx 5.3,$ $p$ $\approx 10^{- 7} .$ Overwhelmingly significant. But the estimated effect is 0.08 points, about $\frac{1}{200}$ of a standard deviation: no student, teacher or employer could ever notice it. (And since these students weren't compared with a randomized control group, even that tiny difference might reflect who chose to take the course.)

Report the effect, not just the p-value. A confidence interval, here roughly $100.08 \pm 0.03$ , shows at once that the effect is precisely estimated and practically negligible. A common standardized measure of effect size is the difference in means divided by the SD (Cohen's $d$ ); here $d \approx 0.005$ , where conventional benchmarks call 0.2 "small".

Watch Out: Significant Isn't Important, and Non-Significant Isn't Zero

A tiny p-value can come from a trivial effect measured on a huge sample (Example 9.11).
A large p-value can come from an important effect measured on a small sample (the blood-pressure trial in Section 9.2's think box).
Always report an estimate of the effect with a confidence interval, and judge its size against what matters in context.

A picture makes the four possibilities clear. Each line is a 95% confidence interval for an effect. The solid vertical line is "no effect"; the dashed line is the smallest effect that would matter in practice.

d <- data.frame(study = factor(c("A: significant, important", "B: significant, trivial",
                                 "C: not significant, inconclusive", "D: not significant, small at most"),
                               levels = rev(c("A: significant, important", "B: significant, trivial",
                                              "C: not significant, inconclusive", "D: not significant, small at most"))),
                est = c(8, 1, 4, 0.3), lo = c(6, 0.5, -3, -0.8), hi = c(10, 1.5, 11, 1.4))
ggplot(d, aes(est, study)) +
  geom_vline(xintercept = 0) + geom_vline(xintercept = 5, linetype = "dashed") +
  geom_errorbarh(aes(xmin = lo, xmax = hi), height = 0.25, linewidth = 1) + geom_point(size = 3) +
  labs(x = "effect", y = NULL)

History: 0.05, Fisher, and the Debate Over Significance

Ronald Fisher's book Statistical Methods for Research Workers (1925) made the p-value and the 5% level standard tools of science. Fisher regarded 5% as a convenient rule of thumb for flagging results worth following up, not as a line between truth and falsehood. In 1928–1933 Jerzy Neyman and Egon Pearson (Karl Pearson's son) developed a different theory: fix α in advance, choose the test with the greatest power, and treat the outcome as a decision, with no interpretation of the p-value as evidence. Fisher and Neyman feuded for the rest of their lives, and the hybrid of the two approaches taught in most textbooks (including, in places, this one) is something neither would have fully endorsed.

Criticism has grown in recent decades. In 2016 the American Statistical Association issued its first-ever formal statement on a statistical method, warning that "a p-value, or statistical significance, does not measure the size of an effect or the importance of a result" and that "scientific conclusions ... should not be based only on whether a p-value passes a specific threshold." In Bernoulli's Fallacy (2021), Aubrey Clayton goes further: he argues that significance testing is logically flawed (it uses $Pr (data ∣ H_{0})$ where we need $Pr (H_{0} ∣ data)$ ), and that the framework's development by Galton, Pearson and Fisher was shaped by their eugenic projects, which needed methods that looked objective (Sections 2.3, 4.6 and 5.3 tell that history). Many statisticians who would not go as far as Clayton agree that p-values are overused and misread, and recommend reporting estimates and intervals, and thinking about prior evidence, alongside them.

Check Yourself

A 95% confidence interval for a mean difference is (−1.2, 3.8). What does a two-sided test of "no difference" at the 5% level conclude? Reject

H_{0}

, since most of the interval is positive.Fail to reject

H_{0}

, since 0 is inside the interval.Nothing can be said without the p-value.

Key Takeaways

A 95% CI contains exactly the null values that a 5%-level two-sided test would not reject.
Statistical significance depends on sample size; practical significance depends on the size of the effect.
Report estimates with intervals, not just p-values or "significant / not significant".

9.7 Testing in a world of many tests

Multiple comparisons

Each test at level 0.05 has a 5% chance of a false alarm when $H_{0}$ is true. Run 20 independent tests of true nulls, and the chance of at least one false alarm is $1$ $- {0.95}^{20}$ $\approx 0.64$ . Search through enough data and "significant" findings are guaranteed. We met this before with Paul the octopus (Section 4.4), cancer clusters (Section 4.7) and blood-test reference ranges (Section 5.3).

Example 9.12: The Dead Salmon ★ lecture example

In 2009, the neuroscientist Craig Bennett and colleagues placed a dead Atlantic salmon in an MRI scanner and showed it photographs of people in social situations, "asking" it to judge the people's emotions. They analysed the scan as a typical brain-imaging study would, testing each of thousands of small brain regions (voxels) separately for a response at a strict per-test threshold. Several voxels in the salmon's brain "lit up". (a) Why? (b) What should the analysis have done?

(a) A typical brain scan contains about 130,000 voxels. With that many tests, even a per-test false-alarm rate of 1 in 1,000 gives about 130 false alarms expected by chance. Some of them happened to cluster together in the salmon's brain cavity. The fish was not thinking.

(b) Correct for multiple comparisons. The simplest method, the Bonferroni correction, tests each of $m$ hypotheses at level $α / m$ , which guarantees that the chance of any false alarm is at most $α$ (by the union bound of Section 3.3, extended to $m$ events: $Pr (any)$ $\leq \sum Pr (each)$ $= m$ $\times α / m$ ). With corrections applied, nothing in the salmon was significant. The authors' point, which won them an Ig Nobel prize in 2012, was that many published brain-imaging studies at the time didn't correct properly.

1 - 0.95^20              # chance of at least one false alarm in 20 tests
0.05 / 20                # Bonferroni threshold for 20 tests

output

[1] 0.6415141
[1] 0.0025

How many significant results are true?

Suppose a field tests many hypotheses, of which only a fraction are real effects. Among the results that come out significant, what fraction are real? This is Bayes' theorem (Section 3.6), with "significant" playing the role of a positive test result.

How many significant results are real?

Imagine a field that tests 400 hypotheses. Only some of them are real effects. Each square is one study. Real effects are detected with probability equal to the power; true nulls come out significant with probability α. Among the significant results, what fraction are false alarms? This is Bayes' theorem from Chapter 3 in disguise: the answer depends on how many effects were real to begin with.

real effects = % power = % α =

If 10% of tested hypotheses are true, tests have power 0.8, and $α = 0.05$ , then out of 1,000 hypotheses, 100 are real and 80 of those come out significant; 900 are null and 45 of those come out significant. So $45 / 125 = 36 %$ of the significant results are false alarms. With power 0.3, typical of small studies, the false fraction rises to $45 / 75 = 60 %$ . In 2005 the epidemiologist John Ioannidis used exactly this arithmetic in a famous paper titled "Why most published research findings are false".

The p-values themselves show what's happening. When every null hypothesis is true, p-values are spread evenly between 0 and 1, and 5% of them fall below 0.05 by chance. When some effects are real, those p-values pile up near 0, on top of the flat floor of false alarms:

set.seed(9)
p_null <- 2 * (1 - pnorm(abs(rnorm(10000))))
real <- runif(10000) < 0.1
z <- rnorm(10000, mean = ifelse(real, 2.8, 0))           # an effect of 2.8 SEs gives power about 0.8
p_mix <- 2 * (1 - pnorm(abs(z)))
d <- rbind(data.frame(p = p_null, world = "all nulls true"),
           data.frame(p = p_mix, world = "10% of effects real"))
d$world <- factor(d$world, levels = c("all nulls true", "10% of effects real"))
ggplot(d, aes(p)) + geom_histogram(breaks = seq(0, 1, by = 0.05), colour = "white") +
  facet_wrap(~ world) + scale_x_continuous(breaks = c(0, 0.5, 1)) +
  theme(panel.spacing = unit(1.5, "lines")) + labs(x = "p-value", y = "number of tests")

History: The Replication Crisis

In 2015 the Open Science Collaboration, a group of 270 researchers, repeated 100 studies published in three leading psychology journals, following the original methods as closely as possible. Of the original studies, 97% had reported significant results; of the replications, only about 36% did, and the effects found were on average about half the size of the originals. Similar projects in cancer biology, economics and other fields have found related problems.

The causes are the ones in this chapter: small, underpowered studies; many analyses tried and only the significant ones reported (sometimes called "p-hacking"); journals that publish significant results and reject null ones; and the misreading of $p < 0.05$ as proof. The remedies are also statistical: larger samples, pre-registration of hypotheses and analysis plans before the data are collected, publishing null results, reporting effect sizes with confidence intervals, and above all replication. One significant result is the start of an investigation, not the end.

Bayesian Inference: What Does P = 0.05 Really Tell You?

A p-value isn't the probability that $H_{0}$ is true, but we can ask how much it could shift that probability. The statisticians Thomas Sellke, M. J. Bayarri and James Berger showed in 2001 that, under broad conditions, a p-value of $p$ can change the odds on $H_{0}$ by at most a factor of about $- e p ln p$ (the Bayes factor bound, for $p < 1 / e$ ). For $p = 0.05$ that factor is about 0.41: if $H_{0}$ and $H_{1}$ start equally likely, the probability of $H_{0}$ can fall no lower than about 29%.

bf_bound <- function(p) -exp(1) * p * log(p)
p <- c(0.05, 0.01, 0.005, 0.001)
data.frame(p, smallest_posterior_H0 = round(bf_bound(p) / (1 + bf_bound(p)), 3))

output

      p smallest_posterior_H0
1 0.050                 0.289
2 0.010                 0.111
3 0.005                 0.067
4 0.001                 0.018

So " $p = 0.05$ " is much weaker evidence than it sounds: at best, it moves 50–50 odds to about 70–30. That's one reason some statisticians have proposed requiring $p < 0.005$ for claims of new discoveries, and why Bayesian analyses report the posterior probability of hypotheses directly.

Causal Inference: Randomized Trials and Observational Tests

A significance test answers "could this difference be due to sampling chance?". In a randomized controlled trial, where a coin toss decides who gets the treatment, that's the only alternative explanation left: randomization balances the confounders on average (Chapter 1), so a significant difference can be attributed to the treatment. In an observational study, significance rules out chance but not confounding.

The most famous cautionary tale is hormone replacement therapy (HRT) for women after menopause. Large observational studies in the 1980s and 1990s found that women taking HRT had significantly fewer heart attacks, and HRT was widely prescribed partly for that reason. But the women who chose HRT tended to be wealthier, more health-conscious and better cared for. In 2002, the Women's Health Initiative randomized trial, with about 16,000 women, was stopped early: the group given HRT had more heart attacks, strokes and breast cancers, not fewer.

Observational studies compared women who chose therapy with women who did not; the randomized trial let a coin decide.

The observational p-values were small, and the conclusion was wrong. A test is only as good as the comparison it tests.

Check Yourself

A researcher runs 20 tests and wants the chance of any false alarm to be at most 5%. What per-test threshold does the Bonferroni correction use?

threshold =

Think About It: The Garden of Forking Paths

A researcher studying whether a new policing strategy reduced crime can choose which crimes to count (all, violent only, property only), which time window to use (6 months, 1 year, 2 years), and which comparison neighbourhoods to include (3 options). She tries every combination and reports the one with the smallest p-value, 0.03. Roughly how many analyses did she effectively run? How much should we trust the result, and what would make it trustworthy?

$3 \times 3 \times 3 = 27$ analyses. Even if the policy did nothing, the chance that at least one of 27 analyses gives $p < 0.05$ is high (though the analyses use overlapping data, so it's less than $1$ $- {0.95}^{27}$ $\approx 0.75$ ). The reported $p = 0.03$ is not the false-alarm rate of what she actually did, so it means very little. The statisticians Andrew Gelman and Eric Loken call this the "garden of forking paths": the problem arises even without any intent to deceive, whenever choices are made after seeing the data. The result would be trustworthy if the analysis had been pre-registered (chosen and published before seeing the data), if all 27 results were reported, or if the finding replicated in a new city with the same analysis fixed in advance.

Key Takeaways

Many tests produce false alarms: the chance of at least one among $m$ independent true nulls is $1 - {(1 - α)}^{m}$ . Bonferroni: test each at $α / m$ .
The fraction of significant results that are real depends on power and on how many tested hypotheses were true to begin with (Bayes' theorem).
Only randomization rules out confounding; significance only rules out chance.
Replication and pre-registration are the best protection against false findings.

Prediction and inference in this chapter

Inference. A hypothesis test asks whether the data contradict a claim about the mechanism: a fair die, a machine on target, no difference between drugs. The p-value measures how surprising the data would be under the claim. We tested means (one and two groups) and proportions, and saw that a confidence interval contains the same information and more.
Decisions. Testing is also a decision rule, with two kinds of error. Power analysis predicts, before any data are collected, how likely a study is to detect an effect of a given size, which is a prediction about the study itself.
The limits. A significant result can be trivial, a non-significant one can hide a real effect, many tests guarantee false alarms, and in observational data significance doesn't rule out confounding. The salmon, the replication crisis and HRT all show what happens when these limits are forgotten.

9.8 Interludes

Interlude (Optional): The Neyman–Pearson Lemma: The Most Powerful Test

Among all tests with false-alarm rate α, which has the greatest power? For the simplest case, a null value $θ_{0}$ against a single alternative $θ_{1}$ , Neyman and Pearson proved in 1933 that the best test rejects $H_{0}$ when the likelihood ratio $\frac{L (θ_{1})}{L (θ_{0})}$ is large, with the cutoff chosen to make the false-alarm rate exactly α. In words: reject the null when the data are much better explained by the alternative than by the null. For the Gaussian mean, the likelihood ratio is large exactly when $\overline{x}$ is far from $μ_{0}$ in the direction of $μ_{1}$ , so the one-sided $z$ test of Section 9.3 (in the direction of $μ_{1}$ ) is the most powerful test. The likelihood (Section 8.3) thus underlies both estimation and testing.

Why it matters: the lemma explains why the standard tests are built the way they are, and the likelihood ratio is also the key quantity in Bayesian testing (it's the Bayes factor for two simple hypotheses) and in forensic science, where DNA and fingerprint evidence is increasingly reported as a likelihood ratio.

Interlude (Optional): Testing a Whole Distribution: Pearson's Chi-Squared

In Section 4.7 we compared the V-1 bomb hits on 576 squares of south London with a Poisson model by eye. Karl Pearson's chi-squared test (1900) makes the comparison formal. With observed counts $O_{k}$ and expected counts $E_{k}$ in each category, $X^{2} = \sum_{k} \frac{{(O_{k} - E_{k})}^{2}}{E_{k}}$ has approximately a chi-squared distribution (Section 5.6) if the model is right, with degrees of freedom equal to the number of categories minus 1, minus the number of parameters estimated from the data.

obs <- c(229, 211, 93, 35, 8)                    # squares with 0, 1, 2, 3, 4+ hits
lambda <- 537 / 576
expected <- 576 * c(dpois(0:3, lambda), 1 - ppois(3, lambda))
round(expected, 1)
X2 <- sum((obs - expected)^2 / expected)
c(X2 = X2, p = 1 - pchisq(X2, df = 5 - 1 - 1))

output

[1] 226.7 211.4  98.5  30.6   8.7
       X2         p 
1.0175894 0.7969959

Here $X^{2} \approx 1.0$ on 3 degrees of freedom, with $p \approx 0.80$ : the data fit the Poisson model very well, which is the formal version of Clarke's conclusion that the bombs fell at random.

Why it matters: the chi-squared test checks whether a whole model fits, not just one parameter, and the same statistic tests independence in two-way tables such as hair and eye colour (Section 6.1). It's one of the oldest tests still in everyday use.

Interlude (Optional): Looking Back: What This Course Was for

The course began with two goals, prediction and inference, and with a warning that the tools of statistics have a complicated history. Prediction asked for good guesses with errors small enough to live with: an airline's overbooking, an elevator's capacity, a son's height, a casino's profit. Inference asked what the data say about the mechanism behind them: whether a die is loaded, whether a drug works, how many tanks an enemy has, whether bombs were aimed.

The same ideas kept returning. Variation is not a nuisance but the subject. Averages are predictable, individuals are not. Every estimate has a standard error, and every interval assumes no bias. Conditional probabilities run in two directions, and confusing them convicts innocent people. Correlation, regression and significance tests measure association, while the causes that produced the data have to come from the design of the study and from knowledge of the mechanism.

Some of these tools were first developed by people who used them to argue for ranking and controlling human beings. The mathematics was sound; the arguments were not, and they did real harm. The best protection against that misuse was never to abandon the tools but to understand them well enough to see what they can and cannot show. That understanding, statistical literacy, is what this course has tried to give you.

9.9 Exercises

Each exercise has a hint and an answer. For every test, state $H_{0}$ and $H_{1}$ , the test statistic, its null distribution, the p-value and a conclusion in plain words.

Exercises

Coin flips. A coin lands heads 8 times in 10 flips. Find the exact one-sided p-value for $H_{0} : p = 0.5$ against $H_{1} : p > 0.5$ , and the two-sided p-value. What do you conclude at the 5% level?

$Pr (X \geq 8)$ for $X$ $\sim Bin (10,$ $0.5)$ ; use binom.test(8, 10).

One-sided $56 / 1024$ $\approx 0.055$ ; two-sided about 0.109. Fail to reject at 5% either way.
Lead levels. Battery-plant workers' blood lead levels have SD 5 µg/dL. The safety target is a mean of 25. A sample of 36 workers averages 26.8. Test $H_{0} : μ = 25$ against $H_{1} : μ > 25$ at the 5% level.

$z$ $= (26.8$ $- 25) / (5 / 6)$ .

$z = 2.16$ , one-sided $p \approx 0.015$ . Reject $H_{0}$ : there's evidence that the mean exceeds the target.
Detection times. An intrusion detection system's times (ms) for 10 attacks are 142, 157, 149, 160, 151, 153, 158, 155, 150, 147. The benchmark is 155 ms. Test $H_{0} : μ = 155$ against a two-sided alternative at the 5% level, assuming Gaussian times.

$\overline{x} = 152.2$ , $s = 5.51$ ; use the $t$ test with 9 degrees of freedom, or t.test(x, mu = 155).

$t$ $= - 2.8 / (5.51 / \sqrt{10})$ $\approx - 1.61$ , $p \approx 0.14$ . Fail to reject: the data are consistent with a mean of 155 ms.
Fingerprint examiners. In a proficiency test, examiners made 22 false identifications in 400 comparisons where the prints came from different people. A lab claims its error rate is 3%. Test $H_{0} : p = 0.03$ against $H_{1} : p > 0.03$ .

$\hat{p} = 0.055$ ; null SE $= \sqrt{0.03 \times 0.97 / 400}$ .

SE $\approx 0.0085$ , $z \approx 2.93$ , $p \approx 0.002$ . Strong evidence that the error rate exceeds 3%. (The exact binomial test, binom.test(22, 400, 0.03, alternative = "greater"), gives $p \approx 0.005$ : the Gaussian approximation is noticeably optimistic in the tail, by a factor of about 3, when $n p_{0}$ is only 12.)
Two groups of plants. R's PlantGrowth gives dried plant weights for a control group and two treatments. Use t.test(weight ~ group, data = subset(PlantGrowth, group %in% c("ctrl", "trt2"))) to test whether treatment 2 changes the mean weight. Report the estimate, the confidence interval and the p-value.

The output gives all three. Interpret the interval in grams.

Control mean 5.03, treatment 2 mean 5.53: $t \approx - 2.13$ , $p \approx 0.048$ , 95% CI for control minus treatment about $(- 0.98,$ $- 0.005)$ . Just significant at 5%; the increase could be anywhere from almost nothing to about 1 gram.
Power. For the vial test (σ = 8, two-sided, α = 0.05), find the power against $μ = 503$ mg with $n = 20$ and with $n = 60$ .

Use power_z from Example 9.9, or compute $Pr (Z > 1.96$ $- 3 / (8 / \sqrt{n}))$ plus the small other tail.

$n = 20$ : SE 1.79, power $\approx Pr (Z > 0.28)$ $\approx 0.39$ . $n = 60$ : SE 1.03, power $\approx Pr (Z > - 0.94)$ $\approx 0.83$ .
Sample size. A trial wants 80% power to detect a 4 mmHg reduction in blood pressure, where individual changes have SD 12 mmHg, with a two-sided 5% test of one group's mean change. How many patients are needed?

$n$ $= {(2.8 \times \frac{12}{4})}^{2}$ , since $1.96$ $+ 0.84$ $= 2.8$ .

${(2.8 \times 3)}^{2}$ $= 70.6$ , so 71 patients. The 10-patient trial in Section 9.2's think box had power of only about 0.2.
Errors in context. For each situation, describe the Type I and Type II errors and say which is worse: (a) a smoke detector ( $H_{0}$ : no fire); (b) a DNA test comparing a suspect with crime-scene evidence ( $H_{0}$ : not the source); (c) an airport security scanner ( $H_{0}$ : no weapon).

Type I: reject $H_{0}$ when it's true. Type II: fail to reject when it's false.

(a) Type I: alarm with no fire (annoying); Type II: no alarm during a fire (deadly), so detectors are tuned to accept many false alarms. (b) Type I: declaring a match for an innocent person (wrongful conviction); Type II: missing the true source. The legal system treats Type I as worse. (c) Type I: searching an innocent passenger; Type II: missing a weapon. Screening accepts many false alarms, followed by a second, more careful check.
Interpreting p. For each statement about a study that found $p = 0.03$ , say whether it's correct. (a) "The probability that the null hypothesis is true is 0.03." (b) "If the null hypothesis were true, results at least this extreme would occur about 3% of the time." (c) "The effect is large." (d) "If we repeat the study, we'll almost certainly get $p < 0.05$ again."

Recall the warning box in Section 9.2 and the p-value widget.

(a) Wrong. (b) Correct. (c) Wrong: p doesn't measure effect size. (d) Wrong: p-values vary a lot; if the true effect is just big enough to give $p = 0.03$ , a replication has only about a 50–60% chance of $p < 0.05$ .
Many tests. A researcher tests 50 food items for a link with a disease, at the 5% level, and finds 3 significant. If none of the foods truly matters, how many significant results would be expected? Is 3 surprising? What would the Bonferroni threshold be?

Expected false alarms $= 50 \times 0.05$ ; the count is roughly $Bin (50, 0.05)$ .

2.5 expected; $Pr (X \geq 3)$ $\approx 0.46$ , so 3 is not surprising at all. Bonferroni: $0.05 / 50$ $= 0.001$ per test.
Duality. A 95% confidence interval for a mean is (4.2, 7.9). For which of these null values would a two-sided 5% test reject: 4, 5, 8? Explain.

Reject exactly when the null value is outside the interval.

Reject $μ_{0} = 4$ and $μ_{0} = 8$ , both outside the interval; don't reject $μ_{0} = 5$ .
The false discovery rate. In a field where 20% of tested hypotheses are true, studies have power 0.5 and use $α = 0.05$ . What fraction of significant results are false? What if power rises to 0.9?

Out of 1,000 hypotheses: 200 true (how many detected?) and 800 null (how many false alarms?).

Power 0.5: 100 true detections and 40 false alarms, so $40 / 140 \approx 29 %$ false. Power 0.9: 180 and 40, so $40 / 220 \approx 18 %$ .
Chi-squared for a die. A die is rolled 120 times, giving 1 to 6 with counts 15, 18, 22, 17, 21, 27. Use Pearson's chi-squared test (Section 9.8) to test whether the die is fair.

Each expected count is 20; 6 categories, no parameters estimated, so 5 degrees of freedom. chisq.test(c(15, 18, 22, 17, 21, 27)).

$X^{2}$ $= (25$ $+ 4$ $+ 4$ $+ 9$ $+ 1$ $+ 49) / 20$ $= 4.6$ on 5 df, $p \approx 0.47$ . No evidence against fairness.
A trial or an observational study? (Compare Chapter 1, Exercise 7.) A city finds that neighbourhoods with more street lights had significantly fewer night-time robberies ( $p = 0.001$ ). (a) Does this show that lighting reduces robbery? (b) Design a study that could.

Think of confounders (Chapter 1) and of how to randomize.

(a) No: wealthier neighbourhoods may have both more lights and fewer robberies; significance only rules out sampling chance. (b) Randomly choose which of a set of similar neighbourhoods get new lighting, and compare robberies before and after with the others. A study like this was carried out in New York City public housing developments, starting in 2016.

Chapter 9Hypothesis testing

Think About It: Before You Start

Lecture 23 — The logic of a test, p-values and tests for a mean

9.1 The logic of a test

Definition: Null and Alternative Hypotheses

Definition: Test Statistic and Null Distribution

Example 9.1: Boys and Girls in 1710 ★ lecture example

Example 9.2: Has the Machine Drifted? ★ lecture example

History: Popper and the Black Swan

Check Yourself

Think About It: Who Carries the Burden of Proof?

Key Takeaways

9.2 p-values

Definition: p-Value

Watch Out: What a p-Value Is Not

Example 9.3: Evidence for Precognition? ★ lecture example

Check Yourself

Think About It: No Evidence of an Effect

Key Takeaways

9.3 Tests for a mean

Theorem: Tests for a Population Mean

Example 9.4: Testing the Vials ★ lecture example

Example 9.5: Student's Sleep Data, Tested ★ lecture example

Comparing two groups

Theorem: The Two-Sample (Welch) T Test

Example 9.6: Smoking and Birth Weight ★ lecture example

Check Yourself

Think About It: One-Sided or Two-Sided?

Key Takeaways

Lecture 24 — Tests for a proportion, errors and power

9.4 Tests for a proportion

Theorem: Tests for a Proportion

Example 9.7: Is This Die Loaded? ★ lecture example

Example 9.8: Was Kerrich's Coin Fair?

Check Yourself

Key Takeaways

9.5 Errors and power

Definition: Type I and Type II Errors, and Power

Example 9.9: How Likely Is the Engineer to Catch a Drift? ★ lecture example

Planning a study: how many observations?

Theorem: Sample Size for a Z Test

Example 9.10: How Many Vials?

Check Yourself

Think About It: Blackstone's Ratio

Key Takeaways

Lecture 25 — Significance, many tests and replication

9.6 Tests and intervals; statistical and practical significance

Tests and confidence intervals are two views of one calculation

Statistical significance is not practical significance

Example 9.11: A Significant Trivial Effect ★ lecture example

Watch Out: Significant Isn't Important, and Non-Significant Isn't Zero

History: 0.05, Fisher, and the Debate Over Significance

Check Yourself

Key Takeaways

9.7 Testing in a world of many tests

Multiple comparisons

Example 9.12: The Dead Salmon ★ lecture example

How many significant results are true?

History: The Replication Crisis

Bayesian Inference: What Does P = 0.05 Really Tell You?

Causal Inference: Randomized Trials and Observational Tests

Check Yourself

Think About It: The Garden of Forking Paths

Key Takeaways

Prediction and inference in this chapter

9.8 Interludes

Interlude (Optional): The Neyman–Pearson Lemma: The Most Powerful Test

Interlude (Optional): Testing a Whole Distribution: Pearson's Chi-Squared

Interlude (Optional): Looking Back: What This Course Was for

9.9 Exercises

Exercises

Chapter 9
Hypothesis testing