Charta is on Google Play: practise on the go. Get the app
Charta

Explain why running many tests together is problematic and how it can distort results.

FRM® Part I · Quantitative Analysis · Hypothesis Testing · FRM1.QA.6.h

The idea

When many hypotheses are tested together, each test at 5 percent has a 5 percent false-positive chance, but the chance that at least one true null is rejected grows quickly. For m independent true nulls, that chance is given by this formula: Chance = 1 − (1 − α)^m. It is about 64 percent for 20 tests. The expected number of false rejections is m × α. Searching across many strategies, factors or models and reporting the best, known as data snooping, makes a chance result look like a discovery. A simple fix is the Bonferroni correction: test each hypothesis at α ÷ m, or equivalently multiply each p-value by m. It protects against false positives but lowers power. The exam point is that an unadjusted p-value of the best of many tests overstates the evidence.

Formulas

Chance of at least one false rejection across m independent tests

FWER = 1 − (1 − α)ᵐ

α
the significance level of each test
m
the number of independent true null hypotheses tested

All nulls true and tests independent. The expected number of false rejections is m × α; the trap is quoting the best p-value as if only one test had been run.

Bonferroni threshold for each of m tests

per-test level = α ÷ m

α
the overall significance level wanted
m
the number of hypotheses tested

Keeps the chance of any false rejection at or below α. Equivalently multiply each p-value by m and compare with α; the price is lower power.

A practice question

From the question bank, with the reason each wrong option is wrong.

What problem arises when a researcher runs many hypothesis tests on the same dataset, each at the 5 percent level?

  1. Each individual test becomes biased toward failing to reject the null.Why not: Gets the direction wrong; the danger is that spurious rejections accumulate.
  2. The Type II error rate of every test falls to zero.Why not: Multiple testing does not eliminate Type II errors; it inflates the chance of false positives.
  3. Each test's significance level automatically becomes 5 percent divided by the number of tests.Why not: No automatic adjustment occurs; the researcher must apply a correction such as Bonferroni.
  4. The chance of at least one false rejection across the tests rises well above 5 percent. - correct

Each test carries a 5 percent false-positive risk, so over many tests the chance that at least one true null is rejected (the family-wise error rate) grows.

A harder one

A quant tests 20 independent trading signals, each at the 5 percent level. Suppose none of them has any real predictive power. The probability that at least one is declared significant is closest to:

  1. 100 percentWhy not: Adds the probabilities (20 x 5 percent), which gives 100 percent and ignores that the events overlap.
  2. 36 percentWhy not: Computes the probability that none is significant, 0.95^20, not at least one.
  3. 64 percent - correct
  4. 5 percentWhy not: Reports the single-test false-positive rate, ignoring that there are 20 tests.

P(at least one) = 1 - 0.95^20 = 1 - 0.358 = 0.642.

The hardest tier

A researcher tests 60 independent signals each at 5 percent and finds 3 significant. If all 60 nulls were in fact true, what is the expected number of significant results, and the probability of at least one?

  1. Expected 3.0, so the finding is what chance alone produces; the probability of at least one is about 95 percent. - correct
  2. Expected 3.0; the probability of at least one is about 4.6 percent.Why not: Reports 0.95^60, the probability that none is significant, rather than at least one.
  3. Expected 3.0; the probability of at least one is 5 percent.Why not: Uses the single-test level and ignores that there are 60 tests.
  4. Expected 6.0; the probability of at least one is about 99.8 percent.Why not: Doubles alpha to 10 percent as if the two-sided test used both tails twice.

Expected false rejections = 60 x 0.05 = 3.0, matching what was found, so there is no evidence of any real signal. P(at least one) = 1 - 0.95^60 = 0.954.

In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.

Start free →

More in Hypothesis Testing