Charta is on Google Play: practise on the go. Get the app
Charta

Distinguish Type I from Type II errors, and relate each to a test's size and power.

FRM® Part I · Quantitative Analysis · Hypothesis Testing · FRM1.QA.6.c

The idea

Two mistakes are possible. A Type I error rejects a null that is true. Its probability is the size of the test, which equals the significance level α you chose. A Type II error fails to reject a null that is false. Its probability is β. Power = 1 − β. Power is the chance of correctly rejecting a false null. For a given sample size the two errors trade off. Lowering α makes rejection harder, so β rises and power falls. Power increases with a larger sample, a larger true departure from the null, and lower variability. The exam point is that failing to reject a hypothesis with a low-power test tells you little, since a false null would often survive it.

Formulas

Power of a test

Power = 1 − β

β
the probability of a Type II error

For a given sample size, lowering α raises β and so lowers power. Power rises with a larger sample, a larger true departure and lower variability.

A practice question

From the question bank, with the reason each wrong option is wrong.

A test has a probability of Type II error of 0.30. Which statement about its power is correct?

  1. Power is 0.30, the probability of rejecting the null hypothesis when it is false.Why not: Reports beta as if it were power; power is 1 minus beta.
  2. Power is 0.70, the probability of rejecting the null hypothesis when it is false. - correct
  3. Power is 0.70, the probability of failing to reject the null hypothesis when it is true.Why not: Mislabels the event; failing to reject a true null has probability 1 minus alpha, not 1 minus beta.
  4. Power is 0.30, the probability of rejecting a true null hypothesis.Why not: Confuses the Type II probability with the size of the test, which is a Type I concept.

Power = 1 - beta = 1 - 0.30 = 0.70, the probability of correctly rejecting a false null.

A harder one

A regulator's backtest rejects a VaR model that in fact is accurate. Which error has been made, and what controls the probability of making it?

  1. Type II error; the significance level (size) of the testWhy not: Rejecting a true null is a Type I error; a Type II error is failing to reject a false null.
  2. Type I error; the power of the testWhy not: Names the right error but power governs the chance of rejecting a false null, not the size of this error.
  3. Type I error; the significance level (size) of the test - correct
  4. Type II error; the power of the testWhy not: Gets both wrong: the error is Type I and its probability is controlled by the significance level.

The null (model is accurate) is true and was rejected, which is a Type I error. Its probability is the significance level alpha chosen for the test.

The hardest tier

A test of H0: mu = 0 against H1: mu > 0 uses a 5 percent significance level, a population standard deviation of 10 and n = 64 independent observations (normal approximation). If the true mean is 3, the power of the test is closest to:

  1. 22.5 percentWhy not: Reports the probability of a Type II error (beta) instead of its complement, the power.
  2. 77.5 percent - correct
  3. 67.0 percentWhy not: Uses the two-sided critical value 1.96 for what is a one-sided test.
  4. 53.8 percentWhy not: Standardizes by sigma = 10 rather than the standard error 10 / 8 = 1.25.

SE = 10/8 = 1.25. Reject if the sample mean exceeds 1.645 x 1.25 = 2.056. With true mean 3, P(mean > 2.056) = P(Z > (2.056 - 3)/1.25) = P(Z > -0.755) = 0.775.

In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.

Start free →

More in Hypothesis Testing