Recognise Type I and Type II errors as they arise when backtesting, and say what each one means.
The idea
Backtesting is a statistical decision, so it can go wrong in two ways. A Type I error is rejecting a model that is in fact correct. Its probability is the significance level, or for a fixed cut-off such as rejecting at 10 or more exceptions, the chance that a correct model produces that many exceptions by luck. A Type II error is accepting a model that is in fact wrong, for example one that understates risk. Its probability depends on how wrong the model is and on the sample size, and 1 minus it is the power of the test. For a given sample the two trade off, since a tighter cut-off lowers Type II errors but raises Type I errors. Wrongly keeping a model that understates risk is usually the costlier mistake for a regulator, while wrongly rejecting a good model wastes effort and capital.
A practice question
In backtesting a VaR model, which pair of definitions is correct?
- Type I error: rejecting a model that is correct. Type II error: accepting a model that is incorrect. - correct
- Type I error: accepting a model that is incorrect. Type II error: rejecting a model that is correct.Why not: The two definitions are swapped.
- Type I error: a model that overstates VaR. Type II error: a model that understates VaR.Why not: Confuses statistical decision errors with the direction of model bias.
- Type I error: rejecting a model for too many exceptions. Type II error: rejecting it for too few.Why not: Both are rejections of a model, and so both would be Type I errors if the model is correct; Type II requires accepting a bad model.
A Type I error is a false rejection (the model was right); a Type II error is a false acceptance (the model was wrong). These are errors in the test decision, not in the model's bias.
A harder one
A regulator changes its rule so that a model is rejected at 5 or more exceptions in 250 days instead of 10 or more. Holding everything else equal, which statement is correct?
- Both Type I and Type II error probabilities fall.Why not: Ignores the trade-off between the two errors for a fixed sample.
- The probability of a Type I error falls and the probability of a Type II error rises.Why not: Reverses the effect: a lower cut-off rejects more models, correct ones included.
- The probability of a Type I error rises and the probability of a Type II error falls. - correct
- The probability of a Type I error is unchanged and Type II falls.Why not: A correct model is more likely to reach 5 exceptions by luck than 10, so Type I changes too.
A lower rejection cut-off catches more flawed models (fewer Type II errors) but also rejects more correct ones (more Type I errors). For a fixed sample size the two errors trade off.
The hardest tier
An analyst rejects a 99% one-day VaR model whenever five or more exceptions occur in 250 days. If the model is in fact correct and independent, the probability of a Type I error is closest to which value?
- 6.7%Why not: Gives P(exactly 5 exceptions) only, not 5 or more.
- 4.1%Why not: Gives P(6 or more exceptions), starting the rejection region one exception too high.
- 10.8% - correct
- 89.2%Why not: Gives P(4 or fewer exceptions), the probability of acceptance, not of rejection.
Type I error = P(X >= 5 | N = 250, p = 0.01) = 1 - P(X <= 4) = 1 - 0.892 = 0.108.
In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.
Start free →More in Backtesting VaR
- Set out what backtesting and exceptions are, and show why backtesting matters for VaR models.
- Account for the main obstacles that make a VaR model hard to backtest.
- Judge a VaR model's accuracy by applying a model verification test to its exceptions or failure rates.
- Show why backtesting should take conditional coverage into account.
- Summarise the Basel requirements that govern VaR backtesting.