Backtesting VaR: exceptions and the two errors a test can make
A value at risk (VaR) figure is a claim: "we expect to lose more than this on only 1% of days." Backtesting checks that claim against what actually happened. If you are new to the measure itself, read value at risk explained first. More topics are on the FRM landing page.
What an exception is
An exception (also called an exceedance or a breach) is a day on which the actual loss is larger than the VaR forecast made for that day. Each day gives a yes or no outcome. The forecast must be the one made before the day started.
Why counting exceptions is a binomial problem
If the model is right, then on every day the chance of an exception equals the tail probability, p = 1 - confidence level. If we also assume days are independent, the number of exceptions X over N days follows a binomial distribution:
P(X = k) = C(N, k) x p^k x (1 - p)N - k
The expected count is N x p, and the standard deviation is the square root of N x p x (1 - p). For 99% VaR over 250 trading days, the expected count is 250 x 0.01 = 2.5. The real question is how far above 2.5 is too far.
Two ways to be wrong
Treat "the model is good" as the null hypothesis and pick a cut-off: reject the model if exceptions reach some number.
- Type I error: rejecting a good model. Its probability is set by the cut-off and the binomial with the correct p.
- Type II error: keeping a bad model, for example one whose true exception rate is higher than claimed.
- Power is 1 minus the Type II probability: the chance of spotting a bad model.
A stricter-sounding cut-off (a higher number of exceptions before you reject) lowers the Type I error but raises the Type II error. You cannot shrink both with a single cut-off and a fixed sample. Only more data helps with both.
Worked example: Harbourline Bank
Harbourline Bank reports a 99% one-day VaR, backtested over 250 days. The figures are exact binomial probabilities computed in Python.
If the model is good (p = 0.01), the chance of each count is:
| Exceptions | 0 | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|---|
| Probability | 0.0811 | 0.2047 | 0.2574 | 0.2149 | 0.1341 | 0.0666 | 0.0275 |
Now suppose the model is bad: the true exception rate is 2%, so about 5 are expected. We compare three cut-offs, rejecting the model at or above:
| Reject at | Type I (good model, p = 1%) | Power (true p = 2%) | Type II (true p = 2%) |
|---|---|---|---|
| 5 exceptions | 10.8% | 56.1% | 43.9% |
| 6 exceptions | 4.1% | 38.4% | 61.6% |
| 7 exceptions | 1.4% | 23.6% | 76.4% |
Moving from 5 to 7 cuts the Type I error from about 11% to about 1%, but the chance of keeping a model that understates risk by half rises from about 44% to 76%. With 250 days, a modest understatement is hard to detect. If the true rate were 3%, the power at a cut-off of 6 would rise to 76.3%.
Traffic-light zones
Supervisors usually turn this logic into zones. A low exception count is a green zone, where the model is accepted. A middle range is a yellow zone, where the result is treated as ambiguous and the firm may face extra scrutiny. A high count is a red zone, where the model is presumed to be flawed. The boundaries come from binomial probabilities, and a higher zone can bring a capital add-on. The exact numbers are set by the regulator and can change, so check the current rules.
The usual mistakes
- Treating 2.5 as a target. Zero exceptions happens about 8% of the time even when the model is right, though a persistently low count may suggest an over-conservative model.
- Confusing the two errors. Rejecting a good model is Type I; keeping a bad one is Type II. Mixing up "reject" and "keep" is the common slip.
- Assuming a stricter cut-off is simply better. It protects against false alarms and leaves more bad models standing.
- Ignoring independence. Clusters of exceptions in a crisis point to a model that reacts too slowly, which a simple count will not show.
- Forgetting that the confidence level sets p. At 95% VaR the expected count over 250 days is 12.5, not 2.5.
Questions typically ask for an expected count, a decision against a cut-off, or which error a statement describes.
Common questions
What is a VaR exception?
A day when the actual loss exceeds that day's VaR forecast. Backtesting counts these and compares the count with what the confidence level implies.
How many exceptions should a 99% VaR have in 250 days?
On average 2.5, because 250 x 0.01 = 2.5. The actual count varies, and the binomial distribution tells you how likely each count is.
What is the difference between a Type I and a Type II error in backtesting?
A Type I error rejects a model that is in fact good. A Type II error keeps a model that is in fact bad. Raising the rejection cut-off reduces the first and increases the second.
Why can a backtest fail to catch a bad model?
With a year of daily data, exceptions are rare, so power is low against a modest understatement of risk. Longer samples help.
Sources
- GARP, the FRM programme: what the exam is and its two parts.
- Basel Committee on Banking Supervision, publications: the supervisory framework behind backtesting zones.
The exam and the syllabus can change. Check GARP's current details before you book.