Charta is on Google Play: practise on the go. Get the app
Charta

Set out what backtesting and exceptions are, and show why backtesting matters for VaR models.

FRM® Part II · Market Risk Measurement and Management · Backtesting VaR · FRM2.MR.4.a

The idea

Backtesting compares the VaR a model produced with what actually happened. Each day you record the VaR forecast made beforehand and the profit or loss realised afterwards. An exception (also called a failure or violation) is a day on which the loss is larger than that VaR. If the model is right, the chance of an exception each day equals the tail probability, 1 minus the confidence level. So a 99 percent one-day VaR should be exceeded on about 1 percent of days, or 2.5 days in 250. Backtesting matters because VaR is a quantile that cannot be observed directly. Only comparing forecasts with outcomes shows whether risk is understated, which leaves capital inadequate, or overstated, which wastes capital. Too few exceptions is a warning sign just as too many are, and regulators use the count to set capital multipliers.

Formulas

Expected number of VaR exceptions

expected exceptions = N × (1 − c)

N
the number of days in the backtest
c
the VaR confidence level

A correct, independent model. Using c instead of 1 − c gives the number of non-exceptions.

A practice question

From the question bank, with the reason each wrong option is wrong.

In VaR backtesting, what is an exception?

  1. A day on which the portfolio records any loss at all.Why not: Confuses any loss with a loss beyond VaR; losses smaller than VaR are expected and are not exceptions.
  2. A day on which the VaR estimate is recalculated because the estimation window has rolled forward.Why not: Confuses routine model updating with the outcome comparison that defines an exception.
  3. A day on which the realised loss is larger than that day's VaR forecast. - correct
  4. A day on which the realised loss equals the average of the losses that lie beyond VaR.Why not: Describes the tail average used by expected shortfall, not the day-level comparison of loss with VaR.

An exception (failure or violation) is a day when the realised loss exceeds the VaR forecast for that day. Counting exceptions and comparing the count with N x (1 - confidence level) is the basis of backtesting.

A harder one

A bank backtests its 99% one-day VaR over 500 trading days and records 9 exceptions. Which pair of statements is correct?

  1. The exception rate is 1.8% against 99% expected, so 9 exceptions were observed against 495 expected.Why not: Uses the confidence level instead of the tail probability to get the expected count.
  2. The exception rate is 0.9% against 1.0% expected, so 9 exceptions were observed against 10 expected.Why not: Uses 1,000 days instead of the 500-day sample, so both the rate and the expected count are wrong.
  3. The exception rate is 1.8% against 5.0% expected, so 9 exceptions were observed against 25 expected.Why not: Uses a 5 percent tail, which belongs to a 95 percent VaR, not the 1 percent tail of a 99 percent VaR.
  4. The exception rate is 1.8% against 1.0% expected, so 9 exceptions were observed against 5 expected. - correct

Observed rate = 9 / 500 = 1.8%. Expected rate = 1 - 0.99 = 1%, so expected exceptions = 500 x 0.01 = 5. The model shows nearly twice as many exceptions as expected.

The hardest tier

A desk's one-day 99% VaR (stated as a positive loss) is set at each day's close and compared with the next day's profit or loss, in USD million. Day 1: VaR 4.0, P&L -3.8. Day 2: VaR 4.2, P&L -4.5. Day 3: VaR 4.1, P&L +5.2. Day 4: VaR 3.9, P&L -4.0. Day 5: VaR 4.5, P&L -4.4. Day 6: VaR 5.5, P&L -6.0. How many exceptions are there?

  1. 4Why not: Counts the +5.2 gain on day 3 as an exception because its absolute value exceeds VaR; only losses count.
  2. 5Why not: Counts every day with a loss as an exception, whether or not the loss exceeded that day's VaR.
  3. 3 - correct
  4. 1Why not: Compares each day's P&L with the VaR set the following day instead of the VaR set beforehand.

A loss exceeds VaR on day 2 (4.5 > 4.2), day 4 (4.0 > 3.9) and day 6 (6.0 > 5.5). Day 1 and day 5 are losses inside VaR, and the day 3 gain is not an exception. Total = 3.

In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.

Start free →

More in Backtesting VaR