Charta is on Google Play: practise on the go. Get the app
Charta

Account for the main obstacles that make a VaR model hard to backtest.

FRM® Part II · Market Risk Measurement and Management · Backtesting VaR · FRM2.MR.4.b

The idea

A VaR model is hard to backtest for several reasons. VaR describes the portfolio held at the start of the day. Reported (actual) profit and loss also contains trading during the day, fees, commissions, reserve changes and carry, so it is contaminated. Banks therefore also compute hypothetical or clean P&L by revaluing the unchanged positions on which the VaR was based, and compare VaR with both. Positions change constantly, so each day's VaR belongs to a different portfolio and the sample is not repeated draws from one distribution. At a high confidence level exceptions are rare. At 99 percent only about 2.5 are expected in 250 days, so the test has little power and one cluster can dominate. Finally, VaR is a single quantile, so exceptions say nothing about the size of losses beyond it.

A practice question

From the question bank, with the reason each wrong option is wrong.

Which is a main reason why backtesting VaR against reported (actual) P&L can be misleading?

  1. Actual P&L is always normally distributed, whereas the VaR distribution is not, so they cannot be compared.Why not: The shape of the P&L distribution is not the issue; the problem is contamination by items outside the VaR's scope.
  2. Actual P&L includes intraday trading results, fees, commissions and reserve changes that the end-of-day VaR did not measure. - correct
  3. Actual P&L is available only quarterly, so exceptions cannot be counted on a daily basis at all.Why not: Daily P&L is routinely produced; the difficulty is its content, not its frequency.
  4. Actual P&L is measured over 10 days, whereas VaR is a one-day figure.Why not: Confuses the P&L comparison with the regulatory 10-day horizon; backtests use daily P&L against one-day VaR.

VaR is built on the positions held at the start of the day. Reported P&L also reflects trades during the day, fees and reserves, so it can mask or add exceptions; hypothetical P&L from frozen positions is used to isolate the model.

A harder one

A bank backtests VaR against actual P&L that includes fee income and intraday trading gains that were, on average, positive. Compared with backtesting against hypothetical P&L from frozen end-of-day positions, the effect is most likely which of the following?

  1. Exceptions are understated, so the VaR model looks more accurate than it is. - correct
  2. Exceptions are overstated, so the VaR model looks worse than it is.Why not: Reverses the sign: positive income offsets market losses and so removes exceptions rather than adding them.
  3. There is no effect, because VaR is computed on the same trades.Why not: VaR is computed on end-of-day positions; later trades and fees are not part of what it measured.
  4. The number of exceptions is unchanged but they become more clustered.Why not: Invents a clustering effect; the contamination biases the exception count itself.

Positive items not in the VaR (fees, intraday profits) reduce reported losses, so some days that would breach VaR on a clean basis do not. Hypothetical P&L is used alongside actual P&L to avoid this bias.

The hardest tier

A new 99% one-day VaR model has produced one exception in its first 60 trading days and a manager proposes scrapping it. Assuming the model is correct and exceptions are independent, what is the probability of at least one exception in 60 days, and what follows?

  1. About 1%, so the exception is very unlikely and the model should be rejected.Why not: Uses the single-day exception probability instead of the probability over 60 days.
  2. About 60%, so one exception is almost certain and the model is confirmed.Why not: Multiplies 60 x 1% instead of compounding; the additive shortcut overstates the probability.
  3. About 55%, so one exception is more likely than not to be avoided and the model is confirmed.Why not: Reports 0.99^60, the probability of no exceptions, rather than its complement.
  4. About 45%, so one early exception is not strong evidence against the model. - correct

P(at least one) = 1 - 0.99^60 = 1 - 0.547 = 0.453. With small samples, isolated exceptions occur often even for a correct model, which is why backtesting needs adequate data.

In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.

Start free →

More in Backtesting VaR