Charta is on Google Play: practise on the go. Get the app
Charta

Judge a VaR model's accuracy by applying a model verification test to its exceptions or failure rates.

FRM® Part II · Market Risk Measurement and Management · Backtesting VaR · FRM2.MR.4.c

The idea

The proportion of failures (Kupiec) test asks whether the observed exception frequency is consistent with the tail probability p, 1 minus the confidence level. With N observations and x exceptions, ln L(q) = (N − x) ln(1 − q) + x ln q is the log likelihood of x exceptions at failure probability q. The statistic is twice the unrestricted log likelihood, at x ÷ N, minus the restricted one, at p, which is LR = 2 × [ln L(x ÷ N) − ln L(p)]. If the model is correct, LR follows a chi-square distribution with 1 degree of freedom, so reject when LR exceeds 3.84 (5 percent) or 6.63 (1 percent). A large-sample shortcut standardises the count using the null probability p, giving z = (x − pN) ÷ √(p(1 − p)N). Reject when the absolute z exceeds 1.96. It is only an approximation. For a 99 percent VaR over 250 days it rejects at 6 exceptions, where LR does not. The test is two-sided, so too many or too few exceptions are rejected. By the LR test, 7 or more exceptions are rejected at the 5 percent level.

Formulas

Kupiec likelihood ratio

LR = −2 × ln[(1 − p)^(N − x) × p^(x)] + 2 × ln[(1 − x ÷ N)^(N − x) × (x ÷ N)^(x)]

p
the tail probability, 1 minus the confidence level
N
the number of observations
x
the number of exceptions

Chi-square with 1 degree of freedom: reject above 3.84 at 5%. Use natural logs and keep the factor of 2; too few exceptions are rejected as well as too many.

Normal approximation to the exception count test

z = (x − p × N) ÷ √(p × (1 − p) × N)

x
the number of exceptions
p
the null tail probability
N
the number of observations

Large samples only; reject when |z| exceeds 1.96. The standard deviation uses the null p, not the observed rate, and the approximation is poor when few exceptions are expected.

A practice question

From the question bank, with the reason each wrong option is wrong.

What is the null hypothesis of the Kupiec proportion-of-failures test?

  1. Exceptions are independent from one day to the next.Why not: That is the null of the independence test used in conditional coverage, not the proportion-of-failures test.
  2. The true probability of an exception is zero, so any exception rejects the model.Why not: A correct VaR is breached with probability 1 minus the confidence level, not never.
  3. The VaR equals the average loss on those days on which an exception occurs.Why not: Mixes up VaR with an expected-shortfall style comparison; the test concerns frequency only.
  4. The true exception probability equals the tail probability, 1 minus the confidence level. - correct

The test compares the observed exception frequency x/N with p = 1 - confidence level. The null is that the model is correct, i.e. the true failure probability equals p.

A harder one

A bank backtests a 99% one-day VaR over 500 days and records 11 exceptions. Using z = (x - pN) / sqrt(p(1-p)N) and a two-sided 5% critical value of 1.96, what is the result?

  1. z = 1.21, so the model is not rejected.Why not: Divides by the variance p(1-p)N instead of its square root.
  2. z = 2.70, so the model is rejected. - correct
  3. z = 1.83, so the model is not rejected.Why not: Uses the observed rate of 2.2 percent in the standard error instead of the null rate of 1 percent.
  4. z = 4.94, so the model is rejected.Why not: Forgets to subtract the expected count of 5 and divides the observed 11 by the standard deviation.

Expected exceptions = 500 x 0.01 = 5; standard deviation = sqrt(500 x 0.01 x 0.99) = 2.225. z = (11 - 5) / 2.225 = 2.70 > 1.96, so reject at 5%.

The hardest tier

A 95% one-day VaR shows 36 exceptions in 500 days (expected 25). Chi-square (1 df) critical values are 3.84 at 5% and 6.63 at 1%. What is the Kupiec test outcome?

  1. LR is about 4.5: reject at the 5% level but not at the 1% level. - correct
  2. LR is about 9.0: reject at both the 5% and 1% levels.Why not: Doubles the likelihood ratio, for example by applying a factor of 4 instead of 2.
  3. LR is about 2.3: do not reject at either level.Why not: Uses the z-score of 2.26 as if it were the likelihood ratio.
  4. LR is about 4.5: do not reject at either level because the critical value is 5.99.Why not: Uses the 2-degree-of-freedom critical value, which belongs to the conditional coverage test.

Observed rate = 36/500 = 7.2%. LR = -2[464 ln 0.95 + 36 ln 0.05] + 2[464 ln 0.928 + 36 ln 0.072] = 4.51. That exceeds 3.84 but is below 6.63.

In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.

Start free →

More in Backtesting VaR