Spot which features of an exceedance-based backtest signal an accurate VaR model, and show how a PIT-based backtest captures those same features.
The idea
An exceedance-based backtest counts the days on which the realised loss exceeded the VaR forecast. An accurate model shows two features. The exceedance frequency is close to one minus the confidence level, about 1% of days for 99% VaR. The exceedances are also independent over time, with no clustering. Kupiec tests the first feature and Christoffersen the second. The weakness is that the test looks at one quantile only and ignores how large each exceedance was. A PIT backtest passes each day's realised P&L through that day's forecast distribution function and checks that the results are independent and uniform on 0 to 1. The share of PITs below 0.01 reproduces the 99% exceedance frequency, the share below 0.05 reproduces the 95% one, and serial dependence among low PITs reproduces clustering. So the PIT test checks the exceedance features at every confidence level at once, using every observation.
Formulas
Probability integral transform of a day's P&L
PITₜ = Fₜ(xₜ)
- Fₜ
- the forecast distribution function for day t, made beforehand
- xₜ
- the realised P&L on day t
For a correct model the PITs are independent and uniform on 0 to 1. The share below 1 − c is the exceedance frequency at confidence c.
A practice question
Which pair of features of the exceedance record of a 99% one-day VaR model indicates that the model is accurate?
- Exceedances on about 1% of days, with no clustering of the exceedances over time. - correct
- Exceedances on about 99% of days, each roughly equal in size to the VaR.Why not: Confuses the confidence level with the exceedance frequency; at 99% confidence, exceedances should be rare, about 1% of days.
- No exceedances at all in the sample, so the VaR is never breached.Why not: Treats zero exceedances as ideal; an exceedance frequency far below 1% signals an overly conservative model, not an accurate one.
- Exceedances on about 1% of days, but bunched in a few turbulent periods the model flags as stress.Why not: Accepts clustering; an accurate model's exceedances are independent, and clustering shows the VaR is slow to respond to changing risk.
An accurate model shows unconditional coverage, an exceedance frequency near 1 minus the confidence level, and independence, meaning exceedances do not cluster. Too few, too many or bunched exceedances all point to a flawed model.
A harder one
A validator computes 500 daily PITs of realised P&L against a 99% one-day VaR model, where a low PIT means a bad outcome. Fourteen of the PITs are below 0.01. Which interpretation is correct?
- 14 is below 5% of 500, which is 25, so the model passes.Why not: Applies the 5% tail of a 95% VaR model to a 99% model; the expected count at 99% is 1% of 500, which is 5.
- The 2.8% exceedance frequency is above 1%, which means the model is conservative.Why not: Reads the direction backwards; more exceedances than promised means the VaR is too low, not too conservative.
- 2.8% is well above the promised 1%, so the model understates tail risk. - correct
- The 14 PITs match an exceedance frequency of 0.28%, which is below 1% and acceptable.Why not: Divides 14 by 5,000 instead of 500, shrinking the exceedance frequency by a factor of ten.
A PIT below 0.01 is a day when the outcome fell in the worst 1% of the forecast distribution, which is a 99% VaR exceedance. The expected count is 5 of 500, and 14 is 2.8%, so risk is understated.
The hardest tier
Models A and B each record 10 exceedances of their 99% one-day VaR over 1,000 trading days. For model A the exceedances are spread through the sample; for model B all ten fall within a 15-day window. Which statement is correct?
- The two models are equally accurate, since an exceedance backtest depends only on the number of exceedances.Why not: Ignores the independence feature; the timing of exceedances matters because clustering shows the model fails to adapt to volatility.
- Both show the right frequency, but B fails the independence feature; its low PITs would be serially dependent even though the share below 0.01 is about 1%. - correct
- Model A fails, because exceedances spread across the sample show the VaR is wrong in many regimes.Why not: Reverses the logic; spread-out exceedances at the right frequency are what an accurate model produces.
- Model B is better, since concentrating exceedances in one window implies the VaR is correct in all other periods.Why not: Treats clustering as a strength; a cluster shows the VaR was far too low during a stress episode the model failed to track.
Both frequencies are 10 / 1,000 = 1%, so unconditional coverage is satisfied. Only B's exceedances are dependent, so only B fails the independence property, and a PIT test picks this up as dependence among the tail PITs.
In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.
Start free →More in Forecasting Distribution Using Probability Integral Transform
- Describe how a validator computes probability integral transforms (PITs) from a VaR model's forecasts.
- Explain how the shape of the PIT distribution signals how good a VaR model is.
- Lay out PIT-based backtesting and set the goodness-of-fit tests for judging PIT distributions against each other: Kolmogorov-Smirnov, Anderson-Darling and Cramér-von Mises.