Lay out PIT-based backtesting and set the goodness-of-fit tests for judging PIT distributions against each other: Kolmogorov-Smirnov, Anderson-Darling and Cramér-von Mises.
The idea
PIT-based backtesting runs in steps. Record each day's forecast distribution, compute the PIT of the realised P&L, then test the null that the PITs are independent and uniform on 0 to 1, and finally read the histogram to see how any failure arises. Three goodness-of-fit tests compare the empirical distribution of the PITs with the uniform one. Kolmogorov-Smirnov uses the largest absolute vertical gap between the two distribution functions. Its approximate 5% critical value is 1.36 divided by the square root of n. Cramér-von Mises sums the squared gaps over the whole range with equal weight. Anderson-Darling also squares the gaps but weights them by 1 ÷ (u × (1 − u)), so it is the most sensitive to the tails. All three assume independence, so test that separately.
Formulas
Approximate 5% critical value of the Kolmogorov-Smirnov statistic
Dcrit = 1.36 ÷ √n
- n
- the number of PITs tested
An approximate value, compared with the largest gap D: reject uniformity if D exceeds it. Dividing by n rather than √n, or comparing D with 1.36 unscaled, are the usual slips.
A practice question
Which test statistic is the largest absolute vertical gap between the empirical distribution function of the PITs and the uniform distribution function?
- Kolmogorov-Smirnov - correct
- Anderson-DarlingWhy not: Describes a weighted sum of squared gaps with weights 1 / (u(1 - u)), not the single largest gap.
- Cramér-von MisesWhy not: Describes the integral of squared gaps with equal weight, not the maximum gap.
- Kupiec proportion of failuresWhy not: A test of the exceedance count at one VaR level, not a comparison of full distribution functions.
Kolmogorov-Smirnov takes D = max |F_n(u) - u|. Cramér-von Mises and Anderson-Darling both aggregate squared gaps over the whole range rather than taking the maximum.
A harder one
A validator has 400 PITs. The Kolmogorov-Smirnov statistic is D = 0.075. The approximate 5% critical value is 1.36 divided by the square root of n. Which conclusion is correct?
- Reject uniformity: the critical value is 1.36 / 20 = 0.068, which D exceeds. - correct
- Do not reject: D = 0.075 is far below the critical value of 1.36.Why not: Forgets to scale 1.36 by the square root of n, so compares D with the unscaled constant.
- Reject: D exceeds 1.36 / 400 = 0.0034.Why not: Divides by n instead of its square root; the critical value is 0.068, and this reaches rejection with the wrong threshold.
- Do not reject: a two-sided test needs D to exceed twice 0.068, which is 0.136.Why not: Doubles the critical value as if the test were two-sided; the KS statistic is already based on absolute gaps.
The critical value is 1.36 / sqrt(400) = 1.36 / 20 = 0.068. Since 0.075 > 0.068, the hypothesis that PITs are uniform is rejected at 5%.
The hardest tier
Six sorted PITs are 0.03, 0.07, 0.12, 0.44, 0.62 and 0.71. The Cramér-von Mises statistic is W² = 1 / (12n) + the sum over i of [u(i) - (2i - 1) / (2n)]². Which value is closest to W²?
- 0.20Why not: Omits the 1 / (12n) term, which adds 0.014.
- 0.22 - correct
- 0.43Why not: Uses i / n in place of (2i - 1) / (2n), so the gaps measure the wrong points.
- 1.01Why not: Sums the absolute gaps rather than squaring them.
Gaps u(i) - (2i - 1) / 12 are -0.053, -0.180, -0.297, -0.143, -0.130 and -0.207 for i = 1 to 6; the squares sum to 0.2034. Adding 1 / 72 = 0.0139 gives W² = 0.217.
In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.
Start free →More in Forecasting Distribution Using Probability Integral Transform
- Spot which features of an exceedance-based backtest signal an accurate VaR model, and show how a PIT-based backtest captures those same features.
- Describe how a validator computes probability integral transforms (PITs) from a VaR model's forecasts.
- Explain how the shape of the PIT distribution signals how good a VaR model is.