Explain the purpose of the function that manages model risk, and recommended methods for managing model risk and validating models.
The idea
The model risk management function exists to identify, measure and control model risk across the whole institution, independently of the people who build and use models. Its tools are a complete model inventory, a tiering of models by materiality, governance with clear ownership and approval before use, and documentation standards. It also relies on independent validation with effective challenge. Effective challenge means validators have the competence, standing and authority to question and to restrict a model. Validation has three core parts. These are evaluating conceptual soundness, ongoing monitoring, and outcomes analysis such as back-testing forecasts against realised results. Validation is repeated periodically and whenever the model changes. Remaining limitations are handled with conservative adjustments, limits and use restrictions. Exam point: independence from developers and users is the key control, and back-testing on its own is not sufficient.
Formulas
Expected number of VaR exceptions
expected exceptions = p × N
- p
- one minus the VaR confidence level
- N
- the number of trading days tested
Assumes the model is correct and exceptions are independent, so the count is binomial. Both too many and too few exceptions signal a deficient model.
Probability of no VaR exceptions
P(no exceptions) = c^(N)
- c
- the VaR confidence level
- N
- the number of trading days
Assumes independent exceptions under a correct model. Over a long window zero exceptions is unlikely and suggests VaR is too conservative.
A practice question
Which activity is an example of outcomes analysis in model validation?
- Backtesting forecasts against realised results. - correct
- Reviewing the developer's documentation to confirm that the model's theory and assumptions are sound.Why not: This is evaluation of conceptual soundness, a different element of validation.
- Checking that the production code reproduces the model specification exactly.Why not: This is process verification and implementation testing, which is part of ongoing monitoring, not outcomes analysis.
- Reviewing who has access rights to change the model and its inputs.Why not: This is a control or governance review of access, not a comparison of forecasts with outcomes.
Validation has three core elements: conceptual soundness, ongoing monitoring and outcomes analysis. Backtesting compares predictions with actual results and so is outcomes analysis.
A harder one
A bank's model validation staff report to the head of the trading desk that built and uses the models they review. Which arrangement would best restore the independence needed for effective challenge?
- The model developers review their own models, since they understand them best.Why not: Developers reviewing their own work removes independence and the challenge it should provide.
- Validators report outside the business line, can restrict a model, and are not paid on desk profit. - correct
- Validators stay in the desk but the desk head signs off after receiving their advice.Why not: Final approval remains with the party that benefits from the model, so challenge has no authority.
- External auditors review each model once at inception and not again unless there is a loss.Why not: Validation must be periodic and triggered by changes; a one-time review at inception is not ongoing monitoring.
Effective challenge needs competence, standing and independence from developers and users, together with authority to act on findings. Reporting to the business line creates a conflict of interest.
The hardest tier
A bank's 99% one-day VaR model recorded no exceptions in 500 consecutive trading days. The model owner presents this as proof that the model is accurate. Which assessment is most appropriate?
- Zero exceptions is unlikely (about 0.7%) under a correct model, so VaR is probably too conservative. - correct
- The result is consistent with a correct model, since the expected number of exceptions over 500 days is about 1 and zero is close to that.Why not: The expected number is 5, not 1 (500 x 1%), and zero is unlikely under a correct model.
- The model must be understating risk, because a model that is wrong always produces fewer exceptions.Why not: A lack of exceptions points to overstatement of risk, not understatement; understated risk produces too many exceptions.
- The result confirms accuracy, so no further validation is needed until a new product is introduced.Why not: Backtesting tests calibration in both directions; zero exceptions is a deviation, and validation also covers soundness and monitoring.
Expected exceptions are 500 x 1% = 5. The probability of none is 0.99^500, about 0.66%. Too few exceptions indicates overly conservative VaR (excess capital), which is also a model deficiency.
In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.
Start free →