Charta is on Google Play: practise on the go. Get the app
Charta

Outline what a strong model validation process includes and the difficulties that can undermine effective validation.

FRM® Part II · Operational Risk and Resilience · Supervisory Guidance on Model Risk Management · FRM2.ORR.15.d

The idea

Model validation is the set of activities that assess whether a model performs as intended and whether its limitations are understood. A strong process has three core elements. Evaluation of conceptual soundness reviews design, theory, assumptions, data and developmental evidence. Ongoing monitoring confirms the model stays appropriate as products and markets change, using process verification, benchmarking against alternatives and sensitivity analysis. Outcomes analysis compares model forecasts with actual results, for example back-testing VaR exceptions against their expected number. Validators must be independent of development and use, competent, and have the influence and incentives to challenge. Reviews should be periodic, at least annually, and scaled to a model's materiality. Difficulties include vendor models with undisclosed code, scarce skilled staff, weak independence, poor or short data and pressure from the business. The usual mistake is equating validation with a one-time back-test at launch.

Formulas

Back-test exception rate

exception rate = exceptions ÷ N

exceptions
days on which the loss exceeded VaR
N
the number of trading days tested

Compare with one minus the confidence level. A rate well above it suggests VaR understates risk.

A practice question

From the question bank, with the reason each wrong option is wrong.

The core elements of an effective model validation process are:

  1. Data collection, coding, implementation testing and release into the production environment.Why not: These are development tasks, not the elements of independent validation.
  2. Documentation, inventory maintenance and internal audit.Why not: These are governance and control activities, not the three elements of validation.
  3. Calibration, optimisation and deployment sign-off.Why not: Calibration and optimisation belong to development; sign-off is not itself an assessment of performance.
  4. Evaluation of conceptual soundness, ongoing monitoring and outcomes analysis. - correct

Validation evaluates whether the design is sound, whether the model keeps performing as conditions change, and whether outputs match actual outcomes.

A harder one

A 1-day 99 percent VaR model is back-tested over 500 trading days and records 12 exceptions. Which interpretation is correct?

  1. The observed exception rate is 4.8 percent, well above the 1 percent expected, so the model understates risk.Why not: Divides 12 by 250 days instead of the 500-day sample.
  2. The observed exception rate is 2.4 percent against 5 percent expected, so the model is conservative.Why not: Uses the 5 percent tail of a 95 percent VaR instead of the 1 percent tail of a 99 percent VaR.
  3. The observed exception rate is 2.4 percent against 1 percent expected, 2.4 times the expected 5 exceptions, which suggests risk is understated. - correct
  4. The observed exception rate is 0.24 percent against 1 percent expected, so the model is conservative.Why not: Misplaces the decimal: 12 / 500 is 0.024, which is 2.4 percent, not 0.24 percent.

Exception rate = 12 / 500 = 2.4 percent. At 99 percent confidence the expected rate is 1 percent, or 5 exceptions in 500 days. Twelve is well above that, so outcomes analysis points to understated risk.

The hardest tier

A vendor refuses to disclose the source code of a model the bank uses. What is the most appropriate validation approach?

  1. Validate with what is available: review the vendor's developmental evidence, run sensitivity analysis, benchmark against alternatives, perform outcomes analysis, and apply conservatism where gaps remain. - correct
  2. Waive validation because a reputable vendor has already tested the model.Why not: The bank remains responsible; vendor testing does not replace the bank's own validation.
  3. Ban all vendor models, since a model that cannot be fully inspected cannot be used.Why not: Guidance accepts vendor models with appropriate validation and controls.
  4. Validate only the vendor's marketing and product documentation.Why not: Marketing material is not evidence of performance; independent testing of behaviour is needed.

Opacity is a recognised difficulty. Validators must use whatever can be tested, namely evidence, behaviour and outcomes, and compensate for gaps with caution in use.

In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.

Start free →

More in Supervisory Guidance on Model Risk Management