Charta is on Google Play: practise on the go. Get the app
Charta

Set out what model risk is and show how it can arise when a model is put into use.

FRM® Part II · Operational Risk and Resilience · Supervisory Guidance on Model Risk Management · FRM2.ORR.15.a

The idea

Model risk is the potential for adverse consequences, such as financial loss, poor business decisions or reputational damage, from decisions based on incorrect or misused model outputs. Supervisory guidance defines a model as a quantitative method that applies statistical, economic, financial or mathematical theories and assumptions to turn input data into quantitative estimates. It has three parts: an input component, a processing component and a reporting component. Risk arises in two ways. The first is fundamental error, where flawed design, data, assumptions or code produce inaccurate output even when the model is used as intended. The second is incorrect or inappropriate use, where a sound model is applied outside its purpose, to a different portfolio, or in changed market conditions. Risk rises with complexity, input uncertainty, breadth of use and potential impact. It can be managed but never eliminated. The usual mistake is treating model risk as only a coding problem.

A practice question

From the question bank, with the reason each wrong option is wrong.

Which statement best describes model risk as used in supervisory guidance on model risk management?

  1. The risk that a statistical estimate differs from the true population value because of random sampling error in the data.Why not: Confuses model risk with sampling error, which is only one narrow source of inaccuracy.
  2. The potential for adverse consequences from decisions based on incorrect or misused model outputs and reports. - correct
  3. The risk of loss on positions when market prices move against the bank, as measured by a model such as VaR.Why not: Confuses model risk with market risk, the exposure that the model is meant to measure.
  4. The risk that a software vendor fails to deliver or support a model on schedule.Why not: Confuses model risk with outsourcing or vendor delivery risk; the definition concerns decisions based on outputs.

Model risk is the risk of adverse consequences from decisions based on models that are wrong or used wrongly. It is distinct from market risk, sampling error and vendor delivery risk.

A harder one

A bank built a retail credit scorecard on prime auto loans. Code review confirms the scorecard calculates exactly as designed. The bank then applies it, unchanged, to subprime personal loans. Which source of model risk does this most clearly illustrate?

  1. Incorrect or inappropriate use of a model outside the purpose and population for which it was built. - correct
  2. A fundamental error in the model's mathematical processing, since the scorecard produces inaccurate output.Why not: The code is verified as correct; the problem is the application to a different population, not a processing error.
  3. An input error caused by late or incomplete data feeds into the scorecard.Why not: Nothing in the scenario points to data feed problems; the issue is the population to which the model is applied.
  4. No model risk, because the verified code proves the model is free of error.Why not: Verified code does not remove model risk; a correct model can still be misused.

Model risk arises both from fundamental errors and from misuse. Here the code is sound but the scorecard is used on a population it was never built or tested for, which is inappropriate use.

The hardest tier

A board member asks whether model risk could be eliminated by buying a vendor model that has been externally validated in full. What is the most accurate response?

  1. No: the bank remains responsible for appropriate use, ongoing monitoring and understanding the vendor model's limitations, so risk is reduced but remains. - correct
  2. Yes: external validation transfers all model risk and accountability to the vendor.Why not: Accountability for use and decisions stays with the bank; risk cannot be transferred by purchase.
  3. Yes, provided the vendor model is simpler than the bank's internal models.Why not: Simplicity lowers but does not remove model risk, and the bank still owns the use of the output.
  4. No: supervisory guidance prohibits the use of vendor models for decision making.Why not: Vendor models are permitted; guidance expects the same standards of validation and monitoring as internal models.

Model risk can be managed but not eliminated. A vendor model is still a model, and the bank must understand it, validate its use and monitor it.

In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.

Start free →

More in Supervisory Guidance on Model Risk Management