Account for good practice in building and implementing models.
The idea
Good practice in building and implementing a model begins with a clear statement of purpose that guides design and use. Design should rest on sound theory and logic supported by research, with assumptions and limitations stated openly. Data must be of suitable quality and relevance, and any proxy data must be justified and documented. Testing covers accuracy, robustness and stability. It includes sensitivity analysis, stress of inputs and out-of-sample or out-of-time tests, because a good in-sample fit may only reflect overfitting. Documentation should let a knowledgeable third party understand how the model works and what its limits are. Implementation requires verified code, controlled systems integration and change control. In use, managers should understand limitations and document any judgmental overrides. They should treat persistent overrides as a sign the model needs rework, and may add conservatism where uncertainty is high. The exam angle is that testing and documentation start in development, not at validation.
A practice question
What should be the first element established when developing a new model?
- A clear statement of the model's purpose, which guides its design, testing and use. - correct
- A back-test of candidate specifications on all of the available historical data for the product.Why not: Testing comes after the purpose and design are defined; without a purpose there is nothing to test against.
- The choice of programming language and computing platform.Why not: Technology choices follow from purpose and design requirements.
- Calibration of parameters to maximise in-sample fit.Why not: Calibration is part of later development; maximising in-sample fit also invites overfitting.
Good practice starts with a clear statement of purpose. Design, data, testing and eventual use are judged against it.
A harder one
A developer is building a model for a new product with very little history and proposes using data from a loosely similar product as a proxy. Which action is most consistent with good practice?
- Use whichever data set gives the most stable parameter estimates, since stability is the key criterion for model quality.Why not: Selecting for stability rather than relevance can mask real differences between products.
- Assess how relevant the proxy is to the new product, document the choice and adjustments, and disclose limitations to users. - correct
- Skip documentation of the proxy, because validation will review the data later.Why not: Proxy choices must be documented at development so that they can be reviewed and understood.
- Postpone the model until ten years of data on the product exist.Why not: Guidance accepts proxies where justified; it does not require waiting for a long product history.
Data must be of suitable quality and relevance. Where proxies are used, relevance is assessed, adjustments are documented and limitations are communicated.
The hardest tier
A developer fits a loss forecasting model on data from 2005 to 2015, reports an R² of 0.92 on that same data, and recommends going live. What is the main weakness in the testing?
- The R² is too high, which proves that the model is wrong and must be rebuilt from scratch with other variables.Why not: A high fit is not itself an error; the issue is that it was measured on the estimation data.
- Ten years of data is too long, because older data should never be used.Why not: The length of the sample is not the weakness; the missing hold-out test is.
- There is no out-of-sample or out-of-time test, so the high in-sample fit may simply reflect overfitting. - correct
- No benchmark model was built, which is the only evidence that matters.Why not: Benchmarking is useful but does not replace out-of-sample testing of the model itself.
Fit on the estimation sample says little about predictive power. Good practice tests on data not used in estimation, covering different periods, to detect overfitting.
In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.
Start free →