Charta is on Google Play: practise on the go. Get the app
Charta

Explain how a scorecard is measured and monitored over time, drawing on the cumulative accuracy profile (CAP) and accuracy ratio (AR) techniques.

FRM® Part II · Credit Risk Measurement and Management · Credit Scoring and Retail Credit Risk Management · FRM2.CR.7.f

The idea

A scorecard is judged on rank ordering and then monitored. The cumulative accuracy profile ranks borrowers from worst score to best. It plots the cumulative share of all borrowers on the horizontal axis against the cumulative share of defaulters captured on the vertical axis. A random model lies on the diagonal and a perfect model captures all defaulters first. The accuracy ratio is the area between the model CAP and the diagonal divided by the same area for the perfect model, so it runs from 0 for no power to 1 for perfect. Let AUC be the area under the ROC curve, which links the two measures as follows. AR = 2 × AUC − 1. Monitoring adds the population stability index, where actual% and expected% are the shares of borrowers in each score band now and at development. PSI = Σ (actual% − expected%) × ln(actual% ÷ expected%). A PSI below 0.10 is usually read as stable. Monitoring also backtests AR and calibration by vintage. AR is rank-based, so a calibration shift does not change it.

Formulas

Accuracy ratio from the ROC area

AR = 2 × AUC − 1

AUC
the area under the ROC curve

AR is rank-based, so recalibrating PDs does not change it. It runs from 0 for a random model to 1 for a perfect one.

Accuracy ratio from CAP areas

AR = (model area − 0.5) ÷ (perfect area − 0.5)

model area
the area under the model's CAP
perfect area
the area under the perfect model's CAP, one minus half the default rate

Remove the diagonal's 0.5 from both areas. Applying 2 × area − 1 to a CAP area is wrong; that shortcut is for the ROC area.

Population stability index

PSI = Σ (actual − expected) × ln(actual ÷ expected)

actual
the share of today's population in a score band
expected
the share of the development population in that band

Natural logs, summed over bands. Below 0.10 is usually read as stable; a base-10 log understates the index.

A practice question

From the question bank, with the reason each wrong option is wrong.

How is a cumulative accuracy profile (CAP) constructed?

  1. Borrowers are ranked from best to worst and the cumulative share of loan value is plotted against elapsed time.Why not: The CAP plots cumulative defaulters against cumulative borrowers, not loan value against time.
  2. Borrowers are ranked worst to best and cumulative defaulters are plotted against cumulative borrowers. - correct
  3. Predicted PD is plotted against observed default rate for each score band.Why not: That is a calibration plot, which is not the CAP.
  4. The score distributions of goods and bads are plotted against each other and the maximum gap is measured.Why not: That is the Kolmogorov-Smirnov test, not the CAP.

A CAP ranks risk worst first, so a strong model captures most defaulters in the lowest-scoring slice of the population.

A harder one

A scorecard's area under the ROC curve is 0.78. Using AR = 2 x AUC - 1, the accuracy ratio is closest to:

  1. 0.78Why not: Equates the accuracy ratio with the AUC.
  2. 0.28Why not: Subtracts the random AUC of 0.5 but fails to scale by the perfect model's area.
  3. 0.22Why not: Uses 1 - AUC, the probability of misranking, as the accuracy ratio.
  4. 0.56 - correct

AR = 2 x 0.78 - 1 = 0.56. AR is the Gini coefficient, a rescaling of AUC so that random = 0 and perfect = 1.

The hardest tier

A scorecard's CAP passes through these points (share of borrowers, share of defaulters): (0, 0), (20%, 50%), (50%, 80%) and (100%, 100%), joined by straight lines. The portfolio default rate is 10%. The accuracy ratio is closest to:

  1. 0.39Why not: Applies 2 x area - 1, which is valid for the ROC area but not for the area under the CAP.
  2. 0.43 - correct
  3. 0.20Why not: Subtracts the diagonal's area but does not divide by the perfect model's area above the diagonal.
  4. 0.73Why not: Divides the area under the CAP by the perfect model's area without removing the random model's 0.5 from both.

Trapezoid area = 0.05 + 0.195 + 0.45 = 0.695. Perfect model area = 1 - 0.10/2 = 0.95. AR = (0.695 - 0.5) / (0.95 - 0.5) = 0.43.

In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.

Start free →

More in Credit Scoring and Retail Credit Risk Management