Charta is on Google Play: practise on the go. Get the app
Charta

Assess how well logistic regression models predict.

FRM® Part I · Quantitative Analysis · Machine Learning and Prediction · FRM1.QA.15.b

The idea

A logistic model is judged by how well it separates and ranks cases it has not seen, so the assessment must use a holdout or cross-validated sample. The confusion matrix at a chosen cutoff gives true and false positives and negatives. The ROC curve plots the true positive rate against the false positive rate as the cutoff moves from 1 to 0. The area under it, the AUC, summarises ranking power. An AUC of 0.5 means no better than chance and 1.0 means perfect ranking. Accuracy can mislead when classes are imbalanced, because a model that never predicts default can score 98 percent if only 2 percent default. A big gap between in-sample and out-of-sample performance signals overfitting. AUC averages over all cutoffs, so two models can differ in the region the business actually uses. Calibration, whether predicted probabilities match observed frequencies, is a separate question from ranking.

A practice question

From the question bank, with the reason each wrong option is wrong.

A fitted logistic default model has an AUC of 0.50 on a holdout sample. This indicates that the model:

  1. separates defaulters from non-defaulters perfectly.Why not: Confuses 0.5 with the perfect-ranking value, which is an AUC of 1.0.
  2. classifies exactly half of the borrowers correctly at the best cutoff.Why not: Treats AUC as an accuracy rate rather than a ranking measure over all cutoffs.
  3. ranks defaulters no better than random chance, so it has no discriminating power. - correct
  4. produces predicted default probabilities that average 50%.Why not: Confuses AUC with the mean predicted probability, which is a calibration matter.

AUC is the probability that a random defaulter receives a higher score than a random non-defaulter. 0.5 is the chance level, the diagonal of the ROC space.

A harder one

A bank tests a default model on 1,000 loans, 20 of which defaulted. The model flags no loan as a default and reports accuracy of 98%. The best assessment is that:

  1. the model is strong, because an accuracy of 98% comfortably exceeds the acceptance thresholds normally applied to credit models.Why not: Ignores class imbalance, so a model that never flags default earns 98% by construction.
  2. the model has an AUC of 98% because its accuracy is 98%.Why not: Confuses accuracy with AUC; a model giving no discrimination has an AUC of about 0.5.
  3. the 98% is a sign of overfitting, so a number of the explanatory variables should now be removed from the model.Why not: Misreads the imbalance effect as overfitting, which would show as a train-versus-holdout gap.
  4. the accuracy only equals the non-default share, and the model detects none of the 20 defaults. - correct

Predicting all non-default gives 980/1,000 = 98% accuracy but a recall of 0 of 20. With imbalanced classes, use recall, precision, ROC or AUC rather than accuracy alone.

The hardest tier

At a cutoff of 0.20, a logistic model flags 28 of 40 actual defaulters and 54 of 360 actual non-defaulters as defaults. The point on its ROC curve for this cutoff, as (false positive rate, true positive rate), is closest to:

  1. (0.70, 0.15)Why not: Swaps the axes: the ROC curve puts the false positive rate on the horizontal axis and the true positive rate on the vertical.
  2. (0.15, 0.70) - correct
  3. (0.66, 0.70)Why not: Divides false positives by all flagged loans (54/82) rather than by all actual non-defaulters.
  4. (0.15, 0.34)Why not: Uses precision, 28 of 82 flagged loans, in place of the true positive rate of 28 of 40.

TPR = 28/40 = 0.70. FPR = 54/360 = 0.15. The point is (0.15, 0.70); each rate is conditioned on the actual class, not on the flag.

In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.

Start free →

More in Machine Learning and Prediction