Charta is on Google Play: practise on the go. Get the app
Charta

Use a confusion matrix to compare the logistic regression and neural network approaches to classification.

FRM® Part I · Quantitative Analysis · Machine Learning and Prediction · FRM1.QA.15.i

The idea

A confusion matrix cross-tabulates actual against predicted classes at a given cutoff. It holds true positives (defaults caught), false positives (false alarms), false negatives (defaults missed) and true negatives. Accuracy = (TP + TN) ÷ total. Recall = TP ÷ (TP + FN). It is also called sensitivity. Precision = TP ÷ (TP + FP). Specificity = TN ÷ (TN + FP). F1 = 2TP ÷ (2TP + FP + FN). F1 also equals 2 × precision × recall ÷ (precision + recall). To compare a logistic regression with a neural network, compute these from each model's matrix on the same holdout data and weigh them by what errors cost. A missed default usually costs far more than a false alarm, so recall and the cost-weighted total matter more than raw accuracy. Raising recall normally lowers precision, and a matrix reflects one cutoff only. The usual mistake is to prefer the model with the higher accuracy under imbalanced classes.

Formulas

Recall (sensitivity)

recall = TP ÷ (TP + FN)

TP
true positives, defaults caught
FN
false negatives, defaults missed

Conditioned on the actual class. Usually the measure to watch when a missed default costs far more than a false alarm.

Precision

precision = TP ÷ (TP + FP)

TP
true positives, defaults caught
FP
false positives, false alarms

Conditioned on the flag. Raising recall normally lowers precision.

F1 score

F1 = 2 × TP ÷ (2 × TP + FP + FN)

TP
true positives
FP
false positives
FN
false negatives

The harmonic mean of precision and recall, so extra false alarms can pull it down even when recall rises.

Accuracy

accuracy = (TP + TN) ÷ total

TP
true positives
TN
true negatives
total
all classified cases

Misleading with imbalanced classes: predicting no default at all scores high when defaults are rare.

A practice question

From the question bank, with the reason each wrong option is wrong.

In a confusion matrix for a default classifier, recall (sensitivity) is calculated as:

  1. TP / (TP + FP), the share of flagged loans that actually default.Why not: Defines precision, which conditions on flagged loans, not on actual defaults.
  2. TP / (TP + FN), the share of actual defaults that the model flags. - correct
  3. (TP + TN) / total, the share of all loans classified correctly.Why not: Defines accuracy, which pools both classes.
  4. TN / (TN + FP), the share of actual non-defaults that are cleared.Why not: Defines specificity, the true negative rate.

Recall = TP/(TP + FN): of the loans that really defaulted, the fraction the model caught.

A harder one

A logistic model is tested on 500 loans, with results TP = 45, FP = 30, FN = 15 and TN = 410. Its recall is:

  1. 60.0%Why not: Computes precision, 45/(45 + 30), instead of recall.
  2. 91.0%Why not: Computes accuracy, (45 + 410)/500.
  3. 75.0% - correct
  4. 93.2%Why not: Computes specificity, 410/(410 + 30), instead of recall.

Recall = TP/(TP + FN) = 45/(45 + 15) = 0.75. The other figures are precision, accuracy and specificity.

The hardest tier

On 1,000 validation loans, a logistic model has TP = 45, FN = 15, FP = 60, TN = 880 and a neural network has TP = 51, FN = 9, FP = 110, TN = 830. A missed default costs USD 100 thousand and a false alarm USD 5 thousand. Which model has the lower total cost, and by how much?

  1. The neural network, by USD 350 thousand. - correct
  2. The neural network, by USD 600 thousand.Why not: Counts only the six fewer missed defaults and ignores the 50 extra false alarms.
  3. The logistic model, with 44 fewer misclassified loans.Why not: Compares error counts, ignoring that the two errors have very different costs.
  4. The logistic model, by USD 4,970 thousand.Why not: Swaps the costs, charging USD 100 thousand per false alarm and USD 5 thousand per miss.

Logistic: 15 x 100 + 60 x 5 = 1,800. Neural network: 9 x 100 + 110 x 5 = 1,450. The neural network is cheaper by 350, although it makes more errors in total.

In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.

Start free →

More in Machine Learning and Prediction