Show how a decision tree is built and how it should be read.
The idea
A decision tree predicts by asking a sequence of yes-or-no questions about the features. It is grown top-down and greedily by recursive binary splitting. Starting with all observations in the root, at each node the algorithm tries every feature and threshold and picks the split that most reduces impurity. For classification, impurity is measured by Gini impurity or by entropy. Gini = 1 − (p₁² + p₂² + …). Here p₁, p₂ and so on are the class proportions in the node. For regression, impurity is measured by the sum of squared errors. The impurity of the two children is weighted by their sizes. Splitting stops at a limit such as maximum depth or minimum leaf size, or the tree is grown fully and then pruned. To read a tree, follow the path from the root. Each leaf gives a rule, with the predicted class from the majority or the predicted value from the leaf mean. Deep trees overfit, and a regression tree cannot predict outside the range of its training leaves.
Formulas
Gini impurity of a node
Gini = 1 − Σpᵢ²
- pᵢ
- the proportion of the node's observations in class i
Zero for a pure node. A split's quality is the parent's impurity minus the size-weighted impurity of the two children.
A practice question
A classification tree is grown by:
- fitting a single linear equation to the whole sample and then rounding the fitted value to the nearest class label.Why not: Describes a linear probability model, not recursive partitioning.
- starting with all observations and repeatedly choosing the feature and threshold that most reduces impurity, then splitting each resulting node in turn. - correct
- evaluating every possible complete tree and keeping the one with the lowest error.Why not: Tree growth is greedy, one split at a time, because exhaustive search is infeasible.
- randomly assigning observations to different branches and then averaging the outcomes observed within each branch.Why not: Splits are chosen to reduce impurity, not assigned at random.
Recursive binary splitting is greedy and top-down: each split is the best available at that node, without looking ahead.
A harder one
A fitted default tree works as follows. The root asks whether debt-to-income exceeds 40%. If yes, it asks whether the borrower has at least 2 missed payments: yes leads to a leaf with default probability 0.62, no to a leaf with 0.25. If the root answer is no, the leaf has default probability 0.08. A borrower has a debt-to-income of 45% and one missed payment. The tree's estimate of default probability is:
- 0.25 - correct
- 0.62Why not: Ignores the missed payment split and takes the higher leaf below the first yes branch.
- 0.08Why not: Ignores the root split and assigns the leaf for borrowers with low debt-to-income.
- 0.32Why not: Averages the three leaf values, although a path leads to exactly one leaf.
Follow the path: debt-to-income above 40% goes to the second question; one missed payment is below 2, so the answer is no and the leaf is 0.25.
The hardest tier
A parent node has 100 loans with 40 defaults. A candidate split sends 60 loans (10 defaults) to the left child and 40 loans (30 defaults) to the right child. Using Gini impurity, the reduction in impurity from this split is closest to:
- 0.317Why not: Reports the weighted impurity of the children rather than its reduction from the parent.
- 0.154Why not: Averages the two child impurities without weighting by size before subtracting.
- 0.144Why not: Weights each child's impurity by the other child's share of the observations.
- 0.163 - correct
Parent 0.48. Left Gini = 1 - (1/6)² - (5/6)² = 0.278; right = 1 - 0.75² - 0.25² = 0.375. Weighted = 0.6 x 0.278 + 0.4 x 0.375 = 0.317. Reduction = 0.48 - 0.317 = 0.163.
In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.
Start free →More in Machine Learning and Prediction
- Say how forecasting a quantity can draw on a linear regression and on a logistic regression, and what each does there.
- Assess how well logistic regression models predict.
- Outline the methods for encoding categorical variables and apply them.
- Explain the usefulness of regularization and set ridge regression alongside LASSO for comparison.
- Explain how a set of learners is combined into an ensemble.
- Describe the underlying intuition and the working steps of two classification methods, K nearest neighbors and support vector machines.
- Describe how neural networks are put together and how the values of their weights are arrived at.
- Use a confusion matrix to compare the logistic regression and neural network approaches to classification.