Examine how underfitting and overfitting differ and what follows from each, and outline possible remedies for both.
The idea
Underfitting means the model is too simple to capture the real pattern. Error is high on the training data and equally high on new data, a problem of high bias. Overfitting means the model is too complex and has learned noise as well as signal. Training error is very low but validation or test error is much higher, a problem of high variance. The gap between training and validation error is the diagnostic. Remedies for overfitting are more training data, fewer features, a simpler model, regularisation such as LASSO or ridge penalties, and early stopping. Remedies for underfitting are more features, interaction or nonlinear terms, a more flexible model, or a weaker penalty. Excess regularisation can swing a model from overfit to underfit.
A practice question
Which description fits overfitting?
- High error on both the training data and new data.Why not: Describes underfitting, where error is high on training data as well.
- Low error on new data but high error on the training data.Why not: Not a typical pattern; overfit models do well in-sample, not out of sample.
- Equal and low error on training, validation and test data.Why not: Indicates a good fit, not overfitting.
- Very low error on the training data but substantially higher error on new data. - correct
An overfit model has learned noise in the training data, so it generalises poorly: low training error, high out-of-sample error.
A harder one
A model has a training error of 1.5 percent and a validation error of 12 percent. Which remedy is most appropriate?
- Add more interaction and polynomial features.Why not: Makes the model more complex and worsens overfitting.
- Apply regularisation or reduce the number of features. - correct
- Reduce the penalty strength to give the model more flexibility.Why not: Increases flexibility, which suits underfit models.
- Test the model only on the training sample to confirm the low error.Why not: Evaluating in sample hides the problem.
The large gap indicates overfitting. Regularisation, fewer features, simpler models or more data reduce variance.
The hardest tier
Polynomial models of degree 1 to 5 are fitted. Training and validation errors in percent are: degree 1, 10.0 and 10.5; degree 2, 6.0 and 7.0; degree 3, 4.0 and 5.5; degree 4, 2.0 and 6.8; degree 5, 0.5 and 9.0. Which model should be selected?
- Degree 5, which has the lowest training error.Why not: Selects on training error, which always falls with complexity.
- Degree 1, which has the smallest training-validation gap.Why not: Minimises the gap but ignores that the model underfits, with the highest validation error.
- Degree 4, the first model with training error of 2 percent or less.Why not: Uses an arbitrary training-error threshold instead of validation performance.
- Degree 3, which has the lowest validation error. - correct
Choose the model with the lowest validation error: 5.5 percent at degree 3. Beyond that, validation error rises as the fit becomes overfit; below it the model is underfit.
In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.
Start free →More in Machine-Learning Methods
- Discuss how machine-learning techniques differ from classical econometrics in both philosophy and practice.
- Compare the two methods used to rescale variables when preparing data, and apply them.
- Explain what the training, validation and test sub-samples are, and the distinct purpose each serves.
- Use principal components analysis to compress a set of features into fewer dimensions, and explain why that works.
- Outline how K-means assigns the observations in a sample to separate clusters.
- Describe how natural language processing works and what it is used for.
- Distinguish unsupervised learning models from supervised ones and from reinforcement learning models.
- Describe the operation of reinforcement learning and calculate the Q-values used when making decisions.