Explain what the training, validation and test sub-samples are, and the distinct purpose each serves.
The idea
A data set is divided so that a model is never judged on data it has already seen. The training sample is used to estimate the model parameters. The validation sample is used to tune hyperparameters and to choose between competing models, such as the penalty size or the number of clusters, so that over-fitting can be detected. The test sample is touched once, at the end, to give an unbiased estimate of how the final model will perform on new data. Typical splits give most of the data to training, for example 60 to 80 percent. With time series the sets should follow chronological order, with the test set the most recent data. Random shuffling would leak future information. Repeated tuning against the test set destroys its purpose.
A practice question
What is the main purpose of the validation sample?
- To estimate the model's parameters.Why not: This is the role of the training sample.
- To give a final unbiased estimate of performance on new data.Why not: This is the role of the test sample.
- To tune hyperparameters and choose between candidate models. - correct
- To increase the number of training observations.Why not: The validation sample is held out and not used to fit parameters.
Training fits parameters, validation selects hyperparameters and models, and the test sample provides the final, unbiased assessment.
A harder one
A quant tries 40 hyperparameter combinations and keeps the one with the lowest error on the test sample, then reports that error as the expected live performance. What is the problem?
- The test sample has been used for selection, so the reported error is optimistically biased. - correct
- The test sample is too small to give a reliable measure of the error.Why not: The issue is its use for tuning, not its size.
- Hyperparameters can only ever be tuned on the training sample itself.Why not: Tuning belongs on the validation sample, not the training sample.
- Forty combinations is too many for any model to be estimated.Why not: The count is not the problem; the choice of sample is.
Once the test sample guides choices it no longer represents unseen data. The tuning should use the validation sample.
The hardest tier
A model's mean error is 2 percent on the training sample, 9 percent on the validation sample, and 10 percent on the test sample. Which conclusion is best supported?
- The model is underfit, because errors rise from training to test.Why not: Underfitting shows high error on all samples, including training.
- The test result is invalid because it differs from the validation result.Why not: The validation and test results are close, which supports the 10 percent estimate.
- The model is overfit, and about 10 percent is the better estimate of live performance than the 2 percent training error. - correct
- Performance on live data should be about 2 percent, since the training sample is the largest.Why not: Training error is an optimistic in-sample measure.
A large training-to-validation gap signals overfitting. Validation and test errors agree, so around 10 percent is the credible estimate.
In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.
Start free →More in Machine-Learning Methods
- Discuss how machine-learning techniques differ from classical econometrics in both philosophy and practice.
- Compare the two methods used to rescale variables when preparing data, and apply them.
- Examine how underfitting and overfitting differ and what follows from each, and outline possible remedies for both.
- Use principal components analysis to compress a set of features into fewer dimensions, and explain why that works.
- Outline how K-means assigns the observations in a sample to separate clusters.
- Describe how natural language processing works and what it is used for.
- Distinguish unsupervised learning models from supervised ones and from reinforcement learning models.
- Describe the operation of reinforcement learning and calculate the Q-values used when making decisions.