Charta is on Google Play: practise on the go. Get the app
Charta

Explain the usefulness of regularization and set ridge regression alongside LASSO for comparison.

FRM® Part I · Quantitative Analysis · Machine Learning and Prediction · FRM1.QA.15.d

The idea

Regularization adds a penalty on coefficient size to the fitting objective, to curb overfitting when there are many predictors or they are correlated. The model gives up a little bias in exchange for lower variance, so it predicts better on new data. Ridge = RSS + λ × (sum of squared coefficients). This is the L2 penalty. LASSO = RSS + λ × (sum of absolute coefficients). This is the L1 penalty. Both shrink coefficients toward zero as λ grows, and λ = 0 gives ordinary least squares. Because of its geometry, LASSO can set coefficients exactly to zero and so performs variable selection, while ridge shrinks all coefficients but rarely to zero. With highly correlated predictors ridge tends to share weight among them, whereas LASSO tends to keep one. Predictors should be standardised first, and λ is chosen by cross-validation. The intercept is normally not penalised.

Formulas

Ridge regression objective

ridge objective = RSS + λ × Σβ²

RSS
the residual sum of squares
λ
the penalty parameter, chosen by cross-validation
β
each slope coefficient; the intercept is not penalised

The L2 penalty shrinks coefficients toward zero but rarely to exactly zero. λ = 0 gives ordinary least squares; standardise predictors first.

LASSO objective

LASSO objective = RSS + λ × Σ|β|

RSS
the residual sum of squares
λ
the penalty parameter, chosen by cross-validation
β
each slope coefficient; the intercept is not penalised

The L1 penalty can set coefficients exactly to zero, so LASSO selects variables. With correlated predictors it tends to keep one.

A practice question

From the question bank, with the reason each wrong option is wrong.

Regularization in regression is mainly used to:

  1. reduce overfitting by penalising large coefficients, accepting a little more bias for lower variance. - correct
  2. raise in-sample fit as far as possible by adding more explanatory terms.Why not: The opposite of the aim: regularization restrains complexity and may lower in-sample fit.
  3. make the regression residuals normally distributed.Why not: Confuses a distributional assumption with a penalty; regularization does not change the residual distribution.
  4. force the fitted line through the origin by removing the intercept.Why not: The intercept is normally left unpenalised and kept; regularization acts on slope sizes.

A penalty on coefficient size trades some bias for a larger reduction in variance, improving out-of-sample prediction.

A harder one

As the penalty parameter lambda in a ridge regression is raised from zero to a large value, the usual effect is that:

  1. coefficients shrink toward zero, bias falls and the variance of predictions rises.Why not: Reverses the tradeoff: a heavier penalty raises bias and lowers variance.
  2. coefficients shrink toward zero, bias rises and the variance of predictions falls. - correct
  3. coefficients move toward their ordinary least squares values.Why not: Lambda = 0 gives OLS; increasing lambda moves coefficients away from OLS toward zero.
  4. coefficients shrink toward zero and both bias and variance fall.Why not: Ignores the tradeoff: shrinking toward zero adds bias even while it cuts variance.

A larger lambda gives a simpler, more constrained model: more bias, less variance. The best lambda balances them, usually by cross-validation.

The hardest tier

A regression has a residual sum of squares of 18.0 and fitted slope coefficients of 2.0, -1.5 and 0.5. With penalty parameter lambda = 2, the ridge objective (RSS plus penalty on the slopes) equals:

  1. 26.0Why not: Uses the sum of absolute values (L1 penalty, as in LASSO): 18 + 2 x 4.
  2. 24.5Why not: Adds the sum of squared coefficients (6.5) without multiplying by lambda.
  3. 31.0 - correct
  4. 20.0Why not: Squares the sum of the coefficients, (2.0 - 1.5 + 0.5)² = 1, instead of summing the squares.

Penalty = lambda x (4.00 + 2.25 + 0.25) = 2 x 6.5 = 13. Objective = 18 + 13 = 31.0.

In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.

Start free →

More in Machine Learning and Prediction