Charta is on Google Play: practise on the go. Get the app
Charta

Explain how a set of learners is combined into an ensemble.

FRM® Part I · Quantitative Analysis · Machine Learning and Prediction · FRM1.QA.15.f

The idea

An ensemble combines many learners so the group predicts better than any one. Bagging, or bootstrap aggregation, trains the same type of learner on many bootstrap resamples drawn with replacement. It averages the outputs for regression or takes a majority vote for classification. Bagging mainly reduces variance, and the gain depends on how correlated the learners are. Variance = ρ × σ² + (1 − ρ) × σ² ÷ B. This is the variance of an average of B learners, each with variance σ² and with pairwise correlation ρ. A random forest lowers ρ by letting each split consider only a random subset of features. Boosting is sequential. Each new learner concentrates on what earlier learners got wrong, by reweighting observations or by fitting residuals, and is added with a small learning rate. Boosting mainly reduces bias. Stacking trains a second model to combine the learners' predictions. A common trap is assuming that more learners cure correlated errors.

Formulas

Variance of an average of correlated learners

Var(ensemble) = ρ × s² + (1 − ρ) × s² ÷ B

ρ
the pairwise correlation between learners' predictions
s
the standard deviation of one learner's prediction
B
the number of learners

More learners shrink only the second term; ρ × s² is a floor. Random forests attack ρ by sampling features at each split.

A practice question

From the question bank, with the reason each wrong option is wrong.

Bagging builds an ensemble by:

  1. training learners one after another, each focused on the errors of its predecessors.Why not: Describes boosting, which is sequential rather than parallel on resamples.
  2. training a final model that learns how to weight the predictions of different learners.Why not: Describes stacking, which uses a meta-learner.
  3. training the same type of learner on many bootstrap resamples and averaging, or voting on, their predictions. - correct
  4. training one learner on the full sample and adding a penalty on its coefficients.Why not: Describes regularization of a single model, not an ensemble.

Bagging = bootstrap aggregation. Each learner sees a resample, and combining them lowers variance.

A harder one

Five independent classifiers each have a 70% probability of classifying a given loan correctly. A majority vote, requiring at least 3 of the 5 to be correct, is used. The probability that the ensemble classifies the loan correctly is closest to:

  1. 70.0%Why not: Assumes that voting leaves accuracy unchanged, ignoring the gain from combining independent classifiers.
  2. 83.7% - correct
  3. 52.8%Why not: Requires at least 4 of the 5 to be correct instead of at least 3.
  4. 96.9%Why not: Counts 2 of 5 correct as a majority.

P(at least 3 of 5) = 10(0.7³)(0.3²) + 5(0.7^4)(0.3) + 0.7^5 = 0.3087 + 0.3602 + 0.1681 = 0.837.

The hardest tier

Each of 100 trees has a prediction variance of 0.04, and the pairwise correlation between any two trees' predictions is 0.30. The variance of the average prediction is closest to:

  1. 0.0004Why not: Assumes independent trees and uses s2/B, ignoring the correlation.
  2. 0.0400Why not: Assumes that averaging does not change the variance of a single tree.
  3. 0.0123 - correct
  4. 0.0280Why not: Uses only (1 - rho) x s2, omitting division by the number of trees B in that term.

Var = rho x s2 + (1 - rho) x s2/B = 0.3 x 0.04 + 0.7 x 0.04/100 = 0.0120 + 0.00028 = 0.01228. The correlation sets a floor that more trees cannot remove.

In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.

Start free →

More in Machine Learning and Prediction