Charta is on Google Play: practise on the go. Get the app
Charta

Use principal components analysis to compress a set of features into fewer dimensions, and explain why that works.

FRM® Part I · Quantitative Analysis · Machine-Learning Methods · FRM1.QA.14.e

The idea

Principal components analysis compresses many correlated features into a few new variables called principal components. Each component is a linear combination of the original features, the components are uncorrelated with each other, and they are ordered so that the first captures the largest share of total variance, the second the largest share of what remains, and so on. Each component explains a share of the total variance. Share = eigenvalue ÷ sum of all eigenvalues. For standardised data the sum of the eigenvalues equals the number of features. You keep enough components to reach a target share, such as 90 percent, and replace the original features by their scores. It works because correlated features carry overlapping information, so a few directions hold most of the variation. PCA is unsupervised: it ignores the target.

Formulas

Share of variance explained by a principal component

share = eigenvalue ÷ sum of eigenvalues

eigenvalue
the variance captured by that component
sum of eigenvalues
the total variance, equal to the number of features for standardised data

Cumulate the shares in order until the target is reached. The components with most variance need not be the most predictive.

A practice question

From the question bank, with the reason each wrong option is wrong.

Why can PCA compress a large set of features into far fewer components with little loss of information?

  1. Components with small eigenvalues are always the most predictive.Why not: Small-variance components carry little of the data's variation.
  2. The original features are correlated, so a few directions capture most of the variation. - correct
  3. PCA removes the target variable from the data.Why not: PCA never uses the target; this does not explain compression.
  4. PCA increases the variance of each feature to make features distinct.Why not: PCA does not change the total variance; it redistributes it.

Correlated features carry overlapping information. PCA rotates them into uncorrelated directions, and a few of them hold most of the total variance.

A harder one

Four standardised features give PCA eigenvalues of 2.4, 0.9, 0.5 and 0.2. What share of total variance does the first principal component explain?

  1. 24 percentWhy not: Treats the eigenvalue 2.4 as a percentage.
  2. 40 percentWhy not: Takes the share left unexplained, 1 - 0.6.
  3. 60 percent - correct
  4. 82.5 percentWhy not: The cumulative share of the first two components.

Total variance is 2.4 + 0.9 + 0.5 + 0.2 = 4.0, equal to the number of standardised features, and 2.4 / 4.0 = 60 percent.

The hardest tier

The first loading vector of a PCA on three standardised features is (0.48, 0.60, 0.64). An observation has standardised values (1.5, -1.0, 0.5). What is its score on the first component?

  1. 1.64Why not: Adds the absolute values of the loading-weighted terms, ignoring the negative sign of the second feature.
  2. 0.33Why not: Takes a simple average of the standardised values instead of weighting by loadings.
  3. 0.19Why not: Uses squared loadings as weights instead of the loadings themselves.
  4. 0.44 - correct

Score = 0.48 x 1.5 + 0.60 x (-1.0) + 0.64 x 0.5 = 0.72 - 0.60 + 0.32 = 0.44.

In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.

Start free →

More in Machine-Learning Methods