Dimensionality reduction with PCA and kernel PCA

Fewer numbers that keep the story

Stocks move together, so 54 monthly return series carry much less than 54 independent pieces of information. Principal component analysis, PCA, finds new axes, the principal components, that are ordered by how much of the variance of the data they carry. The first axis points along the direction of greatest spread, the second is perpendicular to it and carries the next most, and so on. In terms of the covariance matrix, the axes are its eigenvectors and the variances along them are its eigenvalues. The share of variance of component $k$ is $\lambda_k/\sum_j\lambda_j$ . Keeping the first few components gives a short description of the data with a small loss.

\Sigma v_k=\lambda_k v_k,\qquad r_k=\frac{\lambda_k}{\sum_j\lambda_j},\qquad k(x,y)=e^{-\gamma\|x-y\|^{2}}

(1)

PCA works on centred data, and on features with comparable scales, so standardise first. Its axes are linear, which suits the market because most stocks share one common move. Kernel PCA does PCA in a space built by a kernel such as the RBF kernel $e^{-\gamma d^{2}}$ , which scores how close two points are, so it can follow curved structures. Two warnings apply. A component is a mix of all the inputs, so naming it is guesswork. And the leading components are the biggest sources of variance, not the most predictive ones. In finance the first component is the market, the next few are sectors, and a model fitted on components should still be checked on later data.

A 2 by 2 example and 54 stocksTake the covariance matrix

\begin{pmatrix}2&1\\1&2\end{pmatrix}

. Its eigenvalues are

3

and

1

, so the first component carries

3/(3+1)=75\%

of the variance and the second

25\%

. The first axis is

(1,1)/\sqrt2

. The point

(1,1)

projects on it at

(1+1)/\sqrt2=1.414

, while the point

(1,-1)

projects at

0

and is lost by a one-component model, with a squared error of

2

. For the RBF kernel with

\gamma=0.5

and squared distance

2

, the similarity is

e^{-1}=0.368

. On the

319

monthly returns of the

54

real stocks, the first component carries

29.5\%

of the variance, the second

8.4\%

and the third

5.5\%

. It takes

22

components to reach

80\%

and

32

to reach

90\%

, so a short description saves

32

of the

54

numbers at the

80\%

level.

Axes by variance: the first component is the direction of greatest spread, each next one is perpendicular.
Standardise first: PCA follows variance, so a feature with big numbers would dominate.
Variance is not prediction: the biggest components are not necessarily the useful ones.

Lab · what's graded

The share of the first component15 pts
A projection10 pts
A reconstruction error10 pts
An RBF kernel value10 pts
Numbers saved5 pts
Why standardise before PCA10 pts
The steps of PCA10 pts
Match the vocabulary15 pts
A big component is not a useful one15 pts

Fewer numbers that keep the story

\Sigma v_k=\lambda_k v_k,\qquad r_k=\frac{\lambda_k}{\sum_j\lambda_j},\qquad k(x,y)=e^{-\gamma\|x-y\|^{2}}

(1)

A 2 by 2 example and 54 stocksTake the covariance matrix

\begin{pmatrix}2&1\\1&2\end{pmatrix}

. Its eigenvalues are

3

and

1

, so the first component carries

3/(3+1)=75\%

of the variance and the second

25\%

. The first axis is

(1,1)/\sqrt2

. The point

(1,1)

projects on it at

(1+1)/\sqrt2=1.414

, while the point

(1,-1)

projects at

0

and is lost by a one-component model, with a squared error of

2

. For the RBF kernel with

\gamma=0.5

and squared distance

2

, the similarity is

e^{-1}=0.368

. On the

319

monthly returns of the

54

real stocks, the first component carries

29.5\%

of the variance, the second

8.4\%

and the third

5.5\%

. It takes

22

components to reach

80\%

and

32

to reach

90\%

, so a short description saves

32

of the

54

numbers at the

80\%

level.

Axes by variance: the first component is the direction of greatest spread, each next one is perpendicular.
Standardise first: PCA follows variance, so a feature with big numbers would dominate.
Variance is not prediction: the biggest components are not necessarily the useful ones.

Lab · what's graded

The share of the first component15 pts
A projection10 pts
A reconstruction error10 pts
An RBF kernel value10 pts
Numbers saved5 pts
Why standardise before PCA10 pts
The steps of PCA10 pts
Match the vocabulary15 pts
A big component is not a useful one15 pts

Dimensionality reduction with PCA and kernel PCA

Fewer numbers that keep the story

Lab · what's graded

Log in to run your code.

Dimensionality reduction with PCA and kernel PCA

Fewer numbers that keep the story

Lab · what's graded

Log in to run your code.