PCA: the directions where the data varies most

Centre the data, find the direction that keeps the most spread, then the next at right angles. The answer is the eigenvectors of the covariance matrix.

10 min Prerequisites: Lessons 15 and 16 (eigenvectors, SVD)

Symbols in this lessonλ, 𝐯 eigenvalue / eigenvector Σ covariance matrix x̄ bar argmin arg min / arg max

1. The idea

Principal component analysis finds the directions along which data varies most. Centre the data by subtracting each feature's mean, then look for the unit direction 𝐮 that makes the projected values 𝐮ᵀ𝐱 as spread out as possible. That direction is the first principal component. The second is the best direction at right angles to the first, and so on.

The answer is linear algebra you have now met: the principal components are the eigenvectors of the covariance matrix, and each eigenvalue is the variance along its direction. In practice code takes the SVD of the centred data instead, which gives the same directions more accurately.

Keep the first k components and each example shrinks from d numbers to k, keeping as much of the spread as any k directions can. That is used to compress features, to plot high-dimensional embeddings in two dimensions, and to strip out noise.

2. Try it: keep the most spread

Turn the line and watch each point drop onto it. The spread of the dropped points is the variance that direction keeps. Find the best direction by hand, then press the button.

3. The math

Centre the data

\tilde{X} = X - \bar{\mathbf{x}}

Subtract each feature's mean from its column (broadcasting, Module 2).

The covariance matrix

C = \frac{1}{n}\,\tilde{X}^\top\tilde{X} \in \mathbb{R}^{d \times d}

Symmetric: entry (i, j) says how features i and j vary together.

The first component

\mathbf{u}_1 = \arg\max_{\|\mathbf{u}\|_2 = 1}\ \mathbf{u}^\top C\,\mathbf{u}

𝐮ᵀC𝐮 is the variance of the data projected onto 𝐮.

It's an eigenvector

C\mathbf{u}_i = \lambda_i\,\mathbf{u}_i, \qquad \lambda_i = \text{variance along } \mathbf{u}_i

Sort by eigenvalue: biggest first.

Project onto k components

Z = \tilde{X}\,U_k \in \mathbb{R}^{n \times k}

U_k holds the first k components as columns.

4. The code (python)

mm3_pca.py

import numpy as np

rng = np.random.default_rng(1)
t = rng.normal(size=200)
X = np.column_stack([t, 0.5 * t + 0.2 * rng.normal(size=200)])   # points along a line, plus noise

Xc = X - X.mean(axis=0)
C = Xc.T @ Xc / len(Xc)
vals, vecs = np.linalg.eigh(C)
vals, vecs = vals[::-1], vecs[:, ::-1]          # biggest first
print((vals / vals.sum()).round(3))             # [0.977 0.023]
z = Xc @ vecs[:, 0]                             # every point squeezed to one number
print(z.shape)                                  # (200,)

U, s, Vt = np.linalg.svd(Xc, full_matrices=False)   # the same direction from the SVD, up to sign
print(np.allclose(abs(Vt[0]), abs(vecs[:, 0])))       # True

5. You’ll see this in

Principal component analysis in the catalog
Reference notes: matrix decompositions, SVD and LoRA
Module 2: tensors and broadcasting centring is one broadcast subtraction

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done

Q1/4 What is the first step of PCA?

Q2/4 The principal components are the eigenvectors of which matrix?

Q3/4 A 2-D dataset's covariance has eigenvalues 9 and 1. How much variance does the first component keep?

Q4/4 You project n embeddings of length 768 onto their first 2 components. What shape do you get?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.

Learn machine learning from scratch