Eigenvectors: the directions a matrix only stretches

Most arrows turn when a matrix acts on them. Eigenvectors don't: they are only stretched, by their eigenvalue. They describe symmetric matrices completely, and they are where PCA comes from.

10 min Prerequisites: Module 2 (a matrix is a transformation)

Symbols in this lessonλ, 𝐯 eigenvalue / eigenvector det determinant I identity matrix tr trace

1. The idea

Most arrows change direction when a matrix acts on them. A few special ones don't: the matrix only stretches them, shrinks them or flips them round. Those directions are the eigenvectors of the matrix, and the factor each one is stretched by is its eigenvalue: A𝐯 = λ𝐯.

For a symmetric matrix the eigenvectors are at right angles to each other, and together they describe the matrix completely: along those directions it is nothing but a scaling. Writing it that way, A = QΛQᵀ, is called diagonalising. It comes back in PCA (the eigenvectors of a covariance matrix) and in studies of training (the eigenvalues of the loss's Hessian say how curved it is).

Not every matrix has real eigenvectors. A rotation turns every arrow, so none keeps its direction; its eigenvalues are complex numbers. A shear keeps only one direction. The widget has all of them.

2. Try it: find the directions that don't turn

Turn the unit arrow 𝐯 and watch A𝐯. When A𝐯 lines up with 𝐯, you have found an eigenvector. Then try the shear and the rotation.

3. The math

The definition

A\mathbf{v} = \lambda\mathbf{v}, \qquad \mathbf{v} \neq \mathbf{0}

𝐯 keeps its line; λ says how much it is stretched (negative: flipped).

Finding the eigenvalues

\det(A - \lambda I) = 0

A − λI must squash some direction to zero, so its determinant is 0.

An example

A = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}: \quad \lambda = 3,\ \mathbf{v} = (1, 1); \qquad \lambda = 1,\ \mathbf{v} = (1, -1)

Check: A(1, 1) = (3, 3) = 3·(1, 1).

Symmetric matrices diagonalise

A = Q\Lambda Q^\top, \qquad Q^\top Q = I

Q holds the eigenvectors as columns, Λ the eigenvalues on its diagonal.

Trace and determinant

\operatorname{tr} A = \sum_i \lambda_i \qquad \det A = \prod_i \lambda_i

For the example: 2 + 2 = 3 + 1 and 2·2 − 1·1 = 3·1.

4. The code (python)

mm3_eigen.py

import numpy as np

A = np.array([[2.0, 1.0],
              [1.0, 2.0]])
vals, vecs = np.linalg.eigh(A)          # eigh: for symmetric matrices, smallest first
print(vals)                             # [1. 3.]
v = vecs[:, 1]                          # the eigenvector for 3
print(np.allclose(A @ v, 3 * v))        # True
print(round(np.trace(A), 6), round(vals.sum(), 6))          # 4.0 4.0
print(round(np.linalg.det(A), 6), round(vals.prod(), 6))    # 3.0 3.0

R = np.array([[0.0, -1.0],
              [1.0,  0.0]])             # a quarter turn
print(np.linalg.eigvals(R))             # [0.+1.j 0.-1.j]

5. You’ll see this in

Reference notes: matrix decompositions, SVD and LoRA
Principal component analysis in the catalog
Module 2: a matrix is a transformation the same grid, now looking for directions that don't turn

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done

Q1/4 What are the eigenvalues of diag(2, 5)?

Q2/4 A𝐯 = λ𝐯 with λ = −1. What does A do to 𝐯?

Q3/4 Which of these has no real eigenvectors?

Q4/4 A 2 × 2 matrix has eigenvalues 3 and 1. What is its determinant?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.

Learn machine learning from scratch