The SVD: rotate, stretch, rotate

Every matrix, of any shape, is a rotation, a stretch along the axes and another rotation. Keep the biggest stretches and you get the best low-rank copy, the idea behind LoRA.

10 min Prerequisites: Lesson 15 (eigenvectors)

Symbols in this lessonrank rank ᵀ transpose diag diagonal matrix ≪ much less than

1. The idea

Eigenvectors only work for square matrices, and not even for all of those. The singular value decomposition works for every matrix: A = UΣVᵀ. Read it right to left: Vᵀ rotates, Σ stretches along the axes, and U rotates again. Any matrix, whatever its shape, is a rotate, a stretch and a rotate.

The stretch factors σ₁ ≥ σ₂ ≥ … ≥ 0 on Σ's diagonal are the singular values. They say how much A stretches in its most-stretched direction, in the next, and so on. The number of non-zero ones is the rank of A.

Keep only the biggest k singular values and you get the best rank-k approximation of A: no other rank-k matrix is closer. That is low-rank compression, and it is the intuition behind LoRA, which stores a fine-tuning update ΔW as the product of two thin matrices, a rank-r matrix, betting that what the update needs to do lives in a few directions.

2. Try it: three moves, one ellipse

Set a matrix with the four sliders, then step through Vᵀ, Σ and U and watch the unit circle become an ellipse. Then keep only σ₁ and see what a rank-1 copy of A does.

3. The math

The decomposition

A = U\Sigma V^\top, \qquad U^\top U = I, \quad V^\top V = I

U and V are orthogonal (rotations or flips); Σ is diagonal.

Shapes

\underbrace{A}_{m \times n} = \underbrace{U}_{m \times m}\ \underbrace{\Sigma}_{m \times n}\ \underbrace{V^\top}_{n \times n}

Σ has A's shape, with the singular values down its diagonal.

Singular values and rank

\sigma_1 \ge \sigma_2 \ge \cdots \ge 0, \qquad \operatorname{rank} A = \#\{\, i : \sigma_i > 0 \,\}

Zero singular values are directions A squashes flat.

The best rank-k copy

A_k = \sum_{i=1}^{k} \sigma_i\,\mathbf{u}_i\mathbf{v}_i^\top

Each term is a rank-1 matrix; keep the k biggest.

LoRA

W' = W + BA, \qquad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times d},\ r \ll d

Train 2dr numbers instead of d².

4. The code (python)

mm3_svd.py

import numpy as np

rng = np.random.default_rng(0)
# a 6 x 5 matrix that is almost rank 2: two strong directions plus a little noise
M = rng.normal(size=(6, 2)) @ rng.normal(size=(2, 5)) + 0.01 * rng.normal(size=(6, 5))
U, s, Vt = np.linalg.svd(M)
print(U.shape, s.shape, Vt.shape)        # (6, 6) (5,) (5, 5)
print(s.round(3))                        # [4.979 2.174 0.023 0.016 0.005]
M2 = (U[:, :2] * s[:2]) @ Vt[:2]         # keep the two largest singular values
print(round(np.linalg.norm(M - M2) / np.linalg.norm(M), 4))   # 0.0053

d, r = 4096, 8                           # LoRA: a full update against a rank-8 one
print(d * d, 2 * d * r)                  # 16777216 65536

5. You’ll see this in

Reference notes: matrix decompositions, SVD and LoRA the same ideas, with proofs
Module 1: linear combinations and span rank, from the other side

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done

Q1/4 A is 6 × 4. What are the shapes of U, Σ and Vᵀ in its full SVD?

Q2/4 A matrix has singular values 5, 2, 0 and 0. What is its rank?

Q3/4 What is the best rank-1 approximation of A?

Q4/4 LoRA with rank r = 8 on a 4096 × 4096 weight trains how many numbers for that matrix?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.

Learn machine learning from scratch