Matrix Decompositions, SVD & Low-Rank Adaptation (LoRA)

💡 1. Core Intuition & Concepts

Weight matrices in large language models possess high nominal dimensionality but exhibit low intrinsic rank during task adaptation. Singular Value Decomposition (SVD) and the Eckart-Young-Mirsky theorem prove that weight updates ΔW can be factored into two low-rank matrices B * A with r << d, slashing trainable parameter counts by 99% with zero loss in representational capacity.

📐 2. Mathematical Formulations & Derivations

Singular Value Decomposition (SVD)

A = U \Sigma V^T = \sum_{i=1}^r \sigma_i u_i v_i^T \qquad U \in O(m), \; V \in O(n), \; \sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r > 0

U: left singular vectors (eigenvectors of AA^T), V: right singular vectors (eigenvectors of A^T A).

Eckart-Young-Mirsky Low-Rank Approximation Theorem

\min_{\text{rank}(B)=k} \|A - B\|_F = \|A - A_k\|_F = \sqrt{\sum_{i=k+1}^{\min(m,n)} \sigma_i^2} \qquad A_k = \sum_{i=1}^k \sigma_i u_i v_i^T

Truncated SVD provides the optimal rank-k matrix approximation under both Frobenius and Spectral norms.

LoRA Low-Rank Parameterized Forward Pass

h = W_0 x + \Delta W x = W_0 x + \frac{\alpha}{r} (B \cdot A) x \qquad B \in \mathbb{R}^{d \times r}, \; A \in \mathbb{R}^{r \times k}, \; r \ll \min(d, k)

Pretrained weight W_0 is frozen; only low-rank matrices B (init 0) and A (init Gaussian) are updated.

Intrinsic Dimension & Subspace Gradient Projection

\nabla_A \mathcal{L} = \frac{\alpha}{r} B^T (\nabla_h \mathcal{L}) x^T \qquad \nabla_B \mathcal{L} = \frac{\alpha}{r} (\nabla_h \mathcal{L}) (A x)^T

Gradients backpropagate into low-rank r-dimensional subspace without storing full d×k optimizer states.

• Let A = U Sigma V^T be the full SVD of A in R^(m x n). Define A_k = sum_{i=1}^k sigma_i u_i v_i^T.

• The approximation error is A - A_k = sum_{i=k+1}^r sigma_i u_i v_i^T.

• Compute the squared Frobenius norm of the residual error: ||A - A_k||_F^2 = Tr((A - A_k)^T (A - A_k)) = sum_{i=k+1}^r sigma_i^2.

• By the Courant-Fischer min-max theorem for singular values, any arbitrary matrix B with rank(B) <= k satisfies ||A - B||_2 >= sigma_{k+1}.

• Since ||A - A_k||_2 = sigma_{k+1}, A_k attains the theoretical lower bound for both spectral and Frobenius norms. Q.E.D.

⚙️ 3. Step-by-Step Computational Mechanism

1

Orthogonal Coordinate Rotations

V^T rotates input vectors to principal coordinate axes; Sigma stretches along singular values; U rotates to output space.

2

Energy Concentration

In natural language and vision embeddings, 95%+ of matrix variance (Frobenius energy) is concentrated in the top k << d singular values.

3

Zero Initialization of B

Initializing matrix B = 0 guarantees ΔW = B A = 0 at training step 0, preserving exact pretrained model outputs.

💻 4. Code from Scratch (python)

math_svd_lora.py

import numpy as np

# SVD Compression and LoRA Forward/Backward from Scratch
class LoRALayerScratch:
    def __init__(self, d_in: int, d_out: int, rank: int = 4, alpha: float = 8.0):
        self.d_in = d_in
        self.d_out = d_out
        self.rank = rank
        self.scaling = alpha / rank
        
        # Frozen pretrained weight
        self.W0 = np.random.randn(d_out, d_in) * 0.02
        # LoRA adapters: A is Gaussian, B is exact zero
        self.A = np.random.randn(rank, d_in) * (1.0 / np.sqrt(d_in))
        self.B = np.zeros((d_out, rank))

    def forward(self, x: np.ndarray) -> np.ndarray:
        # h = W0 x + (alpha/r) * B (A x)
        self.x = x
        self.Ax = np.dot(x, self.A.T)  # (batch, rank)
        base_out = np.dot(x, self.W0.T)
        lora_out = np.dot(self.Ax, self.B.T) * self.scaling
        return base_out + lora_out

if __name__ == "__main__":
    layer = LoRALayerScratch(d_in=1024, d_out=1024, rank=8)
    x = np.random.randn(2, 1024)
    out = layer.forward(x)
    print(f"Pretrained params: {1024*1024:,} | LoRA params: {1024*8 + 8*1024:,} (98.4% reduction)")
    print(f"Forward output shape: {out.shape}")

🧠 5. Comprehension Checkpoint

Answer all 1 questions correctly to complete the chapter · 0 / 1 done

Q1/1 Why is matrix B initialized to all zeros while matrix A is initialized with random Gaussian weights in LoRA?

Learn ML from Scratch: Beginner to Advanced