Special matrices: diagonal, symmetric, orthogonal, triangular, sparse

Matrices with a pattern are cheaper to store, faster to multiply or easy to invert, and ML code uses every one of these patterns.

10 min Prerequisites: Lesson 12 (transpose and inverse)

Symbols in this lessondiag diagonal matrix ᵀ transpose I identity matrix ⊙ Hadamard (element-wise) product

1. The idea

Some matrices have a pattern, and the pattern makes them cheap to store, fast to multiply or easy to invert. Machine learning code leans on all of these.

A diagonal matrix only scales each coordinate: multiplying by it is the same as multiplying element-wise by its diagonal, which is how the per-feature scale in a normalization layer works. A symmetric matrix equals its own transpose; covariance matrices and Hessians are always symmetric. An orthogonal matrix has columns of length 1 at right angles to each other, so it rotates or flips without stretching anything, and its inverse is simply its transpose.

A triangular matrix is zero on one side of the diagonal. The causal mask in a language model is lower triangular: token i may look at tokens 1 to i but never at later ones. A sparse matrix is mostly zeros and is stored as a list of its non-zero entries; graphs and one-hot encodings over huge vocabularies are full of them.

2. Try it: spot the pattern

Pick a kind of matrix. The diagonal is highlighted and zeros are faded, and the checks below are worked out from the numbers, not typed in.

3. The math

Diagonal

D = \operatorname{diag}(d_1, \dots, d_n) \qquad D\mathbf{x} = \mathbf{d} \odot \mathbf{x}

⊙ is the element-wise product: no full matrix multiply needed.

Symmetric

A = A^\top

Entry (i, j) equals entry (j, i).

Orthogonal

Q^\top Q = I \qquad Q^{-1} = Q^\top \qquad \|Q\mathbf{x}\|_2 = \|\mathbf{x}\|_2

Lengths (and angles) are preserved.

Lower triangular

L_{ij} = 0 \ \text{ for } j > i

Zeros above the diagonal.

A causal mask

M_{ij} = \begin{cases} 0 & j \le i \\ -\infty & j > i \end{cases}

Added to attention scores: −∞ becomes a weight of 0 after softmax, hiding the future.

4. The code (python)

mm2_special.py

import numpy as np

d = np.array([2.0, 0.5, -1.0])
x = np.array([1.0, 4.0, 3.0])
print(np.diag(d) @ x, d * x)                  # [ 2.  2. -3.] [ 2.  2. -3.]

t = np.radians(30)
Q = np.array([[np.cos(t), -np.sin(t)],
              [np.sin(t),  np.cos(t)]])
print(np.allclose(Q.T @ Q, np.eye(2)))        # True
print(round(np.linalg.norm(Q @ np.array([3.0, 4.0])), 6))   # 5.0

n = 4                                         # a causal mask for 4 tokens
mask = np.triu(np.full((n, n), -np.inf), k=1)
print(mask[1].tolist())                       # [0.0, 0.0, -inf, -inf]

5. You’ll see this in

Core Course chapter 17: a tiny attention-based language model the causal mask in use
Normalization layers in the catalog a diagonal scale per feature
Principal component analysis in the catalog a symmetric covariance matrix

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done

Q1/4 Where does diag(2, 3) send the point (1, 1)?

Q2/4 What is the inverse of an orthogonal matrix Q?

Q3/4 Which of these is always symmetric?

Q4/4 In a causal mask, what do the −∞ entries above the diagonal do?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.

Learn machine learning from scratch