Norms: how big is a vector?

1. The idea

A norm measures the size of a vector with one number, never negative and zero only for the zero vector. The everyday one is straight-line length, the L2 norm: ‖𝐱‖₂ = √(x₁² + x₂² + …). It is Pythagoras, in any number of dimensions, and it equals √(𝐱·𝐱).

It isn't the only sensible choice. The L1 norm adds up absolute values, ‖𝐱‖₁ = |x₁| + |x₂| + …, like walking a city grid instead of flying. The L∞ norm keeps only the biggest entry. The widget draws each one's unit ball, every point whose norm is exactly 1: L2 gives a circle, L1 a diamond, L∞ a square.

Norms are how machine learning measures “too big”. Weight decay adds λ‖𝐰‖₂² to the loss to keep weights small. An L1 penalty λ‖𝐰‖₁ pushes many weights to exactly zero, because the diamond's corners sit on the axes, where some coordinates are 0. Gradient clipping shrinks a gradient whose norm is too large, and dividing an embedding by its L2 norm keeps only its direction.

2. Try it: three ways to measure

Pick a norm to draw its unit ball, then drag 𝐱. Put 𝐱 on the L1 diamond and switch to L2: is it inside the circle, on it or outside? Can a point on the diamond ever be outside the circle?

3. The math

4. The code (python)

import numpy as np

x = np.array([3.0, -4.0])
print(np.linalg.norm(x))           # 5.0
print(np.linalg.norm(x, 1))        # 7.0
print(np.linalg.norm(x, np.inf))   # 4.0
print(x / np.linalg.norm(x))       # [ 0.6 -0.8]

g = np.array([6.0, 8.0])           # clip a gradient to a norm of at most 5
max_norm = 5.0
n = np.linalg.norm(g)
if n > max_norm:
    g = g * (max_norm / n)
print(g, np.linalg.norm(g))        # [3. 4.] 5.0

5. You’ll see this in

• Core Course chapter 11: overfitting and regularization the L2 penalty at work
• Ridge regression in the catalog
• Reference notes: linear algebra norms of matrices too

6. Check yourself