Learn machine learning from scratch

Start at chapter 1 and build up: what a model is, how it measures its own mistakes, and how it learns. Everything is worked out in plain Python, with the maths shown rather than assumed. Two shorter tracks hold reference notes on the maths and on modern architectures.

Track Progress: 0 Completed
0%
Maths for MLLEVEL 1 · BEGINNERModule 6 · Optimization

The loss as a landscape

Fix the data and the loss becomes a surface over the parameters. Every point on it is a whole model; training walks downhill across it.

9 min Prerequisites: Module 4 (gradients)
Symbols in this lessonargmin arg min / arg max∇ nabla / gradientΣ summation

1. The idea

Fix the data, and the loss becomes a function of the parameters alone. With two parameters, the w and b of a line, you can draw it: every point (w, b) is a whole candidate line, and the height there is how badly that line fits. Training is a walk across this landscape looking for low ground.

For a linear model with squared error the landscape is a bowl: one lowest point, no traps, with elliptical contour lines. The ellipses are stretched when the parameters affect the loss very differently, which turns the bowl into a long, narrow valley, the shape that makes plain gradient descent zig-zag.

Real networks have millions of parameters, so nobody sees their landscape. The two-parameter picture still gives the right instincts: the gradient points across the contours, the learning rate decides how far each step jumps, and the shape of the valley decides how hard the walk is.

2. Try it: a model is a point

Drag the point on the right (or click anywhere there) and watch its line on the left. Find the green lowest point, and notice which direction the bowl is narrow in.

Left: ten data points and a line. Right: contour lines of the mean squared error over w (across) and b (up), with a draggable point and the lowest point marked in green. Moving the point changes the line; the text gives its loss and the best w and b.

3. The math

The loss over the parameters
L(w,b)=1n∑i=1n(yi−(wxi+b))2L(w, b) = \frac{1}{n}\sum_{i=1}^{n}\big(y_i - (w x_i + b)\big)^2
The data are fixed; w and b are the coordinates of the landscape.
The lowest point
(w∗,b∗)=arg⁡min⁡w, b L(w,b)(w^{*}, b^{*}) = \arg\min_{w,\,b}\ L(w, b)
The best line.
The bottom is flat
∇L(w∗,b∗)=0\nabla L(w^{*}, b^{*}) = \mathbf{0}
Both partial derivatives are zero there.
Why it's a bowl
L(w,b) is quadratic in (w,b)  ⇒  one minimum, elliptical contoursL(w, b) \text{ is quadratic in } (w, b) \;\Rightarrow\; \text{one minimum, elliptical contours}
Neural network losses are not quadratic, which is lesson 29.

4. The code (python)

mm6_landscape.py
import numpy as np

x = np.array([0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0])
y = np.array([1.9, 2.3, 3.4, 3.6, 4.4, 5.1, 5.3, 6.2, 6.8, 7.1])

W, B = np.meshgrid(np.linspace(-0.5, 2.5, 301), np.linspace(-2, 6, 401))
L = np.mean((y - (W[..., None] * x + B[..., None])) ** 2, axis=-1)   # the whole landscape at once
i, j = np.unravel_index(L.argmin(), L.shape)
print(L.shape)                                                       # (401, 301)
print(round(W[i, j], 2), round(B[i, j], 2), round(L[i, j], 4))      # 1.19 1.34 0.0287

5. You’ll see this in

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done
Q1/4 In the (w, b) landscape, what does a single point stand for?
Q2/4 What shape is the squared-error landscape of a linear model?
Q3/4 What is the gradient at the lowest point?
Q4/4 What does a long, narrow valley do to plain gradient descent?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.