Learn machine learning from scratch

Start at chapter 1 and build up: what a model is, how it measures its own mistakes, and how it learns. Everything is worked out in plain Python, with the maths shown rather than assumed. Two shorter tracks hold reference notes on the maths and on modern architectures.

Track Progress: 0 Completed
0%
Maths for MLLEVEL 2 · INTERMEDIATEModule 4 · Calculus for ML

The chain rule, run backwards: backpropagation

Functions inside functions differentiate by multiplying the local slopes. Do it from the loss backwards on a computation graph and you have backpropagation.

10 min Prerequisites: Lesson 19 (partial derivatives), Module 0 lesson 5 (composition)
Symbols in this lesson∂ partial derivative∘ compositiond/dx derivative

1. The idea

Networks are functions inside functions: the loss of the activation of a weighted sum. The chain rule says how to differentiate a chain like that: multiply the local derivatives along the way. If z = g(w) and L = f(z), then dL/dw = (dL/dz) · (dz/dw).

Backpropagation is the chain rule run efficiently on a computation graph. The forward pass computes every intermediate value. The backward pass starts at the loss with dL/dL = 1 and walks back, multiplying by each node's local derivative. Adding passes the gradient through unchanged; multiplying sends each input the other input's value; squaring sends back 2z.

Each node only needs its own small rule, and every intermediate gradient is computed once and reused by everything upstream of it. That is why a backward pass costs about as much as a forward pass, however many weights there are.

2. Try it: step through a backward pass

Set x, w and b, then press Next step: three forward steps fill in the values, six backward steps fill in the gradients in red, from L back to the inputs.

A computation graph: x and w feed a multiply node, its result and b feed an add node giving z, and z feeds a square node giving L. A Next step button fills in each node's value going forward, then each node's gradient going backward, with a sentence explaining every step.

3. The math

The chain rule
dLdw=dLdz dzdw\frac{dL}{dw} = \frac{dL}{dz}\,\frac{dz}{dw}
Multiply the slopes along the path from w to L.
Longer chains
dLdx=dLdz dzdm dmdx\frac{dL}{dx} = \frac{dL}{dz}\,\frac{dz}{dm}\,\frac{dm}{dx}
One factor per link.
The widget's graph
m=w x,z=m+b,L=z2m = w\,x, \qquad z = m + b, \qquad L = z^2
A one-weight model with a squared loss.
Its gradients
∂L∂z=2z,∂L∂w=2z x,∂L∂b=2z,∂L∂x=2z w\frac{\partial L}{\partial z} = 2z, \quad \frac{\partial L}{\partial w} = 2z\,x, \quad \frac{\partial L}{\partial b} = 2z, \quad \frac{\partial L}{\partial x} = 2z\,w
Each is the gradient arriving at a node times that node's local rule.
One you'll meet often
dσdz=σ(z) (1−σ(z))\frac{d\sigma}{dz} = \sigma(z)\,\big(1 - \sigma(z)\big)
The sigmoid's local derivative, needs only its own output.

4. The code (python)

mm4_chain.py
x, w, b = 2.0, -1.5, 1.0

m = w * x                 # forward pass
z = m + b
L = z ** 2
print(m, z, L)            # -3.0 -2.0 4.0

dL_dz = 2 * z             # backward pass: each node multiplies by its local derivative
dL_dm = dL_dz * 1.0
dL_db = dL_dz * 1.0
dL_dw = dL_dm * x
dL_dx = dL_dm * w
print(dL_dz, dL_dw, dL_db, dL_dx)   # -4.0 -8.0 -4.0 6.0

eps = 1e-6                # check dL/dw by nudging w
print(round((((w + eps) * x + b) ** 2 - L) / eps, 3))   # -8.0

5. You’ll see this in

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done
Q1/4 L = (3w)². What is dL/dw at w = 1?
Q2/4 In a backward pass, what does an addition node z = a + b send to a and b?
Q3/4 Why is backpropagation cheap compared with nudging every weight separately?
Q4/4 For a product m = w·x, what is ∂m/∂w?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.