Maths for ML / Read an equation

Gradient descent

The update rule behind nearly all training: find which way is uphill, and step the other way.

θ  ←  θ−η ∇L(θ)\htmlClass{mx-p mx-p-new}{\theta} \;\htmlClass{mx-p mx-p-arrow}{\leftarrow}\; \htmlClass{mx-p mx-p-old}{\theta} \htmlClass{mx-p mx-p-minus}{-} \htmlClass{mx-p mx-p-eta}{\eta}\, \htmlClass{mx-p mx-p-grad}{\nabla L(\theta)}

The parts

Point at any part of the equation, or pick it from the list.

  • the new parameters · output · shape same shape as the old ones

    θ stands for all the model's learned numbers at once, written as one long vector.

  • assignment · operation

    Not an equation to solve: it means “replace θ with the right-hand side”, like θ = θ - … in code. It runs again and again.

  • the current parameters · learned · shape a vector with one entry per parameter

    Where training is now.

  • go against the gradient · operation

    The gradient points uphill, so subtracting it moves downhill, toward a lower loss.

  • the learning rate · constant · shape one small number

    How far to step. You choose it; training does not learn it.

  • the gradient of the loss · operation · shape same shape as θ

    One partial derivative per parameter: how much the loss changes if you nudge that parameter. Backpropagation computes it.

Read it with the seven questions

  1. What comes out? A new value for θ. Nothing is “equal” here; the arrow is an instruction you repeat.
  2. List the parts. Learned: θ. Chosen: η. Computed: ∇L(θ) from the current θ and a batch of data.
  3. Check the shapes. ∇L(θ) has exactly θ's shape, one entry per parameter, so θ − η∇L(θ) does too.
  4. Say what each function does. ∇ takes the loss (one number) and returns a direction (a vector). Multiplying by η scales the step.
  5. Ask what makes it bigger or smaller. Steep slope or big η: big step. Flat slope: small step. At the bottom of the bowl the gradient is zero and θ stops moving.
  6. Push the inputs to extremes. η = 0: nothing ever changes. η too big: each step jumps past the minimum to a higher point on the other side, and the loss blows up.
  7. Write it as code. theta = theta - eta * grad(theta), in a loop.

Change it and watch

The loss here is L(θ) = θ₁² + 5θ₂², a long, narrow bowl with its lowest point at the origin. Click the plot (or use the start sliders) to pick a starting point, then take steps. Raise η past 0.1 to watch the path zig-zag, and past 0.2 to watch it fly off.

Contour lines of a long, narrow bowl-shaped loss, with its minimum at the centre. A path of gradient descent steps runs from a chosen starting point toward the centre. A slider sets the learning rate: small values give a smooth path, larger ones a zig-zag, and above 0.2 the steps diverge.

Now write it in NumPy
import numpy as np

def grad(theta):                       # L(θ) = θ₁² + 5θ₂²
    return np.array([2 * theta[0], 10 * theta[1]])

theta = np.array([3.0, 1.0])
eta = 0.1
for step in range(50):
    theta = theta - eta * grad(theta)

# θ₂ lands on 0 in the first step; θ₁ shrinks by a factor 0.8 every step
print(theta)                           # [4.28174308e-05 0.00000000e+00]

You’ll see this in

Symbols on this page∇ nabla / gradientη eta (learning rate)θ theta (parameters)∂ partial derivative