Learning rates and momentum

One step size has to suit steep and gentle directions at once. Momentum builds speed where the gradients agree and cancels the bouncing where they don't.

10 min Prerequisites: Lesson 27 (the loss landscape)

Symbols in this lessonη eta (learning rate)∇ nabla / gradient θ theta (parameters)

1. The idea

Gradient descent repeats one move: step against the gradient, θ ← θ − η∇L. The learning rate η is the step size, and it is a balance. Too small and progress is slow. Too large and every step overshoots the valley floor, lands higher on the far side, and the loss blows up.

In a long, narrow valley one η has to serve two very different slopes: steep across the valley, gentle along it. A step that is safe across is tiny along. Momentum fixes much of this with a running velocity: 𝐯 ← β𝐯 + ∇L, then θ ← θ − η𝐯. Along the valley the gradients agree from step to step and the velocity builds up; across it they flip sign every step and mostly cancel.

Adam, the default optimizer for most networks, combines momentum with a separate step size for every parameter, scaled by how large that parameter's recent gradients have been. Core Course chapter 10 builds it.

2. Try it: race two optimizers

Both start at the same point in a narrow valley. Green is plain gradient descent, blue adds momentum. Change β, then push η towards 0.1 and see which one breaks first.

3. The math

Plain gradient descent

\theta \leftarrow \theta - \eta\,\nabla L(\theta)

The same step rule as Read an equation's last page.

Momentum

\mathbf{v} \leftarrow \beta\,\mathbf{v} + \nabla L(\theta), \qquad \theta \leftarrow \theta - \eta\,\mathbf{v}

β near 1 remembers many past gradients.

How big a step is safe

0 < \eta < \frac{2}{\lambda_{\max}}

On a quadratic bowl; λ_max is the largest eigenvalue of the Hessian (Module 4).

The widget's valley

L(\theta) = \theta_1^2 + 10\,\theta_2^2 \;\Rightarrow\; H = \operatorname{diag}(2, 20),\ \eta < 0.1

Ten times steeper across than along.

4. The code (python)

mm6_momentum.py

import numpy as np

def grad(t):                          # L = t1^2 + 10 t2^2
    return np.array([2 * t[0], 20 * t[1]])

def final_loss(eta, beta, steps=40):
    t, v = np.array([-3.6, 1.6]), np.zeros(2)
    for _ in range(steps):
        v = beta * v + grad(t)
        t = t - eta * v
    return t[0] ** 2 + 10 * t[1] ** 2

print(f"{final_loss(0.02, 0.0):.4f} {final_loss(0.02, 0.8):.4f}")   # 0.4947 0.0022
print(f"{final_loss(0.105, 0.0):.1f}")                              # 52439.0

5. You’ll see this in

Core Course chapter 10: optimizers momentum, RMSProp and Adam, built
Gradient descent, read with the seven questions
Reference notes: convexity, optimization and duality

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done

Q1/4 What happens when the learning rate is too large?

Q2/4 What does momentum keep track of?

Q3/4 In a narrow valley, what happens to the across-the-valley parts of the gradient under momentum?

Q4/4 For L(θ) = θ², whose Hessian is 2, what is the largest stable learning rate?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.

Learn machine learning from scratch