Partial derivatives and the gradient

With many inputs, nudge one at a time: that's a partial derivative. Stack them all and you get the gradient, an arrow pointing straight uphill.

10 min Prerequisites: Lesson 18 (the derivative)

Symbols in this lesson∂ partial derivative ∇ nabla / gradient · dot product

1. The idea

Most functions in machine learning take many inputs: a loss depends on every weight. A partial derivative, ∂f/∂x, asks the one-variable question about one input at a time. Hold every other input fixed, nudge this one, and see how fast f changes.

Collect all the partial derivatives into a vector and you have the gradient, ∇f. It points in the direction in which f increases fastest, its length says how steep that is, and it always crosses the contour lines (where f is constant) at right angles. The negative gradient is the steepest way down.

That is why training works. The loss is a function of millions of weights; backpropagation computes ∇L; and every step moves the weights a little way along −∇L.

2. Try it: follow the arrows downhill

Drag the point P across the contour lines. The bright arrow is ∇f at P; the faint ones are the gradient elsewhere. Press the button a few times to walk downhill.

3. The math

A partial derivative

\frac{\partial f}{\partial x}(x, y) = \lim_{h \to 0} \frac{f(x + h, y) - f(x, y)}{h}

y is held fixed; only x moves.

The gradient

\nabla f = \left(\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n}\right)

One entry per input, so ∇f has the same shape as the input.

The widget's function

f(x, y) = x^2 + xy + 2y^2 \;\Rightarrow\; \nabla f = (2x + y,\ x + 4y)

Treat the other variable as a constant when differentiating.

Slope in any direction

D_{\mathbf{u}} f = \nabla f \cdot \mathbf{u}, \qquad \|\mathbf{u}\|_2 = 1

Largest when 𝐮 points along ∇f (the dot product, Module 1).

Steepest descent

-\frac{\nabla f}{\|\nabla f\|_2}

The direction gradient descent steps in.

4. The code (python)

mm4_gradient.py

import numpy as np

def f(p):
    x, y = p
    return x**2 + x * y + 2 * y**2

def grad(p):
    x, y = p
    return np.array([2 * x + y, x + 4 * y])

p = np.array([1.5, 1.0])
print(grad(p))                               # [4.  5.5]

h = 1e-5                                     # check each partial by nudging one input at a time
numeric = [(f(p + h * e) - f(p - h * e)) / (2 * h) for e in np.eye(2)]
print(np.round(numeric, 4))                  # [4.  5.5]

u = np.array([1.0, -1.0]) / np.sqrt(2)       # the slope in a chosen direction
print(round(grad(p) @ u, 4))                 # -1.0607

5. You’ll see this in

Core Course chapter 7: the training loop
Gradient descent, read with the seven questions
Module 1: the dot product why the gradient is the steepest direction

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done

Q1/4 f(x, y) = x²y. What is ∂f/∂x?

Q2/4 f takes 5 inputs and returns one number. How many numbers are in ∇f?

Q3/4 How does the gradient meet the contour line through the same point?

Q4/4 To decrease f as fast as possible, which way do you step?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.

Learn machine learning from scratch