Maths for ML / Notation

\nabla_{\theta} L

nabla / gradient

CalculusBeginner

Say it “grad L / del L with respect to theta”

The vector of all partial derivatives. It points in the direction the function increases fastest.

Example

\nabla f(x,y) = \left[\tfrac{\partial f}{\partial x}, \tfrac{\partial f}{\partial y}\right]

In code

loss.backward(); [p.grad for p in model.parameters()]

Where you’ll see it in ML

Gradient descent: θ ← θ − η∇L. The whole of training.

Don’t confuse it with

$\frac{\partial f}{\partial x}$ partial derivative Slope with respect to one input while holding all the others fixed.
$\Delta x$ Delta (change) A (finite) change in a quantity. Lowercase δ often means a small change or an error term.
$\mathbf{J} = \frac{\partial \mathbf{f}}{\partial \mathbf{x}}$ Jacobian For a function with vector input and vector output: the matrix of every output's partial derivative with respect to every input.
$\eta$ eta (learning rate) Conventionally the learning rate — how big a step to take along the gradient.

Lessons that use it

← All 94 symbols