Jacobians and Hessians: derivatives as tables

A function with many outputs has a whole table of derivatives, the Jacobian. A function's second derivatives form another, the Hessian, which describes curvature.

10 min Prerequisites: Lessons 19 and 20 (gradient, chain rule)

Symbols in this lessonJ Jacobian H Hessian ∂ partial derivative X capital letter (matrix)

1. The idea

The gradient is the derivative of a function with one output. A layer has many outputs, so its derivative is a table: the Jacobian J, with one row per output and one column per input, Jᵢⱼ = ∂fᵢ/∂xⱼ. For a linear layer f(𝐱) = W𝐱 the Jacobian is W itself. Backpropagation through a stack of layers multiplies Jacobians, which is the chain rule written with matrices.

Second derivatives describe curvature. For a function with one output and n inputs, the Hessian H is the n × n table of every ∂²f/∂xᵢ∂xⱼ, and it is symmetric. Its eigenvalues say how curved the function is in each direction: all positive at a minimum, mixed signs at a saddle point.

In deep learning the Hessian is mostly a thinking tool: with a million parameters it has a trillion entries, far too many to store. Second-order optimizers approximate it, and “sharp” and “flat” minima are statements about its eigenvalues.

2. Try it: what shape is the derivative?

Set how many numbers go in and come out, or pick an example. The grids are the shapes of the derivatives.

3. The math

The Jacobian

J = \begin{pmatrix} \frac{\partial f_1}{\partial x_1} & \cdots & \frac{\partial f_1}{\partial x_n} \\ \vdots & & \vdots \\ \frac{\partial f_m}{\partial x_1} & \cdots & \frac{\partial f_m}{\partial x_n} \end{pmatrix} \in \mathbb{R}^{m \times n}

Row i is the gradient of output i.

A linear layer's Jacobian

f(\mathbf{x}) = W\mathbf{x} \;\Rightarrow\; J = W

Output i is row i of W dotted with 𝐱, so ∂fᵢ/∂xⱼ = Wᵢⱼ.

The chain rule with matrices

J_{f \circ g}(\mathbf{x}) = J_f\big(g(\mathbf{x})\big)\, J_g(\mathbf{x})

Shapes: (m × k)(k × n) = (m × n), the matrix-product rule from Module 2.

The Hessian

H_{ij} = \frac{\partial^2 f}{\partial x_i\,\partial x_j}, \qquad H = H^\top

Differentiate twice, in either order: the result is the same.

An example

f(x, y) = x^2 + xy + 2y^2 \;\Rightarrow\; H = \begin{pmatrix} 2 & 1 \\ 1 & 4 \end{pmatrix}

Constant here, because f is quadratic. Both eigenvalues are positive: a bowl.

4. The code (python)

mm4_jacobian.py

import numpy as np

W = np.array([[1.0, -2.0, 0.5],
              [0.0,  3.0, 1.0]])         # a layer: 3 inputs, 2 outputs

def layer(x):
    return W @ x

x = np.array([1.0, 2.0, -1.0])
h = 1e-6
J = np.column_stack([(layer(x + h * e) - layer(x)) / h for e in np.eye(3)])
print(J.shape, np.allclose(J, W))       # (2, 3) True

H = np.array([[2.0, 1.0],
              [1.0, 4.0]])              # the Hessian of x^2 + xy + 2y^2
print(np.linalg.eigvalsh(H).round(3))   # [1.586 4.414]
print(1_000_000 ** 2)                   # 1000000000000

5. You’ll see this in

Reference notes: linear algebra and matrix calculus Jacobians and Hessians, more formally
Reference notes: convexity, optimization and duality
Core Course chapter 10: optimizers methods that adapt to curvature without a Hessian

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done

Q1/4 f: ℝ⁴ → ℝ³. What shape is its Jacobian?

Q2/4 What is the Jacobian of f(𝐱) = W𝐱?

Q3/4 At a strict minimum, what are the Hessian's eigenvalues like?

Q4/4 Why don't deep learning libraries compute the full Hessian?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.

Learn machine learning from scratch