Learn machine learning from scratch

Start at chapter 1 and build up: what a model is, how it measures its own mistakes, and how it learns. Everything is worked out in plain Python, with the maths shown rather than assumed. Two shorter tracks hold reference notes on the maths and on modern architectures.

Track Progress: 0 Completed
0%
Maths for MLLEVEL 1 · BEGINNERModule 4 · Calculus for ML

The derivative: slope, and sensitivity

Shrink the gap between two points on a curve and the line through them becomes the tangent. Its slope is the derivative: how much the output moves when you nudge the input.

10 min Prerequisites: Module 0 (reading notation)
Symbols in this lessond/dx derivativelim limitΔ Delta (change)≈ approximately equal

1. The idea

The derivative of f at x is the slope of f right there: how fast f(x) changes when x changes a little. Take two points on the curve, x and x + h. The slope of the line through them is (f(x + h) − f(x)) / h. Let h shrink towards 0 and that secant line turns into the tangent; its slope is f′(x).

In machine learning the derivative is a sensitivity: nudge this weight up by a tiny amount, and how much does the loss change, and which way? A positive derivative says the loss goes up, so move the weight down. That one fact is the whole of gradient descent.

Nobody computes limits by hand for long. A handful of rules cover almost everything: the power rule, sums, constants that come out, and the chain rule for functions inside functions (lesson 20). The nudge formula is still useful: it is how you check a gradient numerically.

2. Try it: shrink the secant

Pick a point x, then drag h down towards 0. The red secant line swings onto the dashed tangent, and its slope approaches f′(x).

The curve f(x) = x cubed over 3 minus x, a point on it at x, a second point at x + h, the red secant line through both and the dashed tangent at x. Sliders set x and h; the text gives the secant's slope, the derivative x squared minus 1, and the gap between them.

3. The math

The definition
f′(x)=lim⁡h→0f(x+h)−f(x)hf'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h}
The slope of the secant, as the gap h shrinks to nothing.
The power rule
ddx xn=n xn−1\frac{d}{dx}\,x^{n} = n\,x^{n-1}
x³ becomes 3x², x becomes 1, a constant becomes 0.
Sums and constants
ddx(af(x)+b g(x))=af′(x)+b g′(x)\frac{d}{dx}\big(a f(x) + b\,g(x)\big) = a f'(x) + b\,g'(x)
Differentiate term by term.
exp and log
ddx ex=exddxln⁡x=1x\frac{d}{dx}\,e^{x} = e^{x} \qquad \frac{d}{dx}\ln x = \frac{1}{x}
The two that losses are built from.
The widget's curve
f(x)=13x3−x  ⇒  f′(x)=x2−1f(x) = \tfrac{1}{3}x^{3} - x \;\Rightarrow\; f'(x) = x^{2} - 1
Flat (f′ = 0) at x = −1 and x = 1.

4. The code (python)

mm4_derivative.py
def f(x):
    return x**3 / 3 - x

def f_prime(x):
    return x**2 - 1

x = 1.5
print([round((f(x + h) - f(x)) / h, 5) for h in (1.0, 0.1, 0.001)])   # [3.08333, 1.40333, 1.2515]
print(f_prime(x))                                                     # 1.25

h = 0.001       # a central difference is far more accurate for the same h
print(round((f(x + h) - f(x - h)) / (2 * h), 7))                      # 1.2500003

5. You’ll see this in

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done
Q1/4 What is the derivative of x⁴?
Q2/4 f′(2) = −3. What happens to f if you nudge x up a little from 2?
Q3/4 What is d/dx (5x² + 3x)?
Q4/4 At the bottom of a smooth valley, what is the derivative?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.