Learn machine learning from scratch

Start at chapter 1 and build up: what a model is, how it measures its own mistakes, and how it learns. Everything is worked out in plain Python, with the maths shown rather than assumed. Two shorter tracks hold reference notes on the maths and on modern architectures.

Track Progress: 0 Completed
0%
Maths for MLLEVEL 2 · INTERMEDIATEModule 7 · Putting it together

Capstone: linear regression three ways

One line, fitted exactly with the normal equation, approximately with gradient descent, and as a maximum-likelihood problem. All three land in the same place, and that is the lesson.

11 min Prerequisites: Modules 2, 4, 5 and 6
Symbols in this lessonᵀ transposeA⁻¹ matrix inverse∇ nabla / gradient𝒩 normal (Gaussian) distributionargmin arg min / arg max

1. The idea

This capstone fits one line three ways, using nearly every module so far. The data are ten points, the model is ŷ = wx + b, and the loss is the mean squared error.

First, exactly. Put a column of ones next to x to make the design matrix X, and the best parameters solve the normal equation XᵀXθ = Xᵀ𝐲: matrices from Module 2, and setting the gradient to zero from Module 4. Second, by gradient descent: start anywhere and walk downhill (Module 6). Third, by maximum likelihood: assume each y is wx + b plus Gaussian noise, and pick the most likely w and b (Module 5). The first gets the answer in one solve, the second creeps up on it, and the third turns out to be the same problem as the first.

That agreement is the point. Squared error isn't an arbitrary choice: it is exactly what you minimize when you believe the noise is Gaussian. And the closed form that works here doesn't exist for neural networks, which is why everything at scale uses the second method.

2. Try it: watch descent catch up with the exact answer

The dashed line is the normal equation's exact answer. Gradient descent starts from a flat line at 0; run it and compare the numbers.

Ten data points, the exact least-squares line drawn dashed, and a solid line for gradient descent's current parameters, starting flat at zero. Buttons run 10 or 100 steps; the text compares the w and b from each method.

3. The math

The design matrix
X=(x11⋮⋮xn1),θ=(wb),y^=XθX = \begin{pmatrix} x_1 & 1 \\ \vdots & \vdots \\ x_n & 1 \end{pmatrix}, \qquad \theta = \begin{pmatrix} w \\ b \end{pmatrix}, \qquad \hat{\mathbf{y}} = X\theta
The column of ones carries the bias.
The loss and its gradient
L(θ)=1n∥y−Xθ∥22,∇L=−2n X⊤(y−Xθ)L(\theta) = \frac{1}{n}\|\mathbf{y} - X\theta\|_2^2, \qquad \nabla L = -\frac{2}{n}\,X^\top(\mathbf{y} - X\theta)
A norm (Module 1), a transpose (Module 2), a gradient (Module 4).
1. The normal equation
∇L=0  ⇒  X⊤X θ∗=X⊤y\nabla L = \mathbf{0} \;\Rightarrow\; X^\top X\,\theta^{*} = X^\top\mathbf{y}
Solve with np.linalg.solve, not by inverting XᵀX.
2. Gradient descent
θ←θ−η ∇L(θ)\theta \leftarrow \theta - \eta\,\nabla L(\theta)
Slower, but it works for any differentiable model.
3. Maximum likelihood
yi=wxi+b+εi,  εi∼N(0,σ2)  ⇒  arg⁡max⁡θlog⁡L=arg⁡min⁡θ∑i(yi−y^i)2y_i = w x_i + b + \varepsilon_i,\ \ \varepsilon_i \sim \mathcal{N}(0, \sigma^2) \;\Rightarrow\; \arg\max_\theta \log L = \arg\min_\theta \sum_i (y_i - \hat{y}_i)^2
Lesson 26: −log of a Gaussian is a squared error plus a constant.

4. The code (python)

mm7_linreg.py
import numpy as np

x = np.array([0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0])
y = np.array([1.9, 2.3, 3.4, 3.6, 4.4, 5.1, 5.3, 6.2, 6.8, 7.1])
X = np.column_stack([x, np.ones_like(x)])          # one row per point: [x, 1]

# 1. exactly: the normal equation
theta_exact = np.linalg.solve(X.T @ X, X.T @ y)
print(theta_exact.round(4))                        # [1.1891 1.34  ]

# 2. gradient descent
theta = np.zeros(2)
for _ in range(5000):
    grad = -2 / len(x) * X.T @ (y - X @ theta)
    theta = theta - 0.05 * grad
print(theta.round(4))                              # [1.1891 1.34  ]

# 3. maximum likelihood with Gaussian noise (sigma = 0.3), searched on a grid of (w, b)
W, B = np.meshgrid(np.linspace(1.0, 1.4, 401), np.linspace(1.0, 1.7, 701))
R = y - (W[..., None] * x + B[..., None])
log_lik = np.sum(-0.5 * (R / 0.3) ** 2 - np.log(0.3 * np.sqrt(2 * np.pi)), axis=-1)
i = np.unravel_index(log_lik.argmax(), log_lik.shape)
print(round(W[i], 3), round(B[i], 3))              # 1.189 1.34

5. You’ll see this in

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done
Q1/4 Where does the normal equation XᵀXθ = Xᵀ𝐲 come from?
Q2/4 Why do neural networks train by gradient descent instead of a closed form?
Q3/4 Maximum likelihood with Gaussian noise gives the same line as…
Q4/4 What does the column of ones in X do?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.