Capstone: logistic regression from scratch

1. The idea

The second capstone builds a classifier from the parts. Two classes of points lie in the plane. The model scores each point with z = 𝐰ᵀ𝐱 + b (a dot product, Module 1), squashes the score into a probability p = σ(z) with the sigmoid, and learns by minimizing cross-entropy, the negative log-likelihood of a Bernoulli (Module 5), with gradient descent (Module 6).

The gradient comes out remarkably clean. By the chain rule (Module 4), the derivative of the cross-entropy with respect to z is just p − y, the predicted probability minus the label. So ∇𝐰L = (1/n) Σ (pᵢ − yᵢ) 𝐱ᵢ: the same shape as linear regression's gradient, with p where ŷ was.

The decision boundary, where p = 0.5, is the line 𝐰ᵀ𝐱 + b = 0. The loss is convex, so training finds the best line from any start. A logistic regression is a one-neuron network; add hidden layers and you have the networks of the Core Course.

2. Try it: train a boundary

Circles are class 1 and squares class 0. The line is where the model says 50/50. Train and watch it turn and settle while the loss falls.

3. The math

4. The code (python)

import numpy as np

rng = np.random.default_rng(3)
n = 200
y = rng.integers(0, 2, size=n)                                # labels, 0 or 1
centres = np.where(y[:, None] == 1, [1.2, 0.9], [-1.1, -0.8])
X = centres + rng.normal(size=(n, 2))                        # two overlapping clouds

w, b = np.zeros(2), 0.0
for step in range(2000):
    p = 1 / (1 + np.exp(-(X @ w + b)))
    w -= 0.5 * X.T @ (p - y) / n
    b -= 0.5 * np.mean(p - y)

p = 1 / (1 + np.exp(-(X @ w + b)))
loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
print(w.round(3), round(b, 3))                               # [2.49  2.049] -0.644
print(round(loss, 4), np.mean((p > 0.5) == y))               # 0.1662 0.915

5. You’ll see this in

• Logistic regression in the catalog
• Core Course chapter 9: softmax the same model with more than two classes
• Cross-entropy, read with the seven questions

6. Check yourself