Sampling, i.i.d. data and likelihood

Probability asks about data with the parameters fixed; likelihood asks about parameters with the data fixed. Its negative log is where the standard losses come from.

10 min Prerequisites: Lesson 25 (distributions), Module 0 lesson 3 (Σ and Π)

Symbols in this lesson~ is distributed as / sampled from Π product log logarithm argmin arg min / arg max

1. The idea

x ~ p(x) says “x is drawn from the distribution p”. Training data is usually assumed to be i.i.d.: independent draws from one and the same distribution. Independence is what lets the probability of a whole dataset be a product, one factor per example.

Probability and likelihood use the same formula from opposite sides. Probability fixes the parameters and asks about data. Likelihood fixes the data you actually observed and asks about parameters: L(θ) = ∏ p(xᵢ | θ) measures how well each candidate θ explains this dataset. Maximum likelihood picks the θ that explains it best.

A product of thousands of numbers below 1 underflows to 0 in floating point, so everyone maximizes the log-likelihood, a sum, instead. Minimizing the negative log-likelihood is where the standard losses come from: a Gaussian model gives mean squared error, and a categorical model gives cross-entropy.

2. Try it: which μ explains the data?

Eight data points, and a Gaussian with σ = 1 whose centre μ you slide. The red sticks are the curve's height at each point; the likelihood multiplies them. Find where the log-likelihood on the right peaks.

3. The math

The likelihood of i.i.d. data

L(\theta) = \prod_{i=1}^{n} p(x_i \mid \theta)

The data are fixed; θ varies.

The log-likelihood

\log L(\theta) = \sum_{i=1}^{n} \log p(x_i \mid \theta)

A sum: no underflow, and the same maximum.

Maximum likelihood

\theta^{*} = \arg\max_{\theta}\ \sum_{i=1}^{n} \log p(x_i \mid \theta)

For a Gaussian mean, θ* is the sample mean.

A Gaussian model gives squared error

-\log p(y \mid \hat{y}) = \frac{(y - \hat{y})^2}{2\sigma^2} + \text{const}

Minimizing the negative log-likelihood is minimizing MSE.

A categorical model gives cross-entropy

-\log p(y \mid \mathbf{p}) = -\log p_{y}

The loss is −log of the probability given to the true class.

4. The code (python)

mm5_likelihood.py

import numpy as np

x = np.array([1.2, 2.9, 2.1, 3.6, 1.8, 2.4, 3.1, 2.7])

def log_lik(mu, sigma=1.0):
    return np.sum(-0.5 * ((x - mu) / sigma) ** 2 - np.log(sigma * np.sqrt(2 * np.pi)))

mus = np.linspace(0, 5, 2001)                          # try mu = 0.0000, 0.0025, ..., 5.0000
best = mus[np.argmax([log_lik(m) for m in mus])]
print(round(best, 3), round(x.mean(), 3))              # 2.475 2.475
print(round(log_lik(best), 3))                         # -9.409
print(np.prod(np.full(1000, 0.01)), round(1000 * np.log(0.01), 2))   # 0.0 -4605.17

5. You’ll see this in

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done

Q1/4 What does i.i.d. stand for?

Q2/4 In a likelihood, what is held fixed and what varies?

Q3/4 Why maximize the log-likelihood instead of the likelihood?

Q4/4 Minimizing the negative log-likelihood of a Gaussian model gives which loss?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.

Learn machine learning from scratch