Common distributions: Bernoulli, categorical, Gaussian

1. The idea

A distribution says how likely each value of a random variable is. A handful of them do almost all the work in machine learning.

The Bernoulli distribution is a single yes-or-no outcome with P(1) = p: a coin flip, a click, a binary label. The categorical distribution extends it to K outcomes whose probabilities add up to 1: the next token, the class of an image. A classifier's sigmoid output is a Bernoulli p; its softmax output is a categorical distribution.

The Gaussian or normal distribution 𝒩(μ, σ²) is the bell curve, centred on μ with spread σ. It is a density: p(x) is a height, not a probability, and probabilities are areas under the curve, so p(x) can be above 1 when σ is small. Gaussians describe measurement noise, the starting weights of a network and the noise that a diffusion model adds and learns to remove.

2. Try it: three shapes of chance

Switch between the three. For the Gaussian, shrink σ until the peak passes 1, which a probability never could; the shaded band is μ ± σ.

3. The math

4. The code (python)

import numpy as np
from math import erf, sqrt

rng = np.random.default_rng(0)
print(round((rng.random(100_000) < 0.7).mean(), 3))                    # 0.701
counts = np.bincount(rng.choice(4, size=100_000, p=[0.4, 0.3, 0.2, 0.1]), minlength=4)
print((counts / counts.sum()).round(2).tolist())                       # [0.4, 0.3, 0.2, 0.1]

x = rng.normal(loc=2.0, scale=0.5, size=100_000)                       # x ~ N(2, 0.5^2)
print(round(x.mean(), 2), round(x.std(), 2))                           # 2.0 0.5
print(round(np.mean(np.abs(x - 2.0) <= 0.5), 3), round(erf(1 / sqrt(2)), 3))   # 0.683 0.683

5. You’ll see this in

• The sigmoid, read with the seven questions a Bernoulli p
• Softmax, read with the seven questions a categorical distribution
• Denoising diffusion models in the catalog Gaussian noise added and removed

6. Check yourself