Learn machine learning from scratch

Start at chapter 1 and build up: what a model is, how it measures its own mistakes, and how it learns. Everything is worked out in plain Python, with the maths shown rather than assumed. Two shorter tracks hold reference notes on the maths and on modern architectures.

Track Progress: 0 Completed
0%
Maths for MLLEVEL 2 · INTERMEDIATEModule 5 · Probability and statistics

Expectation, variance and covariance

The average a random variable settles to, how spread out it is, and whether two of them move together. Their matrix form is the covariance matrix PCA takes apart.

10 min Prerequisites: Lesson 22 (probability), Module 3 lesson 17 (PCA) helps
Symbols in this lesson𝔼 expectationVar varianceΣ covariance matrixσ sigma

1. The idea

The expectation 𝔼[X] is the average value X takes, weighting each value by how likely it is: for a fair die, (1 + 2 + … + 6) / 6 = 3.5. It is what you get by averaging many draws, which is why a batch average estimates it.

The variance, Var(X) = 𝔼[(X − 𝔼[X])²], is the average squared distance from that mean: how spread out X is. Its square root, the standard deviation σ, is in the same units as X. Initialization schemes such as Xavier and He are chosen so that the variance of activations stays about the same from layer to layer.

For two variables, the covariance Cov(X, Y) = 𝔼[(X − 𝔼X)(Y − 𝔼Y)] says whether they move together (positive), in opposite directions (negative) or unrelated (around 0). All the pairwise covariances of a vector make the covariance matrix Σ, the matrix PCA diagonalises. Divide by both standard deviations and you get the correlation ρ, between −1 and 1.

2. Try it: shape a cloud

Set the two spreads and the correlation. The ellipse covers about 86% of the cloud (two standard deviations in 2-D), and the numbers below are measured from these 150 points.

A cloud of 150 points drawn from a two-dimensional Gaussian, with an ellipse showing its shape. Sliders set the standard deviation of each coordinate and their correlation; the text gives the model's covariance matrix and the means, variances and covariance measured from the points.

3. The math

Expectation
E[X]=∑xx P(X=x)\mathbb{E}[X] = \sum_{x} x\,P(X = x)
A sum for discrete X; an integral for continuous X.
Variance
Var⁡(X)=E[(X−E[X])2]=E[X2]−E[X]2\operatorname{Var}(X) = \mathbb{E}\big[(X - \mathbb{E}[X])^2\big] = \mathbb{E}[X^2] - \mathbb{E}[X]^2
σ = √Var(X), the standard deviation.
Covariance
Cov⁡(X,Y)=E[(X−EX)(Y−EY)]\operatorname{Cov}(X, Y) = \mathbb{E}\big[(X - \mathbb{E}X)(Y - \mathbb{E}Y)\big]
Cov(X, X) = Var(X).
Correlation
ρ=Cov⁡(X,Y)σX σY∈[−1,1]\rho = \frac{\operatorname{Cov}(X, Y)}{\sigma_X\,\sigma_Y} \in [-1, 1]
Covariance with the units divided out.
The covariance matrix in 2-D
Σ=(σ12ρ σ1σ2ρ σ1σ2σ22)\Sigma = \begin{pmatrix} \sigma_1^2 & \rho\,\sigma_1\sigma_2 \\ \rho\,\sigma_1\sigma_2 & \sigma_2^2 \end{pmatrix}
Symmetric, as Module 2 promised.

4. The code (python)

mm5_expectation.py
import numpy as np

faces = np.arange(1, 7)
print(faces.mean(), round(faces.var(), 4))      # 3.5 2.9167

rng = np.random.default_rng(0)
rolls = rng.integers(1, 7, size=100_000)
print(round(rolls.mean(), 3))                   # 3.498

z = rng.normal(size=(2, 50_000))                # x2 built to correlate with x1 at rho = 0.6
x1 = 1.2 * z[0]
x2 = 0.8 * (0.6 * z[0] + np.sqrt(1 - 0.6**2) * z[1])
print(np.cov(x1, x2).round(2).tolist())         # [[1.44, 0.58], [0.58, 0.65]]
print(round(np.corrcoef(x1, x2)[0, 1], 2))      # 0.6

5. You’ll see this in

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done
Q1/4 A coin pays 1 for heads and 0 for tails. What is its expectation?
Q2/4 Var(X) = 0. What does that tell you?
Q3/4 Cov(X, Y) is large and positive. What does that mean?
Q4/4 Var(X) = 4. What is the standard deviation?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.