Learn machine learning from scratch

Start at chapter 1 and build up: what a model is, how it measures its own mistakes, and how it learns. Everything is worked out in plain Python, with the maths shown rather than assumed. Two shorter tracks hold reference notes on the maths and on modern architectures.

Track Progress: 0 Completed
0%
Maths for MLLEVEL 1 · BEGINNERModule 0 · Reading maths

Σ and Π are for-loops

A big sigma adds things up in a loop, a big pi multiplies them, and two of them side by side are a nested loop. Run one step at a time and watch.

10 min Prerequisites: Lesson 1 (subscripts)
Symbols in this lessonΣ summationΠ productlog logarithmxᵢ subscripts & superscript indices

1. The idea

Σ, a capital Greek sigma, means “add up”. It has three parts: under it, the index and where it starts; above it, where the index stops; to its right, the thing to add for each value of the index. Read Σᵢ₌₁ⁿ xᵢ as: for i from 1 to n, add xᵢ to a running total.

Π, a capital pi, is the same loop with multiplication: start the total at 1 instead of 0, and multiply each term in. Products appear when probabilities of independent events combine. They are usually turned into sums by taking a log, because the log of a product is the sum of the logs, and sums are easier to work with and kinder to floating-point numbers.

Two Σ signs side by side are a nested loop. Σᵢ Σⱼ Wᵢⱼ adds every entry of a matrix: the outer loop over rows, the inner loop over columns.

2. Try it: run the loop

Choose Σ or Π, choose what to add up, and step through the loop. The highlighted line of code is the one that just ran.

A Σ or Π over i from 1 to n, shown three ways: as the expanded sum or product with its value, as a three-line Python for-loop whose current line is highlighted, and as a running total that updates each time a step button is pressed.

3. The math

A sum
∑i=1nxi=x1+x2+⋯+xn\sum_{i=1}^{n} x_i = x_1 + x_2 + \cdots + x_n
The index i runs from the number below to the number above.
A product
∏i=1nxi=x1⋅x2⋯xn\prod_{i=1}^{n} x_i = x_1 \cdot x_2 \cdots x_n
The same loop with multiplication, starting from 1.
A constant comes out
∑i=1nc xi=c∑i=1nxi\sum_{i=1}^{n} c\,x_i = c \sum_{i=1}^{n} x_i
Multiplying every term by c is the same as multiplying the total by c.
A log turns a product into a sum
log⁡∏i=1npi=∑i=1nlog⁡pi\log \prod_{i=1}^{n} p_i = \sum_{i=1}^{n} \log p_i
Why log-likelihoods are sums, and why losses are full of logs.
A double sum is a nested loop
∑i=1m∑j=1nWij\sum_{i=1}^{m}\sum_{j=1}^{n} W_{ij}
Every entry of an m × n matrix, added up: m · n terms.

4. The code (python)

mm0_sums.py
import numpy as np

x = np.array([2.0, 5.0, 1.0, 4.0])

total = 0.0                       # the sum of x_i for i = 1..4, as a loop
for i in range(1, 5):             # i = 1, 2, 3, 4, as in the maths
    total += x[i - 1]
print(total, x.sum())             # 12.0 12.0

product = 1.0                     # a product starts at 1, not 0
for xi in x:
    product *= xi
print(product, x.prod())          # 40.0 40.0

p = np.array([0.9, 0.8, 0.5])     # the log of a product is the sum of the logs
print(round(np.log(p.prod()), 6), round(np.log(p).sum(), 6))   # -1.021651 -1.021651

5. You’ll see this in

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done
Q1/4 What is Σᵢ₌₁³ i²?
Q2/4 A loop computing Π must start its running total at…
Q3/4 Why do ML papers take the log of a likelihood that is a product of many probabilities?
Q4/4 W is a 3 × 4 matrix. How many numbers does Σᵢ Σⱼ Wᵢⱼ add up?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.