Learn machine learning from scratch

Start at chapter 1 and build up: what a model is, how it measures its own mistakes, and how it learns. Everything is worked out in plain Python, with the maths shown rather than assumed. Two shorter tracks hold reference notes on the maths and on modern architectures.

Track Progress: 0 Completed
0%
Maths for MLLEVEL 1 · BEGINNERModule 5 · Probability and statistics

Bayes' rule: updating on evidence

Turn P(evidence | cause) into P(cause | evidence). The prior matters more than intuition says: a good test for a rare condition mostly finds false alarms.

9 min Prerequisites: Lesson 22 (conditional probability)
Symbols in this lessonP(A|B) conditional probability∝ proportional toP(·) probability

1. The idea

Bayes' rule turns a conditional probability around. Often you know P(evidence | cause), such as how likely a positive test is if someone has a condition, but you want P(cause | evidence): how likely the condition is, given a positive test. Bayes' rule connects the two: P(A | B) = P(B | A) P(A) / P(B).

The P(A) in it, the prior, is what you believed before the evidence, and it matters enormously. If a condition affects 1 person in 100, a test that catches 90% of cases and gives false alarms only 5% of the time still produces mostly false alarms: out of 1000 people, 9 true positives against about 50 false ones.

In machine learning the same rule reads posterior ∝ likelihood × prior: p(θ | data) ∝ p(data | θ) p(θ). Maximum likelihood ignores the prior. Adding one, for example a belief that weights are probably small, gives regularization; in these terms, weight decay is a prior.

2. Try it: a thousand people, one test

Blue squares have the condition; red outlines tested positive. Make the condition rarer and watch the share of positives that are real collapse, then see which slider rescues it.

A grid of 1000 small squares. Filled ones have the condition and outlined ones tested positive. Sliders set how common the condition is, how often the test catches it, and how often it gives a false alarm. The text counts true and false positives and gives the probability of having the condition after a positive test.

3. The math

Bayes' rule
P(A∣B)=P(B∣A) P(A)P(B)P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)}
Posterior = likelihood × prior / evidence.
The denominator, spelled out
P(B)=P(B∣A) P(A)+P(B∣¬A) P(¬A)P(B) = P(B \mid A)\,P(A) + P(B \mid \lnot A)\,P(\lnot A)
All the ways to see the evidence: with the cause, and without it.
The test, worked through
P(ill∣+)=0.9×0.010.9×0.01+0.05×0.99≈0.154P(\text{ill} \mid +) = \frac{0.9 \times 0.01}{0.9 \times 0.01 + 0.05 \times 0.99} \approx 0.154
About one positive in six or seven is real.
For model parameters
p(θ∣D)∝p(D∣θ) p(θ)p(\theta \mid \mathcal{D}) \propto p(\mathcal{D} \mid \theta)\,p(\theta)
∝: equal up to a constant that doesn't depend on θ.

4. The code (python)

mm5_bayes.py
def posterior(prior, sensitivity, false_alarm):
    evidence = sensitivity * prior + false_alarm * (1 - prior)
    return sensitivity * prior / evidence

print(round(posterior(0.01, 0.9, 0.05), 3))     # 0.154
print(round(posterior(0.10, 0.9, 0.05), 3))     # 0.667
print(round(posterior(0.01, 0.99, 0.05), 3))    # 0.167
print(round(posterior(0.01, 0.9, 0.005), 3))    # 0.645

5. You’ll see this in

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done
Q1/4 In Bayes' rule, what is the prior?
Q2/4 Why does a good test for a rare condition give mostly false positives?
Q3/4 Which proportionality is Bayes' rule for parameters?
Q4/4 In the 1-in-100 example, which change raises P(ill | +) the most?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.