Plain, bold and capital letters; hats, bars and stars

Bold, capital and plain letters are different kinds of object, and a hat, a bar or a star changes what a letter means. The widget ties three of them together.

10 min Prerequisites: Lesson 1 (subscripts and superscripts)

Symbols in this lesson𝐱 bold lowercase (vector)X capital letter (matrix)x̂ hat x̄ bar θ* star argmin arg min / arg max

1. The idea

The same letter in a different style is a different kind of object. Plain italic x is a single number, a scalar. Bold lowercase 𝐱 is a list of numbers, a vector. A capital X is a table, a matrix. Papers rarely say this out loud; they expect you to see it in the font.

Marks above a letter tell you its role. A hat, ŷ, is an estimate: what the model predicts for y. A bar, ȳ, is an average. A star, θ*, is the best value of something: the parameters that make the loss as small as it can be.

The widget puts all three in one picture. A model that predicts the same number θ for every example makes predictions ŷ = θ. Its squared-error loss is lowest at θ*, and θ* turns out to be exactly ȳ, the mean of the data. The steps below show why.

2. Try it: hat, bar and star

The model predicts one number θ for all six examples. Slide θ and watch the loss on the right. Where is it lowest, and what is the mean of the data?

3. The math

Scalar, vector, matrix

x \in \mathbb{R} \qquad \mathbf{x} \in \mathbb{R}^{d} \qquad X \in \mathbb{R}^{n \times d}

One number; a list of d numbers; a table with n rows and d columns.

Hat: an estimate

\hat{y} = \theta

Here the model ignores its input and predicts θ for everyone.

Bar: an average

\bar{y} = \frac{1}{n}\sum_{i=1}^{n} y_i

For y = 2, 3, 4, 5, 6, 10: ȳ = 30 / 6 = 5.

Star: the best value

\theta^{*} = \arg\min_{\theta}\ \frac{1}{n}\sum_{i=1}^{n}(y_i - \theta)^2 = \bar{y}

argmin means “the θ that makes this smallest”, not the smallest value itself.

4. How it works

Write the loss

With ŷ = θ for everyone, the loss is L(θ) = (1/n) Σ (yᵢ − θ)², the mean squared error.

Find where its slope is zero

The derivative is dL/dθ = −(2/n) Σ (yᵢ − θ). At the bottom of the U it is zero, so Σ (yᵢ − θ) = 0.

Solve for θ

Σ yᵢ − nθ = 0, so θ = (1/n) Σ yᵢ = ȳ. The best constant prediction under squared error is the mean.

5. The code (python)

mm0_conventions.py

import numpy as np

y = np.array([2.0, 3.0, 4.0, 5.0, 6.0, 10.0])

def loss(theta):                        # predict y_hat = theta for every example
    return np.mean((y - theta) ** 2)

thetas = np.linspace(0, 10, 101)        # try 0.0, 0.1, ..., 10.0
theta_star = thetas[np.argmin([loss(t) for t in thetas])]
print(y.mean(), theta_star)             # 5.0 5.0
print(round(loss(theta_star), 3))       # 6.667

6. You’ll see this in

Core Course chapter 3: Measuring error the mean, and why squared error loves it
The linear model, read with the seven questions ŷ, 𝐰 and 𝐱 in one line

7. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done

Q1/4 In ‖𝐱‖ = 5, what kind of object is 𝐱?

Q2/4 A paper writes ŷ = f(𝐱). What does the hat on y tell you?

Q3/4 Your model must predict one number θ for every example. Under squared error, which θ is best?

Q4/4 Which symbol is most likely a matrix?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.

Learn machine learning from scratch