Maths for ML / Read an equation

Mean squared error

How far off is the model, on average? Take every error, square it, and average the squares.

L=1n∑i=1n(yi−y^i)2\htmlClass{mx-p mx-p-L}{L} = \htmlClass{mx-p mx-p-n}{\frac{1}{n}}\htmlClass{mx-p mx-p-sum}{\sum_{i=1}^{n}}\htmlClass{mx-p mx-p-sq}{\left(\htmlClass{mx-p mx-p-y}{y_i} - \htmlClass{mx-p mx-p-yhat}{\hat{y}_i}\right)^2}

The parts

Point at any part of the equation, or pick it from the list.

  • the loss · output · shape one number, never negative

    One score for the whole dataset. Training tries to make it small.

  • divide by n · constant · shape one number (n is how many examples there are)

    Turns a total into an average, so the loss doesn't grow just because you have more data.

  • sum over examples · index · shape i counts 1, 2, …, n

    A for-loop over the dataset: work out the bracket for each example i and add them up.

  • the true value · data · shape one number per example

    The answer from the data for example i.

  • the prediction · output · shape one number per example

    What the model guessed for example i.

  • the square · operation · shape one number in, one out

    Removes the sign, so misses above and below both count, and punishes big misses much more than small ones: a miss of 3 costs 9, a miss of 1 costs 1.

Read it with the seven questions

  1. What comes out? One number, L, the average squared miss. It can't be negative.
  2. List the parts. Data: every yᵢ. Predictions: every ŷᵢ, which come from the model's parameters. Constant: n. Index: i, running over the examples.
  3. Check the shapes. Each bracket is one number; squared, still one number; summed over n examples and divided by n, one number.
  4. Say what each function does. The square removes signs and stretches big errors; the sum collapses n numbers into one; 1/n averages.
  5. Ask what makes it bigger or smaller. Errors grow the loss quadratically. One miss of 10 adds 100 to the sum, as much as a hundred misses of 1.
  6. Push the inputs to extremes. Perfect predictions give L = 0. A single wild outlier can dominate the whole loss, which is why some models use absolute error instead.
  7. Write it as code. L = np.mean((y - y_hat) ** 2).

Change it and watch

The line is fixed at ŷ = x + 1. Drag a point (or focus it and use the arrow keys) and watch its square grow: the area of each square is that point's squared error, and L is their average.

Five points near a fixed line. On each point sits a square whose side is the point's distance from the line, so its area is the squared error. The points can be dragged up and down, and the average area, L, is shown underneath.

Now write it in NumPy
import numpy as np

y     = np.array([3.0, 5.0, 4.0, 7.0])   # true values
y_hat = np.array([2.5, 5.5, 4.0, 5.0])   # predictions

L = np.mean((y - y_hat) ** 2)            # (0.25 + 0.25 + 0 + 4) / 4
print(L)                                 # 1.125

You’ll see this in

Symbols on this pageΣ summationx² exponentx̂ hatxᵢ subscripts & superscript indices