Maths for ML / Read an equation

The linear model

The simplest model there is, and the inside of every neuron: multiply each input by a weight, add them up, add one more number.

y^=w⊤x+b\htmlClass{mx-p mx-p-yhat}{\hat{y}} = \htmlClass{mx-p mx-p-w}{\mathbf{w}}^{\htmlClass{mx-p mx-p-T}{\top}}\htmlClass{mx-p mx-p-x}{\mathbf{x}} + \htmlClass{mx-p mx-p-b}{b}

The parts

Point at any part of the equation, or pick it from the list.

  • the prediction · output · shape one number

    The hat means “estimated”: this is the model's guess for the true value y.

  • the weights · learned · shape a vector of length d

    One weight per input feature, learned in training. A big positive weight means that feature pushes the prediction up a lot.

  • transpose · operation · shape turns a column into a row

    Written as a row times a column, wᵀx is a (1×d) times (d×1) product, which gives one number. NumPy's w @ x does the same for 1-D arrays.

  • the input features · data · shape a vector of length d

    One example's data: d numbers describing it.

  • the bias · learned · shape one number

    Shifts every prediction by the same amount. It is what the model predicts when every feature is 0.

Read it with the seven questions

  1. What comes out? One number, ŷ: the model's guess for y.
  2. List the parts. Data: x. Learned: w and b. No constants. An index is hiding inside wᵀx: it runs over the d features.
  3. Check the shapes. w and x both have length d, so wᵀx is one number. One number plus b is one number, matching ŷ.
  4. Say what each function does. No functions at all: wᵀx is just w₁x₁ + w₂x₂ + … + w_d x_d, a weighted sum.
  5. Ask what makes it bigger or smaller. A feature raises ŷ if its weight is positive and lowers it if negative. A large |wⱼ| makes ŷ sensitive to xⱼ.
  6. Push the inputs to extremes. Set x = 0 and ŷ = b. Double x and wᵀx doubles. The model is a straight line (a flat plane in more dimensions): it cannot bend.
  7. Write it as code. y_hat = w @ x + b, one line.

Change it and watch

With one feature, wᵀx is just w·x, so the model is a line. Move w and b and try to get the line through the points: the red lines are the errors, and the number under the plot is their mean squared size.

A scatter of eight points rising from left to right, and a line y-hat = w x + b drawn over them. A slider for w tilts the line and a slider for b moves it up and down; dashed lines show each point's distance from the line.

Now write it in NumPy
import numpy as np

w = np.array([0.8, -0.3, 1.5])   # one weight per feature
x = np.array([2.0, 1.0, 0.5])    # one example's features
b = 0.4

y_hat = w @ x + b                # 0.8*2 - 0.3*1 + 1.5*0.5 + 0.4
print(round(y_hat, 2))           # 2.45

You’ll see this in

Symbols on this pagex̂ hat𝐱 bold lowercase (vector)ᵀ transpose· dot product