Vectors: arrows and arrays

A vector is a list of numbers, and for two numbers it is also an arrow. Adding and scaling look the same either way, and they are all a learning step does.

9 min Prerequisites: Module 0 (reading notation)

Symbols in this lesson𝐱 bold lowercase (vector)ℝⁿ real vector / matrix space η eta (learning rate)∇ nabla / gradient

1. The idea

A vector is a list of numbers, 𝐱 = (x₁, x₂, …, x_d). That is all a computer sees: a 1-D array. With two or three numbers there is also a picture, an arrow from the origin to the point with those coordinates. The picture is what makes the operations make sense; the list is what lets them work in 768 dimensions, where nobody can draw.

Adding two vectors adds matching entries. As arrows, put the tail of 𝐛 at the tip of 𝐚: where you end up is 𝐚 + 𝐛. Multiplying by a single number c (a scalar) multiplies every entry: the arrow keeps its line, gets c times as long, and turns round if c is negative.

In machine learning nearly everything is a vector: one example's features, a word's embedding, a layer's output, the gradient. Adding and scaling them is what a training step does: θ − η∇L takes the parameter vector and adds a scaled copy of the gradient vector.

2. Try it: add and scale arrows

Drag the tips of 𝐚 and 𝐛 (or focus one and use the arrow keys). The dashed copy of 𝐛 starts at the tip of 𝐚, and 𝐚 + 𝐛 ends where it does. The slider scales 𝐚 by c; try a negative c.

3. The math

A vector is a list of numbers

\mathbf{a} = \begin{pmatrix} 2 \\ 1 \end{pmatrix} \in \mathbb{R}^2

Two numbers, so an arrow in the plane: 2 across, 1 up.

Adding adds matching entries

\mathbf{a} + \mathbf{b} = (a_1 + b_1,\ a_2 + b_2)

Both vectors must have the same length.

Scaling multiplies every entry

c\,\mathbf{a} = (c\,a_1,\ c\,a_2)

Same line through the origin; c times as long; reversed if c < 0.

A training step is add and scale

\theta_{\text{new}} = \theta + (-\eta)\,\nabla L(\theta)

The new parameters are the old ones plus −η times the gradient.

4. The code (python)

mm1_vectors.py

import numpy as np

a = np.array([2.0, 1.0])
b = np.array([0.5, 2.0])
print(a + b)                   # [2.5 3. ]
print(1.5 * a)                 # [3.  1.5]
print(-1 * a)                  # [-2. -1.]

theta = np.array([0.5, -1.0, 2.0])   # parameters
grad = np.array([1.0, -2.0, 0.5])    # gradient of the loss
eta = 0.1
print(theta - eta * grad)            # [ 0.4  -0.8   1.95]

5. You’ll see this in

Gradient descent, read with the seven questions a vector minus a scaled vector
Core Course chapter 6: arrays and batches
Reference notes: linear algebra vector spaces, more formally

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done

Q1/4 What is (2, −1) + (1, 3)?

Q2/4 What is −2 · (1, −3)?

Q3/4 What does multiplying a vector by c = 0.5 do to its arrow?

Q4/4 Which of these sums is not defined?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.

Learn machine learning from scratch