Learn machine learning from scratch

Start at chapter 1 and build up: what a model is, how it measures its own mistakes, and how it learns. Everything is worked out in plain Python, with the maths shown rather than assumed. Two shorter tracks hold reference notes on the maths and on modern architectures.

Track Progress: 0 Completed
0%
Maths for MLLEVEL 1 · BEGINNERModule 1 · Vectors

Linear combinations and span: which points can you reach?

Scale some vectors and add them up. The points you can reach that way are their span, and when two vectors point along the same line you lose a whole direction.

10 min Prerequisites: Lesson 6 (adding and scaling vectors)
Symbols in this lesson𝐱 bold lowercase (vector)rank rank{ | } set-builder notationℝⁿ real vector / matrix space

1. The idea

A linear combination of vectors scales each one and adds the results: c₁𝐮 + c₂𝐯. With two arrows in the plane you can ask which points you could reach by choosing c₁ and c₂. The set of reachable points is called the span of 𝐮 and 𝐯.

Usually the answer is everywhere: two arrows that point in different directions span the whole plane, and for any target there is exactly one pair (c₁, c₂). But if 𝐯 is just a scaled copy of 𝐮, the two are linearly dependent: 𝐯 adds no new direction, every combination stays on one line, and most targets are out of reach.

This question sits behind the rank of a matrix and behind many “how much can this layer express” arguments. A layer computes W𝐱, which is a linear combination of W's columns with the entries of 𝐱 as the weights, so its outputs live in the span of those columns. LoRA builds on this: it bets that the change a fine-tune needs only uses a few independent directions.

2. Try it: reach the target

Use the sliders to land c₁𝐮 + c₂𝐯 on the red target, or press Solve. Then drag 𝐯 until it lies along 𝐮 and see what happens to the reachable points.

Two draggable arrows u and v and a draggable target point. Sliders set c1 and c2, and the combination c1 u + c2 v is drawn as two dashed arrows placed tip to tail, ending at the result. A Solve button finds the coefficients that reach the target. When u and v are parallel, the line they span is drawn and most targets can't be reached.

3. The math

A linear combination
c1u+c2vc_1\mathbf{u} + c_2\mathbf{v}
Scale each vector, then add.
The span
span⁡(u,v)={ c1u+c2v∣c1,c2∈R }\operatorname{span}(\mathbf{u},\mathbf{v}) = \{\, c_1\mathbf{u} + c_2\mathbf{v} \mid c_1, c_2 \in \mathbb{R} \,\}
Every point some choice of c₁ and c₂ reaches.
Dependent vectors lose a direction
v=k u  ⇒  span⁡(u,v) is a line\mathbf{v} = k\,\mathbf{u} \;\Rightarrow\; \operatorname{span}(\mathbf{u},\mathbf{v}) \text{ is a line}
In ℝ² two independent vectors span everything; two dependent ones only a line.
A matrix times a vector combines the columns
Wx=x1 w1+x2 w2+⋯+xn wnW\mathbf{x} = x_1\,\mathbf{w}_1 + x_2\,\mathbf{w}_2 + \cdots + x_n\,\mathbf{w}_n
𝐰ⱼ is column j of W. The outputs of W𝐱 can only be in the span of W's columns.
Finding the coefficients is solving a system
(u1v1u2v2)(c1c2)=t\begin{pmatrix} u_1 & v_1 \\ u_2 & v_2 \end{pmatrix}\begin{pmatrix} c_1 \\ c_2 \end{pmatrix} = \mathbf{t}
Put 𝐮 and 𝐯 side by side as columns; Module 2 shows how to solve it.

4. The code (python)

mm1_span.py
import numpy as np

u = np.array([2.0, 0.5])
v = np.array([0.5, 1.5])
target = np.array([3.0, 3.0])

A = np.column_stack([u, v])        # u and v as the columns of a matrix
c = np.linalg.solve(A, target)     # the c1, c2 with c1*u + c2*v = target
print(c.round(3))                  # [1.091 1.636]
print((c[0] * u + c[1] * v).round(6))   # [3. 3.]

w = 2 * u                          # dependent: a scaled copy of u
print(np.linalg.matrix_rank(np.column_stack([u, w])))   # 1
print(np.linalg.matrix_rank(A))                         # 2

5. You’ll see this in

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done
Q1/4 With 𝐮 = (1, 0) and 𝐯 = (0, 1), what is 3𝐮 − 2𝐯?
Q2/4 𝐮 = (1, 2) and 𝐯 = (2, 4). What is their span?
Q3/4 At most how many vectors in ℝ² can be linearly independent?
Q4/4 W𝐱 is a linear combination of what?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.