Multiplying matrices: check the shapes first

Each entry of AB is a row of A dotted with a column of B, so the inner sizes must match. (m × k)(k × n) gives (m × n), and most shape errors are this rule broken.

11 min Prerequisites: Lesson 10 (a matrix is a transformation)

Symbols in this lessonX capital letter (matrix)× times / dimension · dot product ᵀ transpose

1. The idea

To multiply a matrix A by a matrix B, take each row of A and each column of B and compute their dot product: entry (i, j) of AB is row i of A · column j of B. For that dot product to exist, a row of A and a column of B must be the same length. A's number of columns must equal B's number of rows.

So shapes come first: (m × k) times (k × n) gives (m × n). The inner sizes must match and disappear; the outer sizes survive. Check this before anything else. Most shape errors in ML code are exactly this mismatch, and NumPy says so with a ValueError naming the two shapes.

Order matters: AB and BA are usually different, and often only one of them exists. A dense layer applied to a batch is XW, with X of shape (batch, inputs) and W of shape (inputs, outputs). The batch survives on the left, the outputs on the right, and the inputs cancel in the middle.

2. Try it: will these multiply?

Set the shapes of A and B, or try the examples. The rule is the inner pair: columns of A against rows of B.

3. The math

One entry is one dot product

(AB)_{ij} = \sum_{k} A_{ik}\,B_{kj}

Row i of A, column j of B, multiplied entry by entry and added.

The shape rule

(m \times k)\,(k \times n) = (m \times n)

Inner sizes match and cancel; outer sizes survive.

Order matters

AB \neq BA \ \text{ in general}

Even for square matrices, where both exist.

A layer on a batch

\underbrace{X}_{B \times d_{\text{in}}}\ \underbrace{W}_{d_{\text{in}} \times d_{\text{out}}} = \underbrace{H}_{B \times d_{\text{out}}}

One row per example in, one row per example out.

A dot product is a tiny matrix product

\mathbf{a}^\top\mathbf{b}:\ (1 \times d)(d \times 1) = (1 \times 1)

A row times a column gives one number.

4. The code (python)

mm2_matmul.py

import numpy as np

A = np.array([[1.0, 2.0, 0.0],
              [0.0, 1.0, 3.0]])          # 2 x 3
B = np.array([[1.0, 0.0],
              [2.0, 1.0],
              [0.0, 4.0]])               # 3 x 2
print((A @ B).shape, (B @ A).shape)      # (2, 2) (3, 3)
print((A @ B).tolist())                  # [[5.0, 2.0], [2.0, 13.0]]
print(A[0] @ B[:, 1])                    # 2.0

X = np.ones((32, 784))                   # a batch of 32 flattened images
W = np.zeros((784, 128))                 # a layer with 128 outputs
print((X @ W).shape)                     # (32, 128)
try:
    W @ X                                # (784 x 128) @ (32 x 784): 128 != 32
except ValueError:
    print("ValueError")                  # ValueError

5. You’ll see this in

Core Course chapter 6: arrays and batches a layer on a whole batch at once
The linear model, read with the seven questions step 3 is the shape check
The multi-layer perceptron in the catalog

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done

Q1/4 A is 4 × 3 and B is 3 × 7. What shape is AB?

Q2/4 Which of these products doesn't exist?

Q3/4 X is 64 × 100 (a batch) and W is 100 × 10 (a layer). What shape is XW?

Q4/4 Which numbers make entry (2, 1) of AB?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.

Learn machine learning from scratch