The dot product and the angle between vectors

1. The idea

The dot product multiplies matching entries and adds them up: 𝐚·𝐛 = a₁b₁ + a₂b₂ + … + a_d b_d. One number comes out. You have met it already: it is the weighted sum inside every neuron, 𝐰ᵀ𝐱.

The same number has a meaning you can see: 𝐚·𝐛 = ‖𝐚‖ ‖𝐛‖ cos θ, where θ is the angle between the arrows. Pointing the same way, cos θ = 1 and the dot product is as large as it can be for those lengths. At right angles, cos θ = 0 and the dot product is 0. Pointing apart, it is negative.

Divide out the lengths and you get cosine similarity, cos θ = 𝐚·𝐛 / (‖𝐚‖ ‖𝐛‖): a number between −1 and 1 that ignores how long the vectors are and keeps only their direction. It is how a search engine finds the sentence embeddings closest to a query, and the score attention gives each word is a dot product between a query vector and a key vector.

2. Try it: turn two arrows

Drag the two tips. Watch the sign of 𝐚·𝐛 as the angle passes 90°. The purple bar is the shadow of 𝐛 on 𝐚: its length times ‖𝐚‖ is the dot product.

3. The math

4. The code (python)

import numpy as np

a = np.array([3.0, 1.0])
b = np.array([1.0, 2.5])
print(a @ b)                       # 5.5

cos = (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))
print(round(cos, 3), round(np.degrees(np.arccos(cos)), 1))   # 0.646 49.8

def cosine(u, v):
    return (u @ v) / (np.linalg.norm(u) * np.linalg.norm(v))

# three made-up word embeddings
cat = np.array([0.9, 0.1, 0.3])
kitten = np.array([0.8, 0.2, 0.35])
car = np.array([0.1, 0.9, -0.2])
print(round(cosine(cat, kitten), 3), round(cosine(cat, car), 3))   # 0.989 0.136

5. You’ll see this in

• Core Course chapter 2: one neuron, many inputs a neuron is a dot product
• Modern Architectures: attention query · key scores
• Sentence embedding models in the catalog cosine similarity search

6. Check yourself