How to read an equation

The seven questions

1. What comes out? Start with the left-hand side. Is it one number (a loss), a list of numbers (a layer's output), a probability between 0 and 1? Once you know what comes out, you know what the right-hand side has to produce.
2. List the parts. Go through every symbol and sort it into a pile: data you feed in, parameters the model learns, constants someone picked, and indices that only count. Learned things are usually w, W, b or Greek letters like θ; data is usually x and y.
3. Check the shapes. Write the shape next to each term: a number, a vector of length d, a matrix with n rows and d columns. The shapes on both sides must agree. If they don't, you've misread something, and in code that is exactly where the bug would be.
4. Say what each function does. exp makes everything positive and blows large values up. log does the opposite and turns products into sums. A sum collapses a list into one number. A square drops the sign and punishes big values more than small ones. max picks one.
5. Ask what makes it bigger or smaller. Look at what sits on top of a fraction and what sits underneath, what is squared, what is inside a log. Which term grows fastest when its input grows?
6. Push the inputs to extremes. Let a variable go to 0, or to something huge, or make all the inputs equal. What the equation does at the edges usually tells you what it is for.
7. Write it as code. Turn it into three to five lines of NumPy and run it on numbers small enough to check by hand. If you can code it, you can read it.

Try them on real equations

Each of these takes one equation from machine learning apart with the seven questions. Point at any piece of it to see what it is, change its inputs and watch the output, then see it written in NumPy.

• Linear model$$\hat{y} = \mathbf{w}^\top\mathbf{x} + b$$
• Mean squared error$$L = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2$$
• Sigmoid$$\sigma(z) = \frac{1}{1 + e^{-z}}$$
• Softmax$$\operatorname{softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$$
• Cross-entropy$$L = -\sum_{i=1}^{K} y_i \log p_i$$
• Gradient descent$$\theta \leftarrow \theta - \eta \nabla L(\theta)$$