Maths for ML / Read an equation

Cross-entropy

The loss for classifiers. It looks at the probability the model gave to the right answer, and charges heavily for confident mistakes.

L=−∑i=1Kyi log⁡pi\htmlClass{mx-p mx-p-L}{L} = \htmlClass{mx-p mx-p-neg}{-}\htmlClass{mx-p mx-p-sum}{\sum_{i=1}^{K}}\htmlClass{mx-p mx-p-y}{y_i}\,\htmlClass{mx-p mx-p-log}{\log \htmlClass{mx-p mx-p-p}{p_i}}

The parts

Point at any part of the equation, or pick it from the list.

  • the loss · output · shape one number, 0 or more

    Small when the model gives the true class a high probability.

  • the minus sign · operation · shape flips the sign

    The log of a probability is 0 or negative. The minus makes the loss positive, so smaller is better.

  • sum over classes · index · shape i counts the K classes

    Adds one term per class.

  • the label · data · shape 1 for the true class, 0 for every other (one-hot)

    Switches off every term except the true class's, so the sum is really just one term.

  • natural log · operation · shape one number in, one out

    log 1 = 0, and log p heads to −∞ as p heads to 0. That is what makes a confident wrong answer so expensive.

  • predicted probability · output · shape between 0 and 1, adding up to 1 over the classes

    The model's probability for class i, usually straight out of a softmax.

Read it with the seven questions

  1. What comes out? One number, the loss for one example. 0 is perfect; there is no upper limit.
  2. List the parts. Data: the labels yᵢ. Predictions: the probabilities pᵢ. Index: i over the K classes. No constants.
  3. Check the shapes. y and p both have length K; each term is a number; the sum is one number.
  4. Say what each function does. log turns probabilities into penalties; the minus sign makes them positive; the one-hot y picks out the true class.
  5. Ask what makes it bigger or smaller. Only the true class's probability matters: with one-hot labels, L = −log p_true. The lower that probability, the bigger the loss, and it grows faster and faster.
  6. Push the inputs to extremes. p_true = 1 gives 0. p_true = 0.5 gives 0.693. p_true = 0.01 gives 4.6, and p_true → 0 sends the loss to ∞. A model that is sure and wrong pays the most.
  7. Write it as code. L = -np.sum(y * np.log(p)).

Change it and watch

Slide the probability the model gives the true class. Near 1 the loss is almost free; below 0.1 it climbs steeply.

The curve minus log p, falling from very high values near p = 0 to zero at p = 1. A slider moves a point along it and shows the loss for that probability.

Now write it in NumPy
import numpy as np

y = np.array([0, 1, 0])            # the true class is the second one
p = np.array([0.2, 0.7, 0.1])      # the model's probabilities

L = -np.sum(y * np.log(p))         # only -log(0.7) survives
print(round(L, 4))                 # 0.3567

You’ll see this in

Symbols on this pagelog logarithmΣ summationP(·) probability𝟙 indicator function