Maths for ML / Read an equation

The sigmoid

Squashes any number into the range 0 to 1, so a score can be read as a probability.

σ(z)=11+e−z\htmlClass{mx-p mx-p-sig}{\sigma(\htmlClass{mx-p mx-p-z}{z})} = \frac{\htmlClass{mx-p mx-p-one}{1}}{\htmlClass{mx-p mx-p-den}{1 + \htmlClass{mx-p mx-p-exp}{e^{-z}}}}

The parts

Point at any part of the equation, or pick it from the list.

  • the output · output · shape one number strictly between 0 and 1

    Read it as a probability, such as “the chance this email is spam”.

  • the score (logit) · data · shape any real number

    Usually the output of a linear model, z = wᵀx + b: big and positive means “yes”, big and negative means “no”.

  • the numerator · constant · shape the number 1

    Caps the output: 1 divided by something bigger than 1 is always below 1.

  • the denominator · operation · shape one number, always greater than 1

    Large when z is very negative (so the output is near 0), close to 1 when z is large (so the output is near 1).

  • exp of minus z · operation · shape one positive number

    e (about 2.718) to the power −z. Always positive; tiny when z is large, enormous when z is very negative.

Read it with the seven questions

  1. What comes out? One number between 0 and 1, never exactly 0 or 1.
  2. List the parts. Input: z. Constants: 1 and e. No learned parameters in the sigmoid itself; they live in whatever produced z.
  3. Check the shapes. A number in, a number out. On a vector, NumPy applies it to each entry separately.
  4. Say what each function does. exp makes e⁻ᶻ positive; adding 1 makes the denominator bigger than 1; dividing 1 by it lands between 0 and 1.
  5. Ask what makes it bigger or smaller. Bigger z makes e⁻ᶻ smaller, the denominator smaller, and the output bigger. The output only ever goes up as z goes up.
  6. Push the inputs to extremes. z = 0 gives exactly 1/2. As z → +∞ the output → 1; as z → −∞ it → 0. Past about |z| = 6 it barely moves, which is why gradients through a saturated sigmoid are tiny.
  7. Write it as code. 1 / (1 + np.exp(-z)).

Change it and watch

Slide z and watch each piece of the fraction. Push z past 6 or below −6 to see the output pin itself near 1 or 0.

An S-shaped curve rising from 0 on the left to 1 on the right, crossing 0.5 at z = 0. A slider moves a point along the curve, and the values of e to the minus z, the denominator and sigma of z are shown.

Now write it in NumPy
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

print(sigmoid(np.array([-4.0, 0.0, 4.0])))   # [0.01798621 0.5        0.98201379]

You’ll see this in

Symbols on this pageσ sigmaexp exponential∞ infinity