Maths for ML / Notation

\operatorname{softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_j e^{z_j}}

softmax

FunctionsBeginner

Say it “softmax of z”

Turns a vector of any scores into positive numbers that sum to 1 — a probability distribution.

Example

\operatorname{softmax}([1,1]) = [0.5, 0.5]

In code

np.exp(z) / np.exp(z).sum()

torch.softmax(z, dim=-1)

Where you’ll see it in ML

Classifier outputs, attention weights, next-token probabilities.

Don’t confuse it with

$e^{x},\ \exp(x)$ exponential Raises e (≈ 2.718) to the power x. Always positive; grows very fast.
$\sum_{i=1}^{n} x_i$ summation Add up the expression for each value of the index — it's a for-loop that accumulates.

Lessons that use it

Read an equation: Softmax

← All 94 symbols