Maths for ML / Notation

D_{\mathrm{KL}}(P \,\Vert\, Q)

KL divergence

Probability and statisticsAdvanced

Say it “K L divergence of P from Q”

Measures how different distribution Q is from P. Not symmetric: swapping P and Q changes the answer.

Example

D_{\mathrm{KL}}(P\Vert Q) = \sum_x P(x)\log\frac{P(x)}{Q(x)}

In code

scipy.stats.entropy(p, q)

torch.nn.functional.kl_div(q.log(), p, reduction='sum')

Where you’ll see it in ML

VAEs, knowledge distillation, RLHF penalty keeping a model near its reference.

Don’t confuse it with

$a \mid b$ vertical bar — many meanings Context decides: 'given' in probability, 'such that' in set-builder, 'divides' in number theory. Two bars ‖ mean a norm or KL divergence.
$\log x,\ \ln x$ logarithm The inverse of exp: what power of the base gives x. In ML papers log almost always means natural log (base e).

← All 94 symbols