Finite Kullback–Leibler divergence

For probability vectors $P=(p_x)$ and $Q=(q_x)$ on the same finite nonempty alphabet, their Kullback–Leibler divergence (or relative entropy) in the stated direction from $P$ to $Q$ is $D(P\mathrm{\parallel }Q):={\sum }_{x:p_x>0}{p}_x\log \frac{p_x}{q_x}\mathrm{.}$ The logarithm is natural. If $p_x>0$ and $q_x=0$ for at least one $x$, set $D(P\mathrm{\parallel }Q)=+\mathrm{\infty }$. If $p_x=0$, its contribution is zero, including when $q_x=0$; equivalently, $0\log (0\mathrm{/}{q}_x)=0$. When $P\ll Q$, all ratios in the sum are positive and finite.

The generator $f(t)=t\log t$ is admissible for Finite f-divergence with support conventions by The finite KL generator has the required convex boundary extension. Its boundary values $f(0)=0$ and $f_{\mathrm{\infty }}=+\mathrm{\infty }$ make that f-divergence exactly the displayed KL formula, including the singular-support case.