Kullback–Leibler divergence

Let $P,Q$ be probability measures on the same measurable space (Probability measures and probability spaces). Absolute continuity $P\ll Q$ has the convention of Absolute continuity of a signed or complex measure with respect to a positive measure. If $P\ll ̸Q$, define $D(P\mathrm{\parallel }Q)=+\mathrm{\infty }$. If $P\ll Q$, assume AC (The Axiom of Choice) as in Likelihood ratio as a Radon–Nikodym derivative and let $r=dP\mathrm{/}dQ$ be a finite-valued nonnegative measurable representative of its Radon–Nikodym class; such a representative exists by A sigma-finite positive measure has a nonnegative Radon–Nikodym density version. On $\{r=0\}$ set $\mathrm{\ell }=0$, and on $\{r>0\}$ set $\mathrm{\ell }=\log r$. This function is measurable: for $a\ge 0$, $\{\mathrm{\ell }>a\}=\{r>\exp (a)\}$, and for $a<0$, $\{\mathrm{\ell }>a\}=\{r=0\}\cup \{r>\exp (a)\}$, by the inverse and monotonicity properties of the natural logarithm (The natural logarithm as the inverse of the exponential function, Order, continuity, range, and the product, quotient, and reciprocal laws for the natural logarithm). These are measurable sets by Extended-real-valued measurable functions. Since $P(\{r=0\})={\int }_{\{r=0\}}rdQ=0$ by Likelihood-ratio change of measure, the value chosen there does not affect $\int \mathrm{\ell }dP$. Define $D(P\mathrm{\parallel }Q):=\int \mathrm{\ell }dP,$ where $\log$ is the natural logarithm (The natural logarithm as the inverse of the exponential function).

The definition is independent of the representative: two representatives agree $Q$-almost everywhere, hence $P$-almost everywhere. Put $\psi (0):=0,\psi (t):=t\log t(t>0)\mathrm{.}$ The function $\psi$ is continuous at zero: for $s\ge 1$, $0\le \frac{\log s}{s}=\frac{1}{s}{\int }_1^s\frac{du}{u}\le \frac{1}{s}{\int }_1^s\frac{du}{\sqrt{u}}\le \frac{2}{\sqrt{s}}\mathrm{.}$ Thus $t\log t\to 0$ as $t\downarrow 0$; here $L=\log$ by The integral logarithm $L$ is the published natural logarithm and The integral logarithm $L(x):={\int }_1^x\frac{dt}{t}$ for $x>0$. On $(0,\mathrm{\infty })$, $\psi$ is a product of continuous functions by Order, continuity, range, and the product, quotient, and reciprocal laws for the natural logarithm and Sums, scalar multiples, products, absolute values, maxima, minima and quotients with nonvanishing denominator of continuous functions are continuous, as are constants, the identity and every polynomial function, so $\psi$ is continuous on $[0,\mathrm{\infty })$ and $\psi (r)$ is measurable by Extended-real-valued measurable functions. Its signed integral is defined by the positive and negative parts (The positive and negative parts of a function, The nonnegative Lebesgue integral).

For $0<t\le 1$, the function $-t\log t$ has derivative $-\log t-1$ and maximum $1\mathrm{/}e$ at $t=1\mathrm{/}e$: $e=\exp (1)>1$ by The real exponential function and the number $e$ by a power series and The exponential function is strictly increasing, $1\mathrm{/}e=\exp (-1)$ by The exponential is positive and satisfies $\exp (-x)=1\mathrm{/}\exp (x)$, and $\log (1\mathrm{/}e)=-1$ by Order, continuity, range, and the product, quotient, and reciprocal laws for the natural logarithm and the inverse definition. The mean value theorem gives increase before and decrease after this point (Sums, scalar multiples, products and quotients: $(f+g{)}^{\mathrm{\prime }}(c)=f^{\mathrm{\prime }}(c)+g^{\mathrm{\prime }}(c)$, $(\alpha f{)}^{\mathrm{\prime }}(c)=\alpha f^{\mathrm{\prime }}(c)$, $(fg{)}^{\mathrm{\prime }}(c)=f^{\mathrm{\prime }}(c)g(c)+f(c)g^{\mathrm{\prime }}(c)$, and $(f\mathrm{/}g{)}^{\mathrm{\prime }}(c)=(f^{\mathrm{\prime }}(c)g(c)-f(c)g^{\mathrm{\prime }}(c))\mathrm{/}g(c{)}^2$ when $g(c)\mathrm{\ne }0$, The integral logarithm satisfies $L^{\mathrm{\prime }}(x)=1\mathrm{/}x$ for $x>0$ and $L(1)=0$, The integral logarithm $L$ is the published natural logarithm, The mean value theorem, as the case $g(x)=x$ of Cauchy's: for $f$ continuous on $[a,b]$ with $a<b$ and differentiable on $(a,b)$ there is $c\in (a,b)$ with $f(b)-f(a)=f^{\mathrm{\prime }}(c)(b-a)$). At $t=0$ the negative part is zero. Since $Q$ is a probability measure, $\int \psi (r{)}^{-}dQ\le \frac{1}{e}\mathrm{.}$ The change-of-measure identity of Likelihood-ratio change of measure, applied to ${\mathrm{\ell }}^{-}$ gives $\int {\mathrm{\ell }}^{-}dP=\int \psi (r{)}^{-}dQ\le 1\mathrm{/}e$, so $\int \mathrm{\ell }dP$ is well-defined; applying it to both positive and negative parts gives $D(P\mathrm{\parallel }Q)=\int \mathrm{\ell }dP=\int \psi (r)dQ=\int r\log rdQ,$ with $0\log 0:=0$ and a finite negative part. Therefore the extended integral is well-defined without an $\mathrm{\infty }-\mathrm{\infty }$ ambiguity.

This convention is directional: the first argument must be absolutely continuous with respect to the second for the integral formula to apply.