Finite-Alphabet Entropy and Divergence — Examples

Substituting $q=(p,1-p)$ into [F1] gives the displayed formula. If $p=0$ or $p=1$, the nonzero coordinate is $1$ and [F3] gives $\ln 1=0$, so $h(p)=0$.

Let $u=(1\mathrm{/}2,1\mathrm{/}2)$. The finite KL formula gives $D(q\mathrm{\parallel }u)=p\ln (2p)+(1-p)\ln (2(1-p))=\ln 2-h(p)$, also at $p=0,1$ because zero terms are omitted. By [F2], this value is nonnegative and vanishes exactly when $q=u$, i.e. $p=1\mathrm{/}2$. Thus $h(p)\le \ln 2$ with a unique maximizer.

If $0<p<1$, then $\ln p<0$ and $\ln (1-p)<0$ by [F3], so both terms in $h(p)$ are positive. Along with step 1.1 this proves that the only zeros are the endpoints.

The support of $p$ is contained in that of $q$ because $q_0=\epsilon >0$. By [F1], $D(p\mathrm{\parallel }q)=1\cdot \ln (1\mathrm{/}\epsilon )=-\ln \epsilon$, a finite real number. Since $p\mathrm{\ne }q$, [F2] makes this value strictly positive, consistent with the formula.

The second coordinate has $q_1=1-\epsilon >0=p_1$. Thus [F1] gives $D(q\mathrm{\parallel }p)=+\mathrm{\infty }$. The two directions are unequal, which refutes the statement.

For the separate disjoint point masses $p={\delta }_0$ and $q={\delta }_1$, coordinate $0$ has $p_0>0=q_0$ and coordinate $1$ has $q_1>0=p_1$; applying [F1] in each direction gives $D(p\mathrm{\parallel }q)=D(q\mathrm{\parallel }p)=+\mathrm{\infty }$. This records the disjoint-support boundary.

If $X,Y$ are independent, then $p_{XY}(x,y)=p_X(x)p_Y(y)$ at every coordinate. Each positive-support ratio in [F1] is $1$, so every summand is zero and $I(X;Y)=0$.

If $Y=f(X)$, then for each $x$ with $p_X(x)>0$, $p_{Y\mathrm{\mid }X=x}$ has mass $1$ at $f(x)$ and $0$ elsewhere, so [F4] gives entropy $-1\ln 1=0$. Averaging these conditional entropies by [F3] gives $H(Y\mathrm{\mid }X)=0$, and [F2] yields $I(X;Y)=H(Y)$.