Measure-Theoretic Information and Conditioning — Examples

On $\{0,1\}$, define $d(x,y)=0$ if $x=y$ and $d(x,y)=1$ otherwise; distinct endpoints force at least one of the two intermediate distances to be 1, so the triangle inequality holds. For a Cauchy sequence, tolerance $1\mathrm{/}2$ forces all sufficiently late terms to be equal, so the metric is complete. The finite set itself is a finite dense subset, and radius-$1\mathrm{/}2$ balls are singletons, so the topology is discrete. By the definitions in [F4], this is a Polish presentation of the discrete sigma-algebra, which is therefore standard Borel. If $p=0$ or $p=1$, X and Y are the same constant almost surely; their joint law and the product of their marginals are the same point mass. Thus [F1] gives a finite-alphabet mutual information, [F2] gives $I(X;Y)=0$, and [F3] gives the same entropy.

Suppose $0<p<1$. The joint law of $(X,Y)$ has masses $p$ at $(1,1)$ and $1-p$ at $(0,0)$, and zero elsewhere. The product law has masses $p^2$ at $(1,1)$ and $(1-p{)}^2$ at $(0,0)$. By [F1]–[F2], $I(X;Y)=p\ln \frac{p}{p^2}+(1-p)\ln \frac{1-p}{(1-p{)}^2}=-p\ln p-(1-p)\ln (1-p)\mathrm{.}$ The last quantity is $H(X)$ by [F3].

The endpoint cases in step 1.1 and the interior case in step 2.1 exhaust $p\in [0,1]$ and establish the claimed identity throughout.

For a finite value set S, define $d(s,t)=0$ for $s=t$ and 1 otherwise; distinct endpoints force at least one intermediate distance to be 1, so the triangle inequality holds. Tolerance $1\mathrm{/}2$ makes every Cauchy sequence eventually constant, hence the metric is complete; S is itself finite and dense, and radius-$1\mathrm{/}2$ balls are singletons, so the topology is discrete. By the definitions in [F2], each such space is standard Borel; its finite products are standard Borel by the finite-product supplier. Given $Z=(i,j)$, the pair $(X,Y)$ is the point $(i,j)$, and its two conditional marginals are point masses at i and j. Their product is the point mass at $(i,j)$, so it agrees with the conditional joint law at every Z-value of positive probability. The conditional-product reference law therefore equals the joint law of $(Z,X,Y)$. By [F1], its KL divergence, and hence $I(X;Y\mid Z)$, is zero.

The variable W has one value. The joint law of $(W,X,Y)$ has mass $1\mathrm{/}2$ at $(w,0,0)$ and $1\mathrm{/}2$ at $(w,1,1)$. The conditional marginals given W are both uniform on $\{0,1\}$; their product reference law assigns mass $1\mathrm{/}4$ to each of the four pairs at w. Therefore [F1]–[F3] give $I(X;Y\mid W)=\frac{1}{2}\ln \frac{1\mathrm{/}2}{1\mathrm{/}4}+\frac{1}{2}\ln \frac{1\mathrm{/}2}{1\mathrm{/}4}=\ln 2.$

The map from Z to W is constant and thus coarsens the conditioning information. The two computed values satisfy $I(X;Y\mid W)>I(X;Y\mid Z)$, so this example rules out a general data-processing claim that coarsening the conditioning variable decreases conditional mutual information.

By [F2], the restriction $P_Z$ is a probability and $P_Z(\{0\})=0$. On each finite target set, define distance 0 on equal points and 1 on distinct points; distinct endpoints force at least one intermediate distance to be 1, so the triangle inequality holds. Tolerance $1\mathrm{/}2$ makes every Cauchy sequence eventually constant; the whole finite set is dense, and radius-$1\mathrm{/}2$ balls are singletons, so the space is complete and discrete, hence Polish and standard Borel by [F1]. The product target is standard Borel by the finite-product supplier in [F1]. For any Borel $C\subseteq [0,1]$ and $A\subseteq \{0,1\}$, replacing ${\delta }_0$ by $K^{\mathrm{\prime }}$ changes ${\int }_{C}K(z,A)P_Z(dz)$ only at $z=0$, a null point. Thus both K and K' satisfy the conditional-law integral identity for X given Z; the constant kernel L satisfies it for Y, and $J(z)={\delta }_{(0,0)}$ satisfies it for $(X,Y)$. These kernels are measurable because their only nonconstant set of values is the Borel singleton $\{0\}$.

Let $J(z)={\delta }_{(0,0)}$ be a conditional law of $(X,Y)$ given Z. With the original kernels, $K(z)\otimes L(z)={\delta }_{(0,0)}$ for every z, so the pointwise KL is zero. With the alternative kernels, the product is ${\delta }_{(1,0)}$ at z=0 and ${\delta }_{(0,0)}$ for z>0. By [F5], $D(J(0)\mathrm{\parallel }{K}^{\mathrm{\prime }}(0)\otimes L^{\mathrm{\prime }}(0))=+\mathrm{\infty }$ because the first law charges $(0,0)$ and the reference gives it mass zero; for z>0 the divergence is zero. Thus the pointwise integrands differ only at a null point, and both conditional-KL integrals in [F4] are zero.

The joint law $P_{ZXY}$ and the conditional-product reference law built from K,L agree. K' equals K $P_Z$-almost everywhere, so [F3] says the reference law built from K',L' is the same law. Consequently both versions give $I(X;Y\mid Z)=D(P_{ZXY}\mathrm{\parallel }{P}_{ZXY})=0$.

Define $P(A)={\lambda }_1(A)$ for Borel $A\subseteq [0,1]$. By [F1], these sets are Lebesgue measurable and the restricted measure is countably additive; the box formula gives $P([0,1])=1$, so P is a probability. For each $x\in [0,1]$, the degenerate interval $[x,x]=\{x\}$ has measure $x-x=0$. Thus every singleton has P-mass zero, and the atomic set $S=\{x:P(\{x\})>0\}$ is empty.

The identity map on $[0,1]$ has law P, and its value space is standard Borel by [F2]. Since $P(S)=0<1$, [F3] gives $H(X)=+\mathrm{\infty }$.

Extending the density by zero outside $[0,1]$ gives $f={\mathbf{1}}_{[0,1]}$. On the interval $f\ln f=1\ln 1=0$, and off the interval the integrand is zero by the convention in [F4]. Thus $f\ln f$ is the zero function on $\mathbb{R}$; its integral is zero by [F4], so the differential entropy is $h(f)=0$.