Measure-Theoretic Information and Conditioning

If $P\ll ̸Q$, choose E_0 with $Q(E_0)=0<P(E_0)$. Since ${\varphi }^{-1}$ is measurable, $\varphi (E_0)=({\varphi }^{-1}{)}^{-1}(E_0)$ is measurable; then ${\varphi }_{\mathrm{\#}}Q(\varphi (E_0))=0<{\varphi }_{\mathrm{\#}}P(\varphi (E_0))$. Both divergences are $+\mathrm{\infty }$ by [F1].

If $P\ll Q$, let $r=dP\mathrm{/}dQ$. [given] The function $s=r\circ {\varphi }^{-1}$ is measurable. For every B in F, ${\int }_{B}sd({\varphi }_{\mathrm{\#}}Q)={\int }_{{\varphi }^{-1}(B)}rdQ=P({\varphi }^{-1}(B))={\varphi }_{\mathrm{\#}}P(B),$ so s is a Radon–Nikodym density of ${\varphi }_{\mathrm{\#}}P$ with respect to ${\varphi }_{\mathrm{\#}}Q$. By [F2], the push-forward integral identity applies separately to the nonnegative functions $(\ln s{)}^{+}$ and $(\ln s{)}^{-}$; the negative-part integrals are finite by [F1]. Therefore $D({\varphi }_{\mathrm{\#}}P\mathrm{\parallel }{\varphi }_{\mathrm{\#}}Q)=\int (\ln s{)}^{+}d({\varphi }_{\mathrm{\#}}P)-\int (\ln s{)}^{-}d({\varphi }_{\mathrm{\#}}P)=\int (\ln r{)}^{+}dP-\int (\ln r{)}^{-}dP=D(P\mathrm{\parallel }Q),$ including the case where the positive-part integral is $+\mathrm{\infty }$.

For fixed $z$, [F2] gives the probability measure ${\mathrm{\Pi }}_z$. [F2, given] If $A\in \mathcal{E}$ and $B\in \mathcal{F}$, then ${\mathrm{\Pi }}_z(A\times B)=K(z,A)L(z,B),$ which is a measurable function of z by the two kernel assumptions in [F1]. Also ${\mathrm{\Pi }}_z(E\times F)=1$ for every z.

Let $\mathcal{D}$ be the class of product-measurable $C\subseteq E\times F$ for which $z\mapsto {\mathrm{\Pi }}_z(C)$ is measurable. Step 1.1 shows every rectangle belongs to $\mathcal{D}$, and the whole product does as well. If $C\in \mathcal{D}$, then ${\mathrm{\Pi }}_z(C^c)=1-{\mathrm{\Pi }}_z(C)$, so $C^c\in \mathcal{D}$. If $C_j$ are pairwise disjoint members, then ${\mathrm{\Pi }}_z({\bigcup }_j{C}_j)={\lim }_n{\sum }_{j=1}^n{\mathrm{\Pi }}_z(C_j)$ by countable additivity, a pointwise limit of measurable functions. Thus $\mathcal{D}$ is a lambda-system.

Rectangles are a pi-system generating $\mathcal{E}\otimes \mathcal{F}$, so [F3] implies $\mathcal{D}=\mathcal{E}\otimes \mathcal{F}$. Every section ${\mathrm{\Pi }}_z$ is a probability by step 1.1; hence z↦${\mathrm{\Pi }}_z$ is the claimed probability kernel. All constructions are pointwise and use no choice axiom.

By [F1], $M(z,\cdot )=K(z,\cdot )\otimes L(z,\cdot )$ is a probability kernel into the standard-Borel product of the X and Y spaces. Apply [F2] with base law $P_Z$ and kernel M. It gives a unique probability law R and the displayed rectangle formula; taking $A$ and $B$ equal to their whole value spaces gives $R(C\times E_X\times E_Y)=P_Z(C)$ for every measurable $C$, so its Z marginal is $P_Z$. The same rectangle identity is the conditional-law identity for M given Z.

Let $K^{\mathrm{\prime }},L^{\mathrm{\prime }}$ be alternative versions. By [F3], there are P_Z-null sets $N_X,N_Y$ such that $K(z,\cdot )=K^{\mathrm{\prime }}(z,\cdot )$ for $z\mathrm{\notin }{N}_X$ and $L(z,\cdot )=L^{\mathrm{\prime }}(z,\cdot )$ for $z\mathrm{\notin }{N}_Y$. Their product measures agree for every $z$ outside $N_X\cup N_Y$. Thus the two joint laws have the same rectangle values $R(C\times A\times B)$ for every measurable C,A,B; [F4] then makes the laws equal. This establishes version independence.

First suppose $P\ll Q$ and $D(P\mathrm{\parallel }Q)<\mathrm{\infty }$. For bounded measurable $f$, let $Z_f=\int e^{f}dQ$ and define $Q_f$ by $d{Q}_f=e^f{Z}_f^{-1}dQ$. Then $Q_f$ is a probability equivalent to $Q$, and $dP\mathrm{/}d{Q}_f=(dP\mathrm{/}dQ)Z_f{e}^{-f}$. Taking logarithms and integrating under $P$ gives $D(P\mathrm{\parallel }{Q}_f)=D(P\mathrm{\parallel }Q)-{\mathrm{\Phi }}_{P,Q}(f)$. By [F2], ${\mathrm{\Phi }}_{P,Q}(f)\le D(P\mathrm{\parallel }Q)$. If $D(P\mathrm{\parallel }Q)=+\mathrm{\infty }$ or $P\ll ̸Q$, this upper bound is automatic.

If $P\ll ̸Q$, choose a measurable $A$ with $Q(A)=0<P(A)$. For $f_t=t{\mathbf{1}}_A$, $\int e^{f_t}dQ=1$ and ${\mathrm{\Phi }}_{P,Q}(f_t)=tP(A)\to +\mathrm{\infty }$. Thus the bounded-function supremum is $+\mathrm{\infty }=D(P\mathrm{\parallel }Q)$ in the singular case.

Suppose $P\ll Q$, let $r=dP\mathrm{/}dQ$, and set $\mathrm{\ell }=\log r$ on $\{r>0\}$, with $\mathrm{\ell }=-\mathrm{\infty }$ on $\{r=0\}$; the latter set is P-null. Define $f_n=\max (-n,\min (n,\mathrm{\ell }))$, taking $f_n=-n$ on $\{r=0\}$. The negative part of $\mathrm{\ell }$ is integrable by [F1]; its clipped negative parts converge by [F4], while the clipped positive parts increase to ${\mathrm{\ell }}^{+}$ and converge in integral by [F4]. Hence $\int f_{n}dP\to D(P\mathrm{\parallel }Q)$, including value $+\mathrm{\infty }$. Pointwise, $e^{f_n}\to r$ Q-a.e. and $0\le e^{f_n}\le r+1$, so [F4] gives $\int e^{f_n}dQ\to \int rdQ=1$. Therefore ${\mathrm{\Phi }}_{P,Q}(f_n)\to D(P\mathrm{\parallel }Q)$ and the bounded-function supremum is at least the divergence.

Fix a finite measurable partition $(A_i{)}_{i=1}^m$ and put $p_i=P(A_i)$, $q_i=Q(A_i)$. For constants $c_i$, the simple function $f={\sum }_i{c}_i{\mathbf{1}}_{A_i}$ has $\mathrm{\Phi }(f)={\sum }_i{p}_i{c}_i-\ln {\sum }_i{q}_i{e}^{c_i}$. If some $p_i>0=q_i$, this expression is unbounded above by taking $c_i\to +\mathrm{\infty }$. Otherwise its supremum over the $c_i$ is ${\sum }_{i:p_i>0}{p}_i\ln (p_i\mathrm{/}{q}_i)$: the upper bound follows by applying [F2] to the finite probability vector with masses proportional to $q_i{e}^{c_i}$, and the lower bound follows by taking $c_i=\ln (p_i\mathrm{/}{q}_i)$ when $p_i>0$ and $c_i\to -\mathrm{\infty }$ on cells with $p_i=0<q_i$. The cases $p_i=q_i=0$ do not affect either sum.

Every bounded measurable $f$ is uniformly approximated by finite-valued measurable simple functions. If $\mathrm{\parallel }f-g{\mathrm{\parallel }}_{\mathrm{\infty }}\le \epsilon$, then $\mathrm{\mid }\int fdP-\int gdP\mathrm{\mid }\le \epsilon$ and $e^{-\epsilon }\int e^{g}dQ\le \int e^{f}dQ\le e^{\epsilon }\int e^{g}dQ$, so $\mathrm{\mid }\mathrm{\Phi }(f)-\mathrm{\Phi }(g)\mathrm{\mid }\le 2\epsilon$. Thus the supremum over bounded functions equals the supremum over simple functions. Step 2.2 identifies the latter with the supremum over finite partitions; step 2.1 identifies it with $D(P\mathrm{\parallel }Q)$. No step subtracts infinite quantities.

Let $\rho =P+Q$ for arbitrary probability measures P,Q on E. [given, choose] For the algebra $\mathcal{A}$ in [F1], the class of measurable $B$ approximable in $\rho$-measure by members of $\mathcal{A}$ is a sigma-algebra: complements preserve symmetric-difference measure, finite unions preserve approximation, and a countable union is approximated by first truncating the union using finiteness of $\rho$ and then approximating that finite union. It contains $\mathcal{A}$, so it is all of $\mathcal{E}$. Hence any bounded measurable real function can be approximated in $L^1(\rho )$ by bounded simple functions from a countable family $\mathcal{G}$ of rational-valued simple functions on $\mathcal{A}$. On a fixed bound $M$, the functional ${\mathrm{\Phi }}_{P,Q}(g)=\int gdP-\ln \int e^{g}dQ$ is continuous under $L^1(\rho )$ approximation, because $e^t$ is Lipschitz on $[-M,M]$ and $\int e^{g}dQ\ge e^{-M}$. Thus [F3] gives $D(P\mathrm{\parallel }Q)={\sup }_{g\in \mathcal{G}}(\int gdP-\ln \int e^{g}dQ)\mathrm{.}$ The family $\mathcal{G}$ is countable and does not depend on P,Q.

For fixed $g\in \mathcal{G}$, the functions $z\mapsto \int g(x)K(z,dx)$ and $z\mapsto \int e^{g(x)}L(z,dx)$ are measurable by [F2], applying it to positive and negative parts for the first function. If $\mathrm{\mid }g\mathrm{\mid }\le M$, then $\int e^{g}dL(z)\ge e^{-M}>0$, so its logarithm is measurable. Therefore each candidate ${\mathrm{\Phi }}_z(g)$ is measurable.

Apply the identity in step 1.1 to $P=K(z,\cdot )$ and $Q=L(z,\cdot )$ for each fixed z. Then $D(K(z,\cdot )\mathrm{\parallel }L(z,\cdot ))={\sup }_{g\in \mathcal{G}}{\mathrm{\Phi }}_z(g)$, a countable supremum of measurable functions by step 2.1. This proves measurability and makes its nonnegative integral against any supplied base probability well-defined.

If two versions of the conditional law of $X$ given $Z$ are supplied, [F2] gives one $P_Z$-null set outside which their probability measures agree. Every atom probability, and hence the finite entropy sum in [F1], then agrees pointwise outside that set. Integrating gives the same $H(X\mid Z)$. The same argument applies to $(X,U)\mid Z$ and to $U\mid (X,Z)$.

Choose regular conditional laws of $(X,U)$ given $Z$ and of $U$ given $(X,Z)$; they exist by [A1, F3], since $(X,U)$ and $(X,Z)$ are standard Borel by [F4]. Write $p_{ij}(z)=P(X=i,U=j\mid Z=z)$, $p_i(z)={\sum }_j{p}_{ij}(z)$, and $m_{ij}(z)=P(U=j\mid X=i,Z=z)$ for the chosen versions. The function $p_i$ is itself a conditional law of $X$ given $Z$, so [F2] makes it agree almost everywhere with the version used in $H(X\mid Z)$. For each $i,j$ and measurable $C$ in the value space of $Z$, the two disintegrations [F3] give ${\int }_C{p}_i(z)m_{ij}(z)P_Z(dz)=P(X=i,U=j,Z\in C)={\int }_C{p}_{ij}(z)P_Z(dz)\mathrm{.}$ Both integrands are bounded and measurable. Testing the sets where one exceeds the other by at least $1\mathrm{/}n$, for each $n\ge 1$, shows $p_i(z)m_{ij}(z)=p_{ij}(z)$ almost everywhere; because the alphabets are finite, one null set works for every $i,j$.

Outside that null set, if $p_i(z)=0$ then every $p_{ij}(z)=0$, so the row contributes zero to the conditional entropy. If $p_i(z)>0$, then $m_{ij}(z)=p_{ij}(z)\mathrm{/}{p}_i(z)$ for every $j$. On positive entries, $\ln p_{ij}=\ln p_i+\ln m_{ij}$; multiplying by $-p_{ij}$ and summing over the finite alphabets gives $H(X,U\mid Z=z)=H(X\mid Z=z)+{\sum }_i{p}_i(z)H(U\mid X=i,Z=z)\mathrm{.}$

Integrate the finite conditional identity from step 2.1 with respect to $P_Z$. The disintegration identity [F3] identifies the last integral with $H(U\mid X,Z)$ and the other terms with their definitions. By the bound $-t\ln t\le 1\mathrm{/}e$ in [F1], each conditional entropy is bounded by its finite alphabet size divided by $e$, so all integrals are finite and no subtraction of infinite quantities occurs. Step 1.1 proves the accompanying version-independence assertion.

For each positive integer n, there are at most n atoms with mass at least $1\mathrm{/}n$, since any n+1 distinct such atoms would have total mass greater than one. For each atom of mass $p>0$, choose n with $1\mathrm{/}n<p$ by [F5]; thus S is a countable union of finite sets and is countable.

Suppose $P(S)=1$ and enumerate the distinct atoms as $(x_i)$ with masses $p_i>0$. By [F2], each singleton $(x_i,x_i)$ is measurable, so the function $r$ that equals $1\mathrm{/}{p}_i$ at $(x_i,x_i)$ and zero elsewhere is measurable. For each finite $N$, let $r_N={\sum }_{i\le N}{p}_i^{-1}{\mathbf{1}}_{\{(x_i,x_i)\}}$; these functions increase to $r$. The rectangle formula gives $(P\otimes P)\{(x_i,x_i)\}=p_i^2$. Hence for every measurable $C\subseteq E\times E$, [F6] gives ${\int }_{C}rd(P\otimes P)={\lim }_N{\sum }_{i\le N:(x_i,x_i)\in C}\frac{1}{p_i}{p}_i^2={\sum }_{i:(x_i,x_i)\in C}{p}_i=P_{XX}(C)\mathrm{.}$ Thus $r$ is the Radon–Nikodym density. Since $p_i\le 1$, $\log r\ge 0$ on the $P_{XX}$-full set of diagonal atoms; the partial sums $g_N={\sum }_{i\le N}\ln (1\mathrm{/}{p}_i){\mathbf{1}}_{\{(x_i,x_i)\}}$ increase to $\log r$ there. Another application of [F6] gives the KL integral ${\sum }_i{p}_i\ln (1\mathrm{/}{p}_i)$, including an infinite sum.

Suppose $P(S)<1$. The set $C=\mathrm{\Delta }\setminus (S\times S)$ is measurable, where $\mathrm{\Delta }=\{(x,x):x\in E\}$ by [F2]. The diagonal joint law assigns C mass $P(E\setminus S)>0$. Its product-measure section at $x\in S$ is empty; at $x\mathrm{\notin }S$ it is $\{x\}$, whose P-mass is zero. By [F3], $(P\otimes P)(C)=0$. Hence $P_{XX}\ll ̸P\otimes P$ and [F1] gives $H(X)=+\mathrm{\infty }$.

The two cases $P(S)=1$ and $P(S)<1$ exhaust all probability laws. Step 1.2 gives the nonnegative countable sum, which agrees on every finite subset with the finite Shannon formula [F4]; step 2.1 gives infinite divergence when there is positive non-atomic mass.

By [F2] and [F1], $I(X;Y)=0$ exactly when $P_{XY}=P_X\otimes P_Y$, which is the stated independence condition.

By [F3] and [F1], $I(X;Y\mid Z)=0$ exactly when the joint law $P_{ZXY}$ of $(Z,X,Y)$ equals the conditional-product reference law R.

Suppose $P_{ZXY}=R$, using the coordinate order $(Z,X,Y)$ of [F3]. For all measurable A,B,C the rectangle formula for R gives $P(X\in A,Y\in B,Z\in C)={\int }_{C}K(z,A)L(z,B)P_Z(dz)\mathrm{.}$ By the regular conditional law identities, $K(Z,A)$ and $L(Z,B)$ are versions of the two conditional expectations in [F4]. The displayed identity says their product is the conditional-expectation version of ${\mathbf{1}}_{\{X\in A,Y\in B\}}$ by [F5]. Thus [F4] holds for indicators. Linearity gives it for bounded simple functions; bounded measurable functions follow by bounded simple approximation and [F6]. Conversely, [F4] applied to indicators gives the same rectangle identity, so $P_{ZXY}$ and R agree on every rectangle $C\times A\times B$. These rectangles determine the probability law by [F7], hence $P_{ZXY}=R$.

Let $({\mathcal{G}}_n)$ be any increasing sequence of sub-sigma-algebras generating $\mathcal{G}$, and let $\rho =P+Q$ for probability measures on $\mathcal{G}$. The union $\mathcal{U}={\bigcup }_n{\mathcal{G}}_n$ is an algebra. The class of $A\in \mathcal{G}$ approximable in $\rho$-measure by a set in $\mathcal{U}$ is a sigma-algebra: complements preserve the symmetric-difference measure, finite unions preserve approximation, and for a countable union one first truncates it using continuity from below [F5] of the finite measure $\rho$, then approximates the finite union. Hence every set in $\mathcal{G}$ is approximable. Approximating bounded measurable functions first by simple functions and then their level sets gives bounded $\mathcal{U}$-measurable functions converging in $L^1(\rho )$, with one ${\mathcal{G}}_n$ containing each finite simple approximation. The approximants may be clipped to the same bound M. On that bound, [F2]'s variational functional is continuous under $L^1(\rho )$ approximation: $e^t$ is Lipschitz on $[-M,M]$, and the exponential integral is bounded below by $e^{-M}$. By [F2] applied to $P,Q$ and their restrictions to each ${\mathcal{G}}_n$, the supremum over all bounded ${\mathcal{G}}_n$-measurable functions is $D(P{\mathrm{\mid }}_{{\mathcal{G}}_n}\mathrm{\parallel }Q{\mathrm{\mid }}_{{\mathcal{G}}_n})$. These test classes increase, so the restricted divergences increase; the approximation shows their supremum equals the unrestricted variational supremum $D(P\mathrm{\parallel }Q)$. Hence $D(P\mathrm{\parallel }Q)={\lim }_{n}D(P{\mathrm{\mid }}_{{\mathcal{G}}_n}\mathrm{\parallel }Q{\mathrm{\mid }}_{{\mathcal{G}}_n})$.

By [F1], countable generating algebras exist; their first-n finite Boolean subalgebras yield increasing finite quantizers by labeling their atoms. For the quantizers in the Statement, the same generated-sigma property is assumed, so in either case $\sigma (q_n(X),r_n(Y))$ increases to the product sigma-algebra of $(X,Y)$. Under the map $(x,y)\mapsto (q_n(x),r_n(y))$, the push-forward of $P_X\otimes P_Y$ is $P_{q_n(X)}\otimes P_{r_n(Y)}$, as is checked on rectangles of the finite output sets. Since $\sigma (q_n(X),r_n(Y))$ consists exactly of preimages of output events, every bounded measurable test function on that sigma-algebra factors through the finite output map. Therefore its restricted KL equals the KL of the push-forward laws, which by [F3] is $I(q_n(X);r_n(Y))$. Applying step 1.1 proves the ordinary mutual-information limit.

For the conditional statement, put ${\mathcal{G}}_n=\sigma (Z,q_n(X),r_n(Y))$. These sigma-algebras increase to the full product sigma-algebra of $(Z,X,Y)$ because the entire Z sigma-algebra is retained. A bounded ${\mathcal{G}}_n$-measurable simple function factors through $(Z,q_n(X),r_n(Y))$: each level set is the preimage of a measurable output set, and finitely many output sets can be made disjoint without changing their preimages. Uniform simple approximation extends this factorization to all bounded variational test functions. Hence the KL restricted to ${\mathcal{G}}_n$ equals the KL between the push-forward joint and reference laws. Under the same map, the push-forward of the reference law in [F4] has, given Z=z, the product of the push-forward kernels $K_z\circ q_n^{-1}$ and $L_z\circ r_n^{-1}$: on a rectangle $C\times A^{\mathrm{\prime }}\times B^{\mathrm{\prime }}$ its mass is ${\int }_C{K}_z(q_n^{-1}{A}^{\mathrm{\prime }})L_z(r_n^{-1}{B}^{\mathrm{\prime }})P_Z(dz)$. By rectangle uniqueness, this is exactly the conditional-product reference law for the quantized variables. Step 1.1 therefore gives the conditional limit.

Steps 2.1 and 2.2 establish both limits. The proof quantizes X and Y only; it makes no claim that coarsening Z decreases conditional mutual information.

If $\mu \ll ̸\nu$, choose A with $\nu (A)=0<\mu (A)$. Then $Q(A\times F)=0<P(A\times F)$ by [F1], so $P\ll ̸Q$. Both $D(P\mathrm{\parallel }Q)$ and $D(\mu \mathrm{\parallel }\nu )+D_{\mu }(K\mathrm{\parallel }L)$ are $+\mathrm{\infty }$. Now suppose $\mu \ll \nu$ but $P\ll ̸Q$. If $D_{\mu }(K\mathrm{\parallel }L)<\mathrm{\infty }$, [F5] implies $D(K(x,\cdot )\mathrm{\parallel }L(x,\cdot ))<\mathrm{\infty }$ for $\mu$-almost every x, hence $K(x,\cdot )\ll L(x,\cdot )$ there by [F6]. For any product-measurable C with $Q(C)=0$, [F1] and [F5] give $L(x,C_x)=0$ for $\nu$-almost every x, hence for $\mu$-almost every x. On the intersection with the conditional absolute-continuity set, $K(x,C_x)=0$, and then [F1] gives $P(C)=0$, a contradiction. Thus $D_{\mu }(K\mathrm{\parallel }L)=+\mathrm{\infty }$ and the identity holds in this case too.

It remains to consider $P\ll Q$. Then $\mu \ll \nu$, since $Q(A\times F)=0$ whenever $\nu (A)=0$, and [F1] gives $P(A\times F)=\mu (A)$. Choose finite-valued nonnegative measurable representatives $a=d\mu \mathrm{/}d\nu$ and $h=dP\mathrm{/}dQ$ by [F3]. In particular, $\mu (\{a=0\})={\int }_{\{a=0\}}ad\nu =0.$ For each B in the countable determining algebra ${\mathcal{A}}_F$ of [F4] and every measurable A in E, [F1] and [F3] give ${\int }_{A}a(x)K(x,B)\nu (dx)=P(A\times B)={\int }_A({\int }_{B}h(x,y)L(x,dy))\nu (dx)\mathrm{.}$ The two integrands are measurable and nonnegative, and their integrals over E are finite. Their equality on every A therefore makes them equal $\nu$-almost everywhere. Intersect the corresponding measurable conull sets over the countable algebra to obtain T. Since $\mu \ll \nu$, the set $S=T\cap \{a>0\}$ has full $\mu$-measure. Let $\rho (t)=1\mathrm{/}t$ for $t>0$ and $\rho (0)=0$; this is Borel on $[0,+\mathrm{\infty })$. The first projection ${\pi }_E:E\times F\to E$ is measurable because ${\pi }_E^{-1}(A)=A\times F$. Define $r(x,y)={\mathbf{1}}_{S\times F}(x,y)h(x,y)\rho (a({\pi }_E(x,y)))\mathrm{.}$ By [F8], $r$ is finite-valued, nonnegative and product-measurable; it equals $h(x,y)\mathrm{/}a(x)$ for $x\in S$ and zero elsewhere. For each $x\in S$ the preceding identities give $K(x,B)={\int }_{B}r(x,y)L(x,dy)(B\in {\mathcal{A}}_F)\mathrm{.}$ The algebra contains F, so the right-hand measure has total mass one. Measure determination [F4] extends the equality to every measurable B: $r(x,\cdot )$ is a Radon–Nikodym representative of $K(x,\cdot )$ with respect to $L(x,\cdot )$ for every $x\in S$. No conditional identity outside the $\mu$-full set S is needed.