Relative Entropy Variational, Chain, and Conditioning Rules

For $P\ll Q$ with density $h$, the negative part of $h\log h$ is integrable because $-u\log u\le 1\mathrm{/}e$ for $0\le u\le 1$; the density definition is therefore unambiguous. Also $u\log u\ge u-1$ for $u\ge 0$, so $D(P\mathrm{\parallel }Q)=\int h\log hdQ\ge \int (h-1)dQ=0$. If $P\ll ̸Q$, the defined value is $+\mathrm{\infty }\ge 0$.

For bounded $\varphi$, let $Q_{\varphi }$ have density $e^{\varphi }\mathrm{/}Q(e^{\varphi })$ relative to $Q$. When $P\ll Q$ and $D(P\mathrm{\parallel }Q)<\mathrm{\infty }$, the density logarithm identity gives $D(P\mathrm{\parallel }{Q}_{\varphi })=D(P\mathrm{\parallel }Q)-P(\varphi )+\log Q(e^{\varphi })\ge 0$ by step 1.1. Thus every variational test is at most $D(P\mathrm{\parallel }Q)$; the same upper bound is immediate if $D=+\mathrm{\infty }$. If $P\ll ̸Q$, choose a measurable $Q$-null set $N$ with $P(N)>0$ and use $\varphi =t{\mathbf{1}}_N$: then $Q(e^{\varphi })=1$ and $P(\varphi )=tP(N)\to \mathrm{\infty }$.

In the remaining case $P\ll Q$, let $h=dP\mathrm{/}dQ$ and set ${\varphi }_m=\max (-m,\min (m,\log h))$, taking ${\varphi }_m=-m$ on $\{h=0\}$. Then $e^{{\varphi }_m}\to h$ and $e^{{\varphi }_m}\le 1+h$, so [F4] gives $Q(e^{{\varphi }_m})\to 1$. The negative part of $h{\varphi }_m$ is bounded by $1\mathrm{/}e$, and its positive part increases to $(h\log h{)}^{+}$; hence [F4] gives $P({\varphi }_m)=\int h{\varphi }_{m}dQ\to D(P\mathrm{\parallel }Q)$, including when the positive entropy part is infinite. Combining this with step 2.1 proves the variational identity.

If $T$ is measurable, every bounded test $\psi$ on the target pulls back to the bounded test $\psi \circ T$ on the source, and $Q(e^{\psi \circ T})=(Q\circ T^{-1})(e^{\psi })$ while $P(\psi \circ T)=(P\circ T^{-1})(\psi )$. Taking suprema in step 3.1 proves the restriction inequality.

Finiteness of $D(P_{S,Y}\mathrm{\parallel }{P}_S\otimes Q_Y)$ gives a joint density and its conditional kernel by [F2]. Pushing forward by $(s,y)\mapsto (r(s),y)$ and using step 4.1 gives finite $D(P_{R,Y}\mathrm{\parallel }{P}_R\otimes Q_Y)$, so [F2] supplies the corresponding conditional kernel for $R$ as well. Thus all conditional densities and references in this clause are defined. For a conditioning observation $R$, applying the variational identity pointwise to the conditional density $k_R(r,y)$ shows $G(R)={\sup }_{\varphi \in B_b}(E_P\varphi (R,Y)-E_P\text{\hspace{0.17em}⁣}[\log \int e^{\varphi (R,y)}{Q}_Y(dy)]),$ where the tests range over bounded measurable functions of $(R,Y)$. The upper bound follows from step 2.1 for each conditional law; the reverse bound follows by using the jointly measurable bounded truncations of $\log k_R$ from step 3.1 and then [F3]–[F4]. When $R$ is a measurable function of $S$, every test depending on $(R,Y)$ is an admissible test depending on $(S,Y)$ with the same objective value, since the $Q_Y$ normalizer still depends only on $R$. Thus $G(R)\le G(S)$; nonnegativity is step 1.1 applied conditionally.

Suppose $P_{XY}$ has density $h(x,y)$ relative to $Q_X\otimes Q_Y$. Put $h_X(x)=\int h(x,y)Q_Y(dy)$. By [F3], $h_X$ is a density for $P_X$. Where $0<h_X(x)<\mathrm{\infty }$, [F2] supplies the conditional density $k(x,y)=h(x,y)\mathrm{/}{h}_X(x)$ relative to $Q_Y$; elsewhere use its supplied filler. On the full $P_X$-measure set of finite positive $h_X$, $h=h_{X}k$ and $\log h=\log h_X+\log k$ $P_{XY}$-almost everywhere. Integrating this identity, with nonnegative conditional entropies and finite total entropy, gives the displayed chain rule. If a restriction entropy is infinite, step 4.1 makes the joint entropy infinite as well, so no subtraction of two infinite terms is used.

Steps 1.1, 2.1 and 3.1 prove nonnegativity and the variational identity, step 4.1 proves restriction monotonicity, step 5.2 proves the product chain rule, and step 5.1 proves monotonicity under additional conditioning.