Expected information gain is joint relative entropy

By [F1] and [F3], Tonelli makes $p_d$ measurable and gives ${\int }_{\mathsf{Y}}{p}_d(y)\nu (dy)={\int }_{\mathrm{\Theta }}{\int }_{\mathsf{Y}}{p}_d(y\mid \theta )\nu (dy){\mu }_0(d\theta )=1$. Thus $P_{Y\mid d}$ is a probability measure and $p_d(y)<\mathrm{\infty }$ for $\nu$-almost every $y$. By [F2], its mass on $\{p_d=0\}\cup \{p_d=\mathrm{\infty }\}$ is zero, and the posterior formula in [F1] is the conditional-law version on $G_d$.

By [F2], the joint law $P_d$ has density $p_d(y\mid \theta )$ relative to ${\mu }_0\otimes \nu$. By step 1.1, $p_d$ is measurable and its lift to the product is product-measurable. Define $M_d(C):={\int }_C{p}_d(y)d({\mu }_0\otimes \nu )$; [F8] makes $M_d$ a measure. By [F11], the product measures ${\mu }_0\otimes \nu$ and $Q_d={\mu }_0\otimes P_{Y\mid d}$ exist. For each measurable rectangle $A\times B$, Tonelli [F3] gives $M_d(A\times B)={\mu }_0(A){\int }_B{p}_d(y)\nu (dy)=Q_d(A\times B)$. By [F10], choose $B_k\uparrow \mathsf{Y}$ with $\nu (B_k)<\mathrm{\infty }$. The measurable rectangles form a generating pi-system, and $\mathrm{\Theta }\times B_k$ is an increasing exhaustion in it; both $M_d$ and $Q_d$ assign it the same finite mass $P_{Y\mid d}(B_k)\le 1$. Thus [F4] gives $M_d=Q_d$ on the full product sigma-algebra. By [F9], for every nonnegative product-measurable $g$, $\int gd{Q}_d=\int g{p}_d(y)d({\mu }_0\otimes \nu )$. Since $G_d$ is measurable, define $r_d(\theta ,y)=p_d(y\mid \theta )\mathrm{/}{p}_d(y)$ on $G_d$ where $p_d(y\mid \theta )<\mathrm{\infty }$, and set it to zero elsewhere. The numerator and lifted denominator are product-measurable, and division is Borel on this region, so $r_d$ is finite-valued and product-measurable. For each $y\in G_d$, the omitted set where $p_d(y\mid \theta )=\mathrm{\infty }$ is ${\mu }_0$-null because its integral is the finite number $p_d(y)$; hence $r_d(\cdot ,y)$ is a density of ${\mu }_d^y$. The set where $p_d(y\mid \theta )=\mathrm{\infty }$ is product-null because the joint density has total integral one. On $\{p_d=0\}$, [F1] gives ${\int }_{\mathrm{\Theta }}{p}_d(y\mid \theta ){\mu }_0(d\theta )=0$, so [F5] and Tonelli show that this region has zero joint mass; the $\{p_d=\mathrm{\infty }\}$ fibers are $\nu$-null by step 1.1. Thus $P_d(G_d^c)=0$. Since $Q_d=M_d$ has density $p_d$ relative to ${\mu }_0\otimes \nu$, it also gives the product-null set where the likelihood is infinite zero mass, and step 1.1 gives $Q_d(G_d^c)=0$. On $G_d$ outside that product-null set, $r_d{p}_d=p_d(y\mid \theta )$. Applying [F9] to $g={\mathbf{1}}_C{r}_d$ therefore gives ${\int }_C{r}_{d}d{Q}_d={\int }_C{p}_d(y\mid \theta )d({\mu }_0\otimes \nu )=P_d(C)$ for every product-measurable $C$, so $r_d=d{P}_d\mathrm{/}d{Q}_d$.

For each $y\in G_d$, [F1] makes $r_d(\cdot ,y)$ a probability density relative to ${\mu }_0$, so [F6] gives $D({\mu }_d^y\mathrm{\parallel }{\mu }_0)=\int r_d(\theta ,y)\log r_d(\theta ,y){\mu }_0(d\theta )$ and bounds the fiber's negative part by $1\mathrm{/}e$. Weighting by $p_d(y)$ and using step 1.1 bounds the full negative part by $\frac{1}{e}{\int }_{G_d}{p}_d(y)\nu (dy)=1\mathrm{/}e$. Tonelli [F3] on the positive and negative parts separately, together with the product density in step 2.1, therefore gives $\mathrm{EIG}(d)=\int r_d\log r_{d}d{Q}_d$ with finite negative part and possibly infinite positive part.

Substituting the fiber identity from step 3.1 into the definition of EIG and using the separate Tonelli exchanges from that step gives $\mathrm{EIG}(d)=\int r_d\log r_{d}d{Q}_d$. By [F6] and step 2.1, this is $D(P_d\mathrm{\parallel }{Q}_d)$; [F7] gives nonnegativity. The finite negative part from step 3.1 rules out an $\mathrm{\infty }-\mathrm{\infty }$ subtraction, proving the stated extended identity.