KL prior mass gives a posterior denominator lower bound in probability

Define $k(\theta ,x)=p_0(x)\log (p_0(x)\mathrm{/}{p}_{\theta }(x))$ as $0$ when $p_0=0$ and as $+\mathrm{\infty }$ when $p_0>0,p_{\theta }=0$. The sets $\{p_0=0\}$, $\{p_0>0,p_{\theta }=0\}$, and $\{p_0>0,p_{\theta }>0\}$ are product-measurable; on the last set the logarithm is a difference of measurable logarithms by [F13]–[F14]. Thus $k^{+}$ and $k^{-}$ are product-measurable by [F10]. Where $p_{\theta }>p_0>0$, setting $u=p_0\mathrm{/}{p}_{\theta }\in (0,1)$ gives $-u\log u\le 1\mathrm{/}e$, so $k^{-}\le p_{\theta }\mathrm{/}e$ and $\int k^{-}d\mu \le 1\mathrm{/}e$. If $P_0\ll P_{\theta }$, then $P_0\{p_{\theta }=0\}=0$; [F17] gives $p_0=0$ $\mu$-almost everywhere there. Consequently, for every measurable $A$, the function $r_{\theta }=p_0\mathrm{/}{p}_{\theta }$ on $\{p_{\theta }>0\}$ and $0$ otherwise satisfies ${\int }_A{r}_{\theta }d{P}_{\theta }={\int }_{A\cap \{p_{\theta }>0\}}{p}_{0}d\mu =P_0(A)$. By [F18], it is a Radon–Nikodym derivative, so [F3] identifies $\mathrm{K}\mathrm{L}(P_0\mathrm{\parallel }{P}_{\theta })=\int k(\theta ,x)d\mu$. Conversely, if $P_0\{p_{\theta }=0\}=0$, then every $P_{\theta }$-null set $A$ has $p_{\theta }=0$ $\mu$-almost everywhere on $A$, hence $P_0(A)=0$; thus failure of $P_0\ll P_{\theta }$ implies $P_0\{p_{\theta }=0\}>0$. In that case the same integral is $+\mathrm{\infty }$, again equal to [F3]. Finally, Tonelli [F4] makes the integrals of $k^{+}$ and $k^{-}$ measurable in $\theta$; the latter is finite, so their difference gives a measurable extended-real KL map. For the log-square coordinate, the specified ratio conventions and [F10], [F13], [F14] give a nonnegative product-measurable integrand, whose section integral is measurable by [F4]. Hence $K_n\in \mathcal{T}$.

Set $Y_{\theta ,i}:=p_{\theta }(X_i)\mathrm{/}{p}_0(X_i)$ when $p_0(X_i)>0$ and $0$ otherwise. This is a measurable function of $X_i$ by [F10, F14]. Since $P_0\{p_0=0\}={\int }_{\{p_0=0\}}{p}_{0}d\mu =0$, change of density using [F1, F2] gives ${\mathbb{E}}_{P_0}{Y}_{\theta ,i}={\int }_{\{p_0>0\}}{p}_{\theta }d\mu \le 1$. For each fixed $\theta$, the $Y_{\theta ,i}$ are independent measurable transforms of the IID coordinates; [F6, F7, F12] therefore give ${\mathbb{E}}_{P_0^n}{\prod }_i{Y}_{\theta ,i}\le 1$. Joint measurability and Tonelli [F4] imply ${\mathbb{E}}_{P_0^n}{D}_n\le 1$, so [F15] gives $D_n<\mathrm{\infty }$ almost surely.

By [A1, F3], every $\theta \in K_n$ satisfies $P_0\ll P_{\theta }$, so $P_0\{p_{\theta }=0\}=0$. For each fixed $\theta$, the IID sample therefore has ${\prod }_i{p}_{\theta }(X_i)>0$ almost surely. Apply Tonelli [F4] to the zero-product indicator on $K_n\times {\mathsf{X}}^n$ with the product of the prior conditioned on K_n and the n-fold sampling law: for $P_0^n$-almost every sample, the conditional prior measure of parameters with zero product is zero. Hence the product likelihood is positive for prior-almost every $\theta \in K_n$, and [F17] implies its integral over the positive-mass set $K_n$ is positive. On the full-probability event where each $p_0(X_i)>0$, this implies $D_n>0$.

Write ${\bar{\mathrm{\Pi }}}_n=\mathrm{\Pi }(\cdot \mid K_n)$ and define the jointly measurable extended-real function $\mathrm{\ell }(\theta ,x)$ by the ratio conventions in the Statement: it is zero on $\{p_0=0\}$, equals $\log (p_{\theta }\mathrm{/}{p}_0)$ where $p_0,p_{\theta }>0$, and is $-\mathrm{\infty }$ where $p_0>0,p_{\theta }=0$; the stated arithmetic and Borel-composition facts [F10, F13, F14] justify this measurability. This avoids assuming a common $P_0$-full set on which all densities in an uncountable family are positive. For every $\theta \in K_n$, finite KL gives $P_0\ll P_{\theta }$, so $\mathrm{\ell }(\theta ,\cdot )$ is finite $P_0$-almost everywhere; the local second-moment condition gives ${\mathbb{E}}_{P_0}\mathrm{\ell }(\theta ,X_1{)}^2\le {ϵ}_n^2$. Holder [F16] on the probability space yields ${\mathbb{E}}_{P_0}\mathrm{\mid }\mathrm{\ell }(\theta ,X_1)\mathrm{\mid }\le {ϵ}_n$, and the directional convention [F3] gives ${\mathbb{E}}_{P_0}\mathrm{\ell }(\theta ,X_1)=-\mathrm{K}\mathrm{L}(P_0\mathrm{\parallel }{P}_{\theta })\ge -{ϵ}_n^2$. Tonelli [F4] applied to the positive and negative parts therefore makes $Z(x):=\int \mathrm{\ell }(\theta ,x){\bar{\mathrm{\Pi }}}_n(d\theta )$ finite almost everywhere, measurable after assigning zero on the null exceptional set, and integrable with ${\mathbb{E}}_{P_0}Z(X_1)\ge -{ϵ}_n^2$. Tonelli also gives ${\mathbb{E}}_{P_0}\int \mathrm{\ell }(\theta ,X_1{)}^2{\bar{\mathrm{\Pi }}}_n(d\theta )\le {ϵ}_n^2$; for $P_0$-almost every $x$, Jensen [F5] for $z\mapsto z^2$ gives $Z(x{)}^2\le \int \mathrm{\ell }(\theta ,x{)}^2{\bar{\mathrm{\Pi }}}_n(d\theta )$. Hence ${\mathbb{E}}_{P_0}Z(X_1{)}^2\le {ϵ}_n^2$.

Set $Z_i:=Z(X_i)$. By [F6] and [F7], the $Z_i$ are IID; [F8] and step 1.4 imply $\mathrm{Var}(Z_i)\le {ϵ}_n^2$, so the centered variables are square-integrable and integrable. For $i\mathrm{\ne }j$, the centered variables are independent measurable transforms of independent coordinates, so [F12] factors their product expectation and their covariance is zero. Applying [F11] to their finite sum therefore gives $\mathrm{Var}(n^{-1}{\sum }_i{Z}_i)=n^{-2}{\sum }_i\mathrm{Var}(Z_i)\le {ϵ}_n^2\mathrm{/}n$.

The mean of $n^{-1}{\sum }_i{Z}_i$ is at least $-{ϵ}_n^2$ and its variance is at most ${ϵ}_n^2\mathrm{/}n$, by independence in step 2.1. Thus the event $n^{-1}{\sum }_i{Z}_i<-2{ϵ}_n^2$ forces a deviation from its mean of at least ${ϵ}_n^2$. Chebyshev [F9] yields $P_0^n\text{\hspace{0.17em}⁣}\{n^{-1}{\sum }_i{Z}_i<-2{ϵ}_n^2\}\le \frac{1}{n{ϵ}_n^2}\mathrm{.}$

By Tonelli [F4] and the local second-moment bound, for $P_0^n$-almost every sample each of the finitely many functions $\theta \mapsto \mathrm{\ell }(\theta ,X_i)$ is square-integrable under ${\bar{\mathrm{\Pi }}}_n$; Cauchy–Schwarz [F16] makes their sum integrable. Step 1.3 and the $P_0^n$-full event where each $p_0(X_i)>0$ ensure that $\exp ({\sum }_i\mathrm{\ell }(\theta ,X_i))={\prod }_i{p}_{\theta }(X_i)\mathrm{/}{p}_0(X_i)$ for ${\bar{\mathrm{\Pi }}}_n$-almost every $\theta$. Step 1.2 bounds the integrated exponential by the finite $D_n\mathrm{/}\mathrm{\Pi }(K_n)$, so Jensen [F5] applies to that sum and the exponential. On the complementary event, it gives $D_n\ge \mathrm{\Pi }(K_n)\int \exp \text{\hspace{0.17em}⁣}({\sum }_i\mathrm{\ell }(\theta ,X_i)){\bar{\mathrm{\Pi }}}_n(d\theta )\ge \mathrm{\Pi }(K_n)\exp \text{\hspace{0.17em}⁣}\left({\sum }_i{Z}_i\right)\ge e^{-(C+2)n{ϵ}_n^2}\mathrm{.}$ Combining with step 3.1 proves the probability bound.