KL-neighborhood prior mass lower-bounds the likelihood denominator

The jointly measurable finite-valued densities and [F12], [F13] make $\{(\theta ,x):p_{\theta }(x)=0\}$ measurable. The density formula gives $P_0\ll P_{\theta }$ exactly when $\int 1_{\{p_{\theta }=0\}}{p}_{0}d\mu =0$; Tonelli [F4] makes this condition measurable in $\theta$. On this measurable set, the extended log-density parts defining $K(\theta )$ are product-measurable and Tonelli makes their section integrals measurable; outside it the KL value is $+\mathrm{\infty }$ by [F2]. Thus $\theta \mapsto K(\theta )$ and $B_{ϵ}$ are measurable. By [F1], for $\mathrm{\Pi }$-almost every $\theta \in B_{ϵ}$, ${\mathbb{E}}_{P_0}\log \frac{p_{\theta }(X)}{p_{{\theta }^{\ast }}(X)}=-(K(\theta )-K^{\ast })\ge -ϵ\mathrm{.}$ Both KL values are finite on this set, so [F2] makes the log ratio integrable.

For each fixed $\theta$ in this full-prior subset of $B_{ϵ}$, $L_i^{\theta }:=\log (p_{\theta }(X_i)\mathrm{/}{p}_{{\theta }^{\ast }}(X_i))$ are IID integrable real variables: the data are IID and [F6] preserves independence under the measurable log-ratio map. By [F5], $\frac{1}{n}{\sum }_{i=1}^n{L}_i^{\theta }⟶-(K(\theta )-K^{\ast })\ge -ϵP_0^{\mathrm{\infty }}\text{-almost surely}\mathrm{.}$ The event of this convergence is jointly measurable in $(\theta ,X_1,X_2,\dots )$. Tonelli [F4] applied to the indicator of its failure, under $\mathrm{\Pi }{\mathrm{\mid }}_{B_{ϵ}}\times P_0^{\mathrm{\infty }}$, therefore gives a $P_0^{\mathrm{\infty }}$-full event on which convergence holds for $\mathrm{\Pi }$-almost every $\theta \in B_{ϵ}$.

For fixed $n$, the finite product measures ${\mu }^n$ and $P_0^n$ exist by [F17]. The nonnegative jointly measurable likelihood ${\prod }_{i=1}^n{p}_{\theta }(x_i)$ is product-measurable by [F15] and integrates to one over ${\mu }^n$ for every $\theta$, by [F16] and repeated integration. Tonelli [F4] therefore gives ${\int }_{{\mathsf{X}}^n}{\int }_{\mathrm{\Theta }}{\prod }_{i=1}^n{p}_{\theta }(x_i)\mathrm{\Pi }(d\theta ){\mu }^n(d{x}_{1:n})=1.$ Let $Z_n(x_{1:n})={\int }_{\mathrm{\Theta }}{\prod }_i{p}_{\theta }(x_i)\mathrm{\Pi }(d\theta )$. By [F11], $Z_n<\mathrm{\infty }$ ${\mu }^n$-almost everywhere. The sample vector has law $P_0^n$ by [F9]. Its product density relative to ${\mu }^n$ is ${\prod }_i{p}_0(x_i)$: repeated Tonelli gives this density's rectangle formula, and [F10] identifies the resulting measure with $P_0^n$. Therefore $Z_n<\mathrm{\infty }$ $P_0^n$-almost surely.

For $\mathrm{\Pi }$-almost every $\theta \in B_{ϵ}$, finite KL gives $P_0\ll P_{\theta }$ by [F2], so $p_{\theta }(X_i)>0$ almost surely for each coordinate. The coordinate products are measurable by [F15]; Tonelli [F4] applied to the indicator of a zero likelihood shows that, for $P_0^n$-almost every sample, $p_{\theta }(X_i)>0$ for every $i$ for $\mathrm{\Pi }$-almost every $\theta \in B_{ϵ}$. Since $\mathrm{\Pi }(B_{ϵ})>0$, $Z_n>0$. Finally, $p_{{\theta }^{\ast }}(X_i)$ is finite and positive $P_0$-almost surely, so the relative denominator $D_n$ is finite and positive $P_0^n$-almost surely.

Fix $\alpha >ϵ$ and choose ${\alpha }^{\mathrm{\prime }}$ with $ϵ<{\alpha }^{\mathrm{\prime }}<\alpha$. On the event from step 1.2, for $\mathrm{\Pi }$-almost every $\theta \in B_{ϵ}$ the log likelihood average is eventually greater than $-{\alpha }^{\mathrm{\prime }}$. Therefore $e^{n\alpha }{\prod }_{i=1}^n\frac{p_{\theta }(X_i)}{p_{{\theta }^{\ast }}(X_i)}\ge e^{n(\alpha -{\alpha }^{\mathrm{\prime }})}⟶+\mathrm{\infty }\mathrm{.}$ Fatou's lemma [F14] for the nonnegative integrands gives ${\mathrm{lim\hspace{0.17em}inf}}_{n\to \mathrm{\infty }}{e}^{n\alpha }{D}_n\ge {\int }_{B_{ϵ}}{\mathrm{lim\hspace{0.17em}inf}}_{n\to \mathrm{\infty }}{e}^{n\alpha }{\prod }_{i=1}^n\frac{p_{\theta }(X_i)}{p_{{\theta }^{\ast }}(X_i)}\mathrm{\Pi }(d\theta )=+\mathrm{\infty },$ since $\mathrm{\Pi }(B_{ϵ})>0$. Hence $D_n\ge e^{-n\alpha }$ eventually.

Steps 1.3–2.2 establish finite-positive evidence, and step 2.2 establishes the eventual exponential lower bound asserted in the Statement.