Posterior consistency from KL support and uniformly summable tests

For $n\ge 1$ put $L_n(\theta ,x_{1:n})={\prod }_{i=1}^{n}p(\theta ,x_i)$ and $Z_n(x_{1:n})={\int }_{\mathrm{\Theta }}{L}_n(\theta ,x_{1:n})\mathrm{\Pi }(d\theta )$. The coordinate maps into $\mathrm{\Theta }\times \mathsf{X}$ are measurable by the product sigma-algebra definition [F20]; joint measurability of $p$ and closure under finite products [F23] make $L_n$ product-measurable. For each fixed $\theta$, repeated Tonelli [F6, F21] and $\int p(\theta ,x)d\mu =1$ give $\int L_n(\theta ,\cdot )d{\mu }^n=1$. By [F24], $L_n(\theta ,\cdot )d{\mu }^n$ is a measure. On every measurable rectangle ${\prod }_{i=1}^n{A}_i$, repeated Tonelli and the density identity give mass ${\prod }_i{P}_{\theta }(A_i)$; this is also the rectangle mass of $P_{\theta }^n$ by [F8, F10]. Rectangles generate the product sigma-algebra [F20], and both are probability measures, so [F22], using the whole sample space as a finite-measure exhaustion, identifies these measures. Tonelli over $\mathrm{\Pi }$ and ${\mu }^n$ now gives $\int Z_{n}d{\mu }^n=1$, hence $Z_n<\mathrm{\infty }$ ${\mu }^n$-almost everywhere and $P_0^n$-almost surely.

Fix $0<a<b$, choose measurable $B_a\subseteq \{\theta :D(P_0\mathrm{\parallel }{P}_{\theta })<a\}$ with $\mathrm{\Pi }(B_a)>0$, and let ${\mathrm{\Pi }}_a$ be the normalized restriction of $\mathrm{\Pi }$ to $B_a$. For each $\theta \in B_a$, finite KL gives $P_0\ll P_{\theta }$. Define $r_{\theta }=p_0\mathrm{/}p(\theta ,\cdot )$ where $p(\theta ,\cdot )>0$ and zero elsewhere. Since $P_0(\{p(\theta ,\cdot )=0\})=0$, direct integration of the two supplied densities gives ${\int }_A{r}_{\theta }d{P}_{\theta }=P_0(A)$ for every measurable $A$, so $r_{\theta }$ is a Radon–Nikodym derivative. Define $Z_{\theta }=\log (p(\theta ,\cdot )\mathrm{/}{p}_0)$ where both densities are positive and zero elsewhere; then $Z_{\theta }=-\log r_{\theta }$ $P_0$-almost surely. For $u\ge 1$, [F19] gives $\log u\le u$, so ${\mathbb{E}}_0{Z}_{\theta }^{+}\le {\int }_{\{p(\theta ,x)>p_0(x)\}}p(\theta ,x)d\mu \le 1$. Pinsker [F17] gives $D(P_0\mathrm{\parallel }{P}_{\theta })\ge 0$, and [F5] yields ${\mathbb{E}}_0{Z}_{\theta }^{-}=D(P_0\mathrm{\parallel }{P}_{\theta })+{\mathbb{E}}_0{Z}_{\theta }^{+}\le a+1$. Thus ${\mathbb{E}}_0\mathrm{\mid }{Z}_{\theta }\mathrm{\mid }\le a+2$.

Fix $\eta >0$ with $V_{\eta }\mathrm{\ne }\mathrm{\varnothing }$, and let ${\alpha }_n=P_0^n{\varphi }_n$ and ${\beta }_n={\sup }_{\theta \in V_{\eta }}{P}_{\theta }^n(1-{\varphi }_n)$. The two summable series imply ${\alpha }_n,{\beta }_n\to 0$. Choose $m\ge 1$ with ${\alpha }_m<1\mathrm{/}4$ and ${\beta }_m<1\mathrm{/}4$. For $\mathrm{\ell }=\lfloor n\mathrm{/}m\rfloor$, group the first $\mathrm{\ell }m$ observations into disjoint consecutive blocks of length $m$. By [F8, F10] the block vectors are independent and identically distributed under each fixed IID law; [F9] makes $Y_j:={\varphi }_m(\text{block }j)$ independent $[0,1]$ random variables. Their means are at most $1\mathrm{/}4$ under $P_0$ and at least $3\mathrm{/}4$ under every $P_{\theta }$ with $\theta \in V_{\eta }$. Let ${\tilde{\varphi }}_n=\mathbf{1}\{{\mathrm{\ell }}^{-1}{\sum }_{j=1}^{\mathrm{\ell }}{Y}_j>1\mathrm{/}2\}$ for $\mathrm{\ell }\ge 1$, and $0$ otherwise.

Since $P_0=P_{{\theta }_0}\ll \mu$, $p_0(X_i)>0$ for every sampled coordinate almost surely. On this event define the relative evidence $D_n=Z_n(X_{1:n})\mathrm{/}{\prod }_{i=1}^n{p}_0(X_i)$; it is finite $P_0^n$-almost surely by step 1.1.

Let $u(\theta ,x)=p(\theta ,x)\mathrm{/}{p}_0(x)$ when both densities are positive and $0$ otherwise; the lift of $p_0$ to $\mathrm{\Theta }\times \mathsf{X}$ is measurable by [F2,F20], and the piecewise quotient is measurable by [F23,F25]. Extend $Z(\theta ,x)=\log u(\theta ,x)$ on $u>0$ by $0$ on $u=0$. For every real $s$, its upper threshold is $\{u>\exp (s)\}$ if $s\ge 0$ and $\{u=0\}\cup \{u>\exp (s)\}$ if $s<0$; [F27,F29] and [F25] therefore make $Z$ product-measurable. For each $\theta \in B_a$, [F5] identifies $D(P_0\mathrm{\parallel }{P}_{\theta })=-{\mathbb{E}}_0{Z}_{\theta }$; applying [F6] to the positive and negative parts shows that these section integrals, and hence $\theta \mapsto D(P_0\mathrm{\parallel }{P}_{\theta })$ on $B_a$, are measurable. Tonelli [F6] and the uniform bound in step 1.2 show $G(x):=\int Z(\theta ,x){\mathrm{\Pi }}_a(d\theta )$ is finite for $P_0$-almost every $x$ and integrable, with ${\mathbb{E}}_{0}G=-\kappa$ where $\kappa :=\int D(P_0\mathrm{\parallel }{P}_{\theta }){\mathrm{\Pi }}_a(d\theta )\le a$. For each $\theta \in B_a$, $P_0\{x:p(\theta ,x)=0\}=0$; Tonelli [F6] applied to the measurable indicator of this set implies ${\mathrm{\Pi }}_a\{\theta :p(\theta ,X_i)=0\}=0$ almost surely for each coordinate $i$. Intersecting these probability-one events over all $i$, $\exp ({\sum }_{i=1}^{n}Z(\theta ,X_i))={\prod }_{i=1}^{n}p(\theta ,X_i)\mathrm{/}{p}_0(X_i)$ for ${\mathrm{\Pi }}_a$-almost every $\theta$ and every $n$. Set $G=0$ on the measurable $P_0$-null set where the integral is not finite.

For $\mathrm{\ell }\ge 1$, [F26] writes the exponential of the centered sum as a product; the finite product law [F10] and Tonelli [F6] factor its expectation, and [F12] bounds each factor by $e^{{\lambda }^2\mathrm{/}8}$. Thus $\mathbb{E}\exp \{\lambda {\sum }_{j=1}^{\mathrm{\ell }}(Y_j-\mathbb{E}{Y}_j)\}\le e^{\mathrm{\ell }{\lambda }^2\mathrm{/}8}$. Under $P_0$, rejection requires an upward deviation greater than $\mathrm{\ell }\mathrm{/}4$; Markov [F15] with $\lambda =1$ gives $P_0^n{\tilde{\varphi }}_n\le e^{-\mathrm{\ell }\mathrm{/}8}$. Under every $P_{\theta }$ in $V_{\eta }$, nonrejection requires a downward deviation at least $\mathrm{\ell }\mathrm{/}4$; Markov with $\lambda =-1$ gives $P_{\theta }^n(1-{\tilde{\varphi }}_n)\le e^{-\mathrm{\ell }\mathrm{/}8}$. Since $\mathrm{\ell }\ge n\mathrm{/}m-1$, both errors are at most $e^{1\mathrm{/}8}{e}^{-cn}$ for $c:=1\mathrm{/}(8m)$ and all $n\ge m$.

The variables $G(X_i)$ are IID by [F8, F9] and integrable by step 2.2. The strong law [F11] gives $n^{-1}{\sum }_{i=1}^{n}G(X_i)\to -\kappa$; because $\kappa \le a<b$, eventually ${\sum }_{i}G(X_i)>-nb$. Tonelli and the bound in step 1.2 also give $\int {\sum }_i\mathrm{\mid }{Z}_{\theta }(X_i)\mathrm{\mid }{\mathrm{\Pi }}_a(d\theta )<\mathrm{\infty }$ for $P_0^n$-almost every sample; the exponential is integrable by step 1.1. By step 2.2, the likelihood ratio in $D_n$ agrees with $\exp ({\sum }_i{Z}_{\theta }(X_i))$ for ${\mathrm{\Pi }}_a$-almost every $\theta$, so $D_n\ge \mathrm{\Pi }(B_a)\int \exp ({\sum }_i{Z}_{\theta }(X_i)){\mathrm{\Pi }}_a(d\theta )$. Jensen [F13, F14] bounds this integral below by $\exp ({\sum }_{i}G(X_i))$, giving $D_n\ge \mathrm{\Pi }(B_a)e^{-nb}$ eventually.

On $\{p_0(X_i)>0\text{ for }1\le i\le n\}$, define $A_n:=(1-{\tilde{\varphi }}_n){\int }_{V_{\eta }}{\prod }_{i=1}^n[p(\theta ,X_i)\mathrm{/}{p}_0(X_i)]\mathrm{\Pi }(d\theta )$, and set $A_n=0$ otherwise. The product-density identity in step 1.1 and Tonelli [F6] imply ${\mathbb{E}}_{P_0^n}{A}_n={\int }_{V_{\eta }}{P}_{\theta }^n((1-{\tilde{\varphi }}_n){\mathbf{1}}_{\{p_0(X_i)>0\mathrm{\forall }i\}})\mathrm{\Pi }(d\theta )\le e^{1\mathrm{/}8}{e}^{-cn}$ for $n\ge m$, by step 2.3 and $\mathrm{\Pi }(V_{\eta })\le 1$.

On valid posterior fibres, split the likelihood numerator over the test and its complement. The posterior formula [F7] gives ${\mathrm{\Pi }}_n(V_{\eta }\mid X_{1:n})\le {\tilde{\varphi }}_n+A_n\mathrm{/}{D}_n$. Step 3.1 ensures the denominator is finite and positive for all sufficiently large $n$ almost surely.

Choose $0<a<b<c$ and use step 3.1 with these radii. Since $c>0$ and $c-b>0$, [F27] gives $e^{-c},e^{-(c-b)}\in (0,1)$; [F28] makes the corresponding geometric series finite. For each $r>0$, Markov [F15] gives ${\sum }_{n\ge m}{P}_0^n(e^{nb}{A}_n>r)\le r^{-1}{e}^{1\mathrm{/}8}{\sum }_{n\ge m}{e}^{-n(c-b)}<\mathrm{\infty }$. Borel–Cantelli [F16], first for $r=1\mathrm{/}k$ and then by countability, gives $e^{nb}{A}_n\to 0$ almost surely. The type-I bound from step 2.3 is also summable, so [F16] gives ${\tilde{\varphi }}_n\to 0$ almost surely. Combining these limits with $D_n\ge \mathrm{\Pi }(B_a)e^{-nb}$ from step 3.1 and step 4.1 proves ${\mathrm{\Pi }}_n(V_{\eta }\mid X_{1:n})\to 0$ almost surely.

The consistency conclusion holds simultaneously for all $\eta >0$: intersect the probability-one events for positive rational $q$; for any $\eta >0$ choose rational $q<\eta$, so $V_{\eta }\subseteq V_q$.

For each $\theta$ with finite KL, Pinsker [F17] and the local comparison [F18] give $h(P_0,P_{\theta }{)}^2\le 2\sqrt{D(P_0\mathrm{\parallel }{P}_{\theta })\mathrm{/}2}$, hence $h(P_0,P_{\theta }{)}^4\le 2D(P_0\mathrm{\parallel }{P}_{\theta })$. The parameter map $\theta \mapsto h(P_0,P_{\theta }{)}^2$ is measurable: [F25] makes both square-root sections measurable, [F23] makes their squared difference product-measurable, and Tonelli [F6] makes its $\mu$-integral measurable in $\theta$. Thus every Hellinger ball is measurable. Taking a positive-measure KL subset $B_{{\eta }^4\mathrm{/}2}$ places it inside $\{\theta :h(P_0,P_{\theta })<\eta \}$, proving positive prior mass.