Posterior consistency at a KL projection

Let $M={\sup }_{n\ge 0}{S}_n$, a measurable $[0,+\mathrm{\infty }]$-valued random variable. For each $m\ge 1$, removing the first $m$ increments changes later partial sums only by the finite number $S_m$, and the removed partial sums form a finite set. Consequently $\{M=+\mathrm{\infty }\}=\{{\sup }_{k\ge 0}{\sum }_{i=m+1}^{m+k}{X}_i=+\mathrm{\infty }\}$ belongs to $\sigma (X_{m+1},X_{m+2},\dots )$. It is a tail event, so its probability is zero or one.

Suppose its probability were zero. Then $M$ is finite almost surely. Define $M^{\mathrm{\prime }}={\sup }_{k\ge 0}{\sum }_{i=2}^{k+1}{X}_i$, which is also finite almost surely and has the same law as $M$: for every real $b$, the event that the supremum is at most $b$ is the decreasing intersection of the corresponding finite-prefix events, whose probabilities agree by the IID hypothesis. The variable $M^{\mathrm{\prime }}$ is independent of $X_1$ by independence of their coordinate sigma-algebras. Decomposing the supremum according to whether the empty partial sum is used gives $M=\max (0,X_1+M^{\mathrm{\prime }})$ almost surely. On null exceptional sets replace $M$ and $M^{\mathrm{\prime }}$ by zero if needed, without changing their laws or these almost-sure assertions.

Put $A=\max (0,M^{\mathrm{\prime }}+X_1)$ and $B=M^{\mathrm{\prime }}$, so $A$ and $B$ are finite nonnegative variables with the same law. The map $u\mapsto \max (0,u)$ is one-Lipschitz and $B\ge 0$, hence $\mathrm{\mid }A-B\mathrm{\mid }\le \mathrm{\mid }{X}_1\mathrm{\mid }$. For each positive integer $K$, equality in law gives $\mathbb{E}[\min (A,K)-\min (B,K)]=0$. The difference converges almost surely to $A-B$ and has absolute value at most $\mathrm{\mid }A-B\mathrm{\mid }\le \mathrm{\mid }{X}_1\mathrm{\mid }$. Dominated convergence therefore yields $\mathbb{E}(A-B)=0$, without requiring either $A$ or $B$ to have finite expectation.

The identity $A-B=X_1+R$, where $R=(-B-X_1{)}^{+}$ and $u^{+}=\max (u,0)$, gives $0\le R\le \mathrm{\mid }{X}_1\mathrm{\mid }$. Centering and step 3.1 imply $\mathbb{E}R=0$, so $R=0$ almost surely. Thus $B+X_1=A$ almost surely and $B+X_1$ has the same law as $B$. Independence now gives ${\phi }_B(t){\phi }_{X_1}(t)={\phi }_B(t)$ for every real $t$.

Continuity and ${\phi }_B(0)=1$ supply $\delta >0$ such that ${\phi }_B(t)\mathrm{\ne }0$ for $\mathrm{\mid }t\mathrm{\mid }<\delta$. Step 4.1 then gives ${\phi }_{X_1}(t)=1$ on this interval. Choose a positive integer $q$ with $1\mathrm{/}q<\delta$. For every integer $k\ge q$, taking real parts gives $\mathbb{E}[1-\cos (X_1\mathrm{/}k)]=0$, so $\cos (X_1\mathrm{/}k)=1$ almost surely. Intersecting these countably many probability-one events, fix a resulting value $x$ of $X_1$. If $x\mathrm{\ne }0$, some $k\ge q$ has $0<\mathrm{\mid }x\mathrm{\mid }\mathrm{/}k<2\pi$, contradicting $\cos (x\mathrm{/}k)=1$. Therefore $X_1=0$ almost surely, contrary to the hypothesis.

The zero-probability alternative in step 1.1 is impossible, so $M=+\mathrm{\infty }$ almost surely. Each finite prefix has finite maximum, and hence removing it preserves an infinite supremum. Thus the limsup of the partial sums is also $+\mathrm{\infty }$ almost surely.

Fix $\theta$ and define $r_{\theta }=p_{\theta }\mathrm{/}{p}_{{\theta }^{\ast }}$ on $\{p_{{\theta }^{\ast }}>0\}$ and $r_{\theta }=0$ elsewhere. The supplied finite-valued nonnegative densities make $r_{\theta }$ nonnegative and finite everywhere. For every real $a\ge 0$, $\{r_{\theta }>a\}=\{p_{{\theta }^{\ast }}>0\}\cap \{p_{\theta }-a{p}_{{\theta }^{\ast }}>0\}\mathrm{.}$ The difference is an everywhere-defined measurable function by [F2], so this intersection is measurable. For $a<0$, the threshold set is $\mathsf{X}$. Thus [F5] proves measurability of $r_{\theta }$, and $Q_{\theta }(A)={\int }_A{r}_{\theta }d{P}_0$ is defined for every $A\in \mathcal{A}$.

By [F3], $Q_{\theta }$ is a measure. For each integer $k\ge 1$, set $E_k=\{x:r_{\theta }(x)\le k\}$. These sets are measurable by step 1.1 and increase to $\mathsf{X}$ because $r_{\theta }$ is finite everywhere. Moreover, $Q_{\theta }(E_k)={\int }_{E_k}{r}_{\theta }d{P}_0\le k{P}_0(E_k)\le k$ by [F4]. Thus $(E_k{)}_{k\ge 1}$ is a countable cover by finite-$Q_{\theta }$-measure sets, and $Q_{\theta }$ is sigma-finite. For the selected minimizer, $r_{{\theta }^{\ast }}=1_{\{p_{{\theta }^{\ast }}>0\}}$, so $Q_{{\theta }^{\ast }}(\mathsf{X})=P_0\{p_{{\theta }^{\ast }}>0\}=1$ by the assumed almost-sure positivity.

The jointly measurable finite-valued densities and [F12], [F13] make $\{(\theta ,x):p_{\theta }(x)=0\}$ measurable. The density formula gives $P_0\ll P_{\theta }$ exactly when $\int 1_{\{p_{\theta }=0\}}{p}_{0}d\mu =0$; Tonelli [F4] makes this condition measurable in $\theta$. On this measurable set, the extended log-density parts defining $K(\theta )$ are product-measurable and Tonelli makes their section integrals measurable; outside it the KL value is $+\mathrm{\infty }$ by [F2]. Thus $\theta \mapsto K(\theta )$ and $B_{ϵ}$ are measurable. By [F1], for $\mathrm{\Pi }$-almost every $\theta \in B_{ϵ}$, ${\mathbb{E}}_{P_0}\log \frac{p_{\theta }(X)}{p_{{\theta }^{\ast }}(X)}=-(K(\theta )-K^{\ast })\ge -ϵ\mathrm{.}$ Both KL values are finite on this set, so [F2] makes the log ratio integrable.

For each fixed $\theta$ in this full-prior subset of $B_{ϵ}$, $L_i^{\theta }:=\log (p_{\theta }(X_i)\mathrm{/}{p}_{{\theta }^{\ast }}(X_i))$ are IID integrable real variables: the data are IID and [F6] preserves independence under the measurable log-ratio map. By [F5], $\frac{1}{n}{\sum }_{i=1}^n{L}_i^{\theta }⟶-(K(\theta )-K^{\ast })\ge -ϵP_0^{\mathrm{\infty }}\text{-almost surely}\mathrm{.}$ The event of this convergence is jointly measurable in $(\theta ,X_1,X_2,\dots )$. Tonelli [F4] applied to the indicator of its failure, under $\mathrm{\Pi }{\mathrm{\mid }}_{B_{ϵ}}\times P_0^{\mathrm{\infty }}$, therefore gives a $P_0^{\mathrm{\infty }}$-full event on which convergence holds for $\mathrm{\Pi }$-almost every $\theta \in B_{ϵ}$.

For fixed $n$, the finite product measures ${\mu }^n$ and $P_0^n$ exist by [F17]. The nonnegative jointly measurable likelihood ${\prod }_{i=1}^n{p}_{\theta }(x_i)$ is product-measurable by [F15] and integrates to one over ${\mu }^n$ for every $\theta$, by [F16] and repeated integration. Tonelli [F4] therefore gives ${\int }_{{\mathsf{X}}^n}{\int }_{\mathrm{\Theta }}{\prod }_{i=1}^n{p}_{\theta }(x_i)\mathrm{\Pi }(d\theta ){\mu }^n(d{x}_{1:n})=1.$ Let $Z_n(x_{1:n})={\int }_{\mathrm{\Theta }}{\prod }_i{p}_{\theta }(x_i)\mathrm{\Pi }(d\theta )$. By [F11], $Z_n<\mathrm{\infty }$ ${\mu }^n$-almost everywhere. The sample vector has law $P_0^n$ by [F9]. Its product density relative to ${\mu }^n$ is ${\prod }_i{p}_0(x_i)$: repeated Tonelli gives this density's rectangle formula, and [F10] identifies the resulting measure with $P_0^n$. Therefore $Z_n<\mathrm{\infty }$ $P_0^n$-almost surely.

For $\mathrm{\Pi }$-almost every $\theta \in B_{ϵ}$, finite KL gives $P_0\ll P_{\theta }$ by [F2], so $p_{\theta }(X_i)>0$ almost surely for each coordinate. The coordinate products are measurable by [F15]; Tonelli [F4] applied to the indicator of a zero likelihood shows that, for $P_0^n$-almost every sample, $p_{\theta }(X_i)>0$ for every $i$ for $\mathrm{\Pi }$-almost every $\theta \in B_{ϵ}$. Since $\mathrm{\Pi }(B_{ϵ})>0$, $Z_n>0$. Finally, $p_{{\theta }^{\ast }}(X_i)$ is finite and positive $P_0$-almost surely, so the relative denominator $D_n$ is finite and positive $P_0^n$-almost surely.

Fix $\alpha >ϵ$ and choose ${\alpha }^{\mathrm{\prime }}$ with $ϵ<{\alpha }^{\mathrm{\prime }}<\alpha$. On the event from step 1.2, for $\mathrm{\Pi }$-almost every $\theta \in B_{ϵ}$ the log likelihood average is eventually greater than $-{\alpha }^{\mathrm{\prime }}$. Therefore $e^{n\alpha }{\prod }_{i=1}^n\frac{p_{\theta }(X_i)}{p_{{\theta }^{\ast }}(X_i)}\ge e^{n(\alpha -{\alpha }^{\mathrm{\prime }})}⟶+\mathrm{\infty }\mathrm{.}$ Fatou's lemma [F14] for the nonnegative integrands gives ${\mathrm{lim\hspace{0.17em}inf}}_{n\to \mathrm{\infty }}{e}^{n\alpha }{D}_n\ge {\int }_{B_{ϵ}}{\mathrm{lim\hspace{0.17em}inf}}_{n\to \mathrm{\infty }}{e}^{n\alpha }{\prod }_{i=1}^n\frac{p_{\theta }(X_i)}{p_{{\theta }^{\ast }}(X_i)}\mathrm{\Pi }(d\theta )=+\mathrm{\infty },$ since $\mathrm{\Pi }(B_{ϵ})>0$. Hence $D_n\ge e^{-n\alpha }$ eventually.

Steps 1.3–2.2 establish finite-positive evidence, and step 2.2 establishes the eventual exponential lower bound asserted in the Statement.

Fix $\theta$ and $n\ge 1$. By [F2] and [F3], the product measures $Q_{\theta }^n$ and $P_0^n$ exist and are sigma-finite. For $x_{1:n}=(x_1,\dots ,x_n)$, every coordinate map is measurable by [F7], so $f_n(x_{1:n}):={\prod }_{i=1}^n{r}_{\theta }(x_i)$ is nonnegative and measurable.

Define $R_n(A):={\int }_A{f}_{n}d{P}_0^n$ on the product sigma-algebra. By [F6], $R_n$ is a measure. For every measurable rectangle $A_1\times \cdots \times A_n$, repeated Tonelli integration [F5] gives $R_n(A_1\times \cdots \times A_n)={\prod }_{i=1}^n{\int }_{A_i}{r}_{\theta }d{P}_0={\prod }_{i=1}^n{Q}_{\theta }(A_i)=Q_{\theta }^n(A_1\times \cdots \times A_n),$ where the final equality is [F4]'s rectangle formula.

The measures $R_n$ and $Q_{\theta }^n$ agree on the generating pi-system of measurable rectangles by step 2.1. The product-measure theorem [F4] gives uniqueness for this rectangle formula, so $R_n=Q_{\theta }^n$ on the whole product sigma-algebra. By the definition of $R_n$, its density is $f_n={\prod }_{i=1}^n{r}_{\theta }(x_i)$, as claimed.

Fix ${\theta }_1\mathrm{\ne }{\theta }^{\ast }$ and put $r_1=p_{{\theta }_1}\mathrm{/}{p}_{{\theta }^{\ast }}$ on $\{p_{{\theta }^{\ast }}>0\}$, with value zero on its $P_0$-null complement. The finite likelihood-ratio mean makes $r_1$ integrable. For $r_1\ge 1$, [F11] gives $0\le \log r_1={\int }_1^{r_1}{u}^{-1}du\le r_1-1\le r_1$, so the positive part is integrable. The log ratio is measurable by [F14]. Also ${\mathbb{E}}_{P_0}\log r_1=-(K({\theta }_1)-K^{\ast })<0,$ by the extended relative-contrast hypothesis, with value $-\mathrm{\infty }$ allowed.

For $t\downarrow 0$, let $A(t)={\mathbb{E}}_{P_0}{r}_1^t$. Since $r_1^t\le 1+r_1$ for $0<t\le 1$, [F8] gives $A(t)\to P_0(r_1>0)\le 1$. If this limit is less than one, then $A(t)<1$ for all sufficiently small $t>0$. If the limit equals one, split into $\{r_1\ge 1\}$ and $\{0<r_1<1\}$. On the first set, $(r_1^t-1)\mathrm{/}t\to \log r_1$ and, for fixed $t_0<1$ and $0<t\le t_0$, the mean-value theorem bounds this quotient above by $r_1^{t_0}\log r_1\le C{r}_1$ for a finite constant $C$: the function $r^{t_0-1}\log r$ is continuous on $[1,\mathrm{\infty })$ and tends to zero at infinity. On the second set, write $y=-\log r_1>0$. Then $(1-r_1^t)\mathrm{/}t=y(1-e^{-yt})\mathrm{/}(yt)$. The function $u\mapsto (1-e^{-u})\mathrm{/}u$ decreases on $u>0$, since its derivative is $((u+1)e^{-u}-1)\mathrm{/}{u}^2\le 0$ by $e^u\ge 1+u$: for $u\ge 0$, $h(u)=\exp (u)-1-u$ has $h(0)=0$ by the exponential definition and $h^{\mathrm{\prime }}(u)=\exp (u)-1\ge 0$ by the derivative formula and exponential monotonicity, the mean value theorem then gives $h(u)\ge 0$. Hence the quotient increases to $y$ as $t\downarrow 0$. Dominated convergence on the first set and monotone convergence [F9] on the second show $\frac{A(t)-1}{t}⟶{\mathbb{E}}_{P_0}\log r_1<0$ in the extended sense. Thus $A(t)<1$ for all sufficiently small $t>0$.

Choose such a $t$ with $s:=1-t$ in the left-neighborhood from the fractional-moment hypothesis. Then $M_s({\theta }_1,{\theta }_1)=A(t)<1$. By continuity at ${\theta }_1$, there are an open neighborhood $G_{{\theta }_1}$ and $0<a_{{\theta }_1}<1$ such that ${\sup }_{\theta \in G_{{\theta }_1}}{M}_s({\theta }_1,\theta )\le a_{{\theta }_1}$.

Define ${\varphi }_{n,{\theta }_1}=\mathbf{1}\{{\prod }_{i=1}^n{r}_1(X_i)>1\}\mathrm{.}$ The score and test are measurable by [F10], [F14], and [F15]. On the rejection event the product raised to $t$ exceeds one. Repeated Tonelli integration under $P_0^n$ therefore gives $P_0^n{\varphi }_{n,{\theta }_1}\le {\left({\mathbb{E}}_{P_0}{r}_1^t\right)}^n=M_s({\theta }_1,{\theta }_1{)}^n\le a_{{\theta }_1}^n\mathrm{.}$ For the second error, let $g_{\theta ,{\theta }_1,s}$ be the integrand defining $M_s({\theta }_1,\theta )$, with the zero and infinite conventions in the Statement. On the $P_0$-full set where $p_{{\theta }^{\ast }}>0$, if $p_{\theta }>0$ and $p_{{\theta }_1}>0$ then algebra gives $g_{\theta ,{\theta }_1,s}=r_{\theta }{r}_1^{-s}$; the displayed conventions extend this identity to the boundary values. On nonrejection, ${\prod }_i{r}_1(X_i)\le 1$. If every factor is positive, this gives ${\prod }_i{r}_{\theta }(X_i)\le {\prod }_i(r_{\theta }(X_i)r_1(X_i{)}^{-s})={\prod }_i{g}_{\theta ,{\theta }_1,s}(X_i)\mathrm{.}$ If some $r_{\theta }(X_i)=0$, both the density product and the right-hand product are zero under $0\cdot \mathrm{\infty }=0$; if some $r_1(X_i)=0$ while every $r_{\theta }(X_i)>0$, the right-hand product is infinite and the inequality still holds. The product-density identity [F6] and repeated Tonelli [F7] now give, for $\theta \in G_{{\theta }_1}$, $Q_{\theta }^n(1-{\varphi }_{n,{\theta }_1})\le {({\mathbb{E}}_{P_0}{g}_{\theta ,{\theta }_1,s}(X))}^n=M_s({\theta }_1,\theta {)}^n\le a_{{\theta }_1}^n\mathrm{.}$ The moment hypothesis makes these integrals finite on the chosen neighborhood.

Put $C_{\eta }=\{\theta :\rho (\theta ,{\theta }^{\ast })\ge \eta \}$. If $\rho (\theta ,{\theta }^{\ast })<\eta$, then every ${\theta }^{\mathrm{\prime }}$ with $\rho ({\theta }^{\mathrm{\prime }},\theta )<\eta -\rho (\theta ,{\theta }^{\ast })$ satisfies $\rho ({\theta }^{\mathrm{\prime }},{\theta }^{\ast })<\eta$ by [F13]. Thus the complement of $C_{\eta }$ is open, so $C_{\eta }$ is closed by [F12]. By [F5], it is compact. For every point of $C_{\eta }$, step 3.1 supplies a neighborhood; [F4] gives a finite subcover $G_{{\theta }_1},\dots ,G_{{\theta }_k}$.

Set ${\varphi }_n={\max }_{1\le j\le k}{\varphi }_{n,{\theta }_j}$. The union bound and step 4.1 give $P_0^n{\varphi }_n\le {\sum }_{j=1}^k{a}_{{\theta }_j}^n,{\sup }_{\theta \in C_{\eta }}{Q}_{\theta }^n(1-{\varphi }_n)\le {\max }_{1\le j\le k}{a}_{{\theta }_j}^n\mathrm{.}$ Since the subcover is finite and each $a_{{\theta }_j}<1$, both bounds are at most $C_{\eta }{e}^{-c_{\eta }n}$ for suitable $c_{\eta },C_{\eta }>0$ and all sufficiently large $n$. The set $\{\rho (\theta ,{\theta }^{\ast })>\eta \}$ is contained in $C_{\eta }$, so the same type-II bound holds there. If $C_{\eta }=\mathrm{\varnothing }$, the zero test has both errors zero.

Choose $\alpha$ with $ϵ<\alpha <\beta$. By [A1] and [F1], the denominator $D_n$ is at least $e^{-n\alpha }$ eventually almost surely. By the type-I premise, ${\varphi }_n(X_1,\dots ,X_n)\to 0$ almost surely under the same infinite IID law. Intersect these two full-probability events.

Define $N_n:={\int }_V{\prod }_{i=1}^n\frac{p_{\theta }(X_i)}{p_{{\theta }^{\ast }}(X_i)}(1-{\varphi }_n)\mathrm{\Pi }(d\theta ),Y_n:=e^{n\alpha }{N}_n\mathrm{.}$ By [F2], [F4], [F8], and the type-II bound, ${\mathbb{E}}_{P_0^n}{Y}_n\le e^{n\alpha }\mathrm{\Pi }(V)e^{-n\beta }\le e^{-n(\beta -\alpha )}\mathrm{.}$

Set $a_n=e^{-n(\beta -\alpha )\mathrm{/}2}$. For every sufficiently large $n$, Markov's inequality [F5] gives $P_0^n\{Y_n>a_n\}\le e^{-n(\beta -\alpha )\mathrm{/}2}$. The right-hand side is summable, and the finite initial terms do not matter. Thus [F6] and [F7] imply $Y_n\to 0$ almost surely.

On the full-probability event from the denominator lemma, the ordinary evidence is $Z_n:={\int }_{\mathrm{\Theta }}{\prod }_i{p}_{\theta }(X_i)\mathrm{\Pi }(d\theta )=({\prod }_i{p}_{{\theta }^{\ast }}(X_i))D_n,$ which is finite and positive. Bayes' formula [F3] therefore gives the posterior as the normalized likelihood integral, and division by this positive common factor expresses it using $D_n$. Decompose the tested likelihood integral over $V$ into its ${\varphi }_n$ and $1-{\varphi }_n$ parts. Since $0\le {\varphi }_n\le 1$, ${\int }_V{R}_n(\theta )\mathrm{\Pi }(d\theta )={\int }_V{\varphi }_n{R}_n(\theta )\mathrm{\Pi }(d\theta )+N_n\le {\varphi }_n{\int }_{\mathrm{\Theta }}{R}_n(\theta )\mathrm{\Pi }(d\theta )+N_n={\varphi }_n{D}_n+N_n,$ where $R_n(\theta )={\prod }_i{p}_{\theta }(X_i)\mathrm{/}{p}_{{\theta }^{\ast }}(X_i)$. Therefore $\mathrm{\Pi }(V\mid X_1,\dots ,X_n)\le {\varphi }_n+\frac{N_n}{D_n}\le {\varphi }_n+e^{n\alpha }{N}_n={\varphi }_n+Y_n\mathrm{.}$ Steps 1.1 and 2.1 make the right-hand side tend to zero.