Posterior consistency at a KL projection — Examples

For ${\delta }_j=1\mathrm{/}(j+4)$, direct substitution gives $\mathrm{K}\mathrm{L}(P_0\mathrm{\parallel }{P}_j)=\frac{1}{2}\log \frac{1\mathrm{/}2}{1\mathrm{/}2+{\delta }_j}+\frac{1}{2}\log \frac{1\mathrm{/}2}{1\mathrm{/}2-{\delta }_j}=-\frac{1}{2}\log (1-4{\delta }_j^2)>0.$ Thus ${\mathrm{\Theta }}^{\ast }=\{0\}$. Every relative KL neighborhood contains $0$ and therefore has prior mass at least $1\mathrm{/}2$, while every $j\ge 1$ lies in the distance-one alternative set.

Fix $n$ and choose $j_n\ge n^2$. For one coordinate define a joint law of $(X,Y)$ by masses $1\mathrm{/}2-{\delta }_{j_n}$ at $(0,0)$, $1\mathrm{/}2$ at $(1,1)$, and ${\delta }_{j_n}$ at $(0,1)$, with zero mass at $(1,0)$. Its marginals are $P_0$ and $P_{j_n}$, and its disagreement probability is ${\delta }_{j_n}$. Take the $n$-fold product of this finite joint law. By the union bound, the coupled vectors differ with probability at most $n{\delta }_{j_n}\le 1\mathrm{/}n$. For every event $A$, the indicators of $\{X_{1:n}\in A\}$ and $\{Y_{1:n}\in A\}$ can differ only when the vectors differ, so by [F2], $\mathrm{\parallel }{P}_0^n-P_{j_n}^n{\mathrm{\parallel }}_{\mathrm{T}\mathrm{V}}\le 1\mathrm{/}n\mathrm{.}$

For probability laws $P,Q$ on the finite set $\{0,1{\}}^n$, $\mathrm{\parallel }P-Q{\mathrm{\parallel }}_{\mathrm{T}\mathrm{V}}={\sum }_{z:P(z)>Q(z)}(P(z)-Q(z))$: the set of points with positive difference attains the supremum over events, and no event can sum more than all positive differences. Hence for every test ${\varphi }_n$ with $0\le {\varphi }_n\le 1$, $P{\varphi }_n-Q{\varphi }_n={\sum }_z{\varphi }_n(z)(P(z)-Q(z))\le {\sum }_{z:P(z)>Q(z)}(P(z)-Q(z))=\mathrm{\parallel }P-Q{\mathrm{\parallel }}_{\mathrm{T}\mathrm{V}}\mathrm{.}$ Taking $P=P_0^n$, $Q=P_{j_n}^n$ and rearranging gives $P_0^n{\varphi }_n+P_{j_n}^n(1-{\varphi }_n)\ge 1-\mathrm{\parallel }{P}_0^n-P_{j_n}^n{\mathrm{\parallel }}_{\mathrm{T}\mathrm{V}}\ge 1-\frac{1}{n}\mathrm{.}$ If $P_0^n{\varphi }_n\to 0$, then ${\sup }_{j\ge 1}{P}_j^n(1-{\varphi }_n)\ge P_{j_n}^n(1-{\varphi }_n)\to 1$. Since the selected minimizer is the true law, $Q_j=P_j$ here. Thus no test sequence has both error limits required by the KL-projection theorem.

Since $p_a=p_b$, their KL losses are equal. Direct calculation gives $K(c)-K(a)=\frac{3}{5}\log \frac{1\mathrm{/}2}{1\mathrm{/}5}+\frac{2}{5}\log \frac{1\mathrm{/}2}{4\mathrm{/}5}=\frac{3}{5}\log \frac{5}{2}+\frac{2}{5}\log \frac{5}{8}=\frac{1}{5}\log \frac{3125}{512}>0.$ Thus the minimizer set is ${\mathrm{\Theta }}^{\ast }=\{a,b\}$.

The likelihoods at $a$ and $b$ agree for every data vector, so their posterior odds remain $\mathrm{\Pi }(a)\mathrm{/}\mathrm{\Pi }(b)=1$. The posterior odds of $c$ against $a$ have log equal to $\log (\mathrm{\Pi }(c)\mathrm{/}\mathrm{\Pi }(a))+{\sum }_{i=1}^n\log (p_c(X_i)\mathrm{/}{p}_a(X_i))$. By [F2], the sum divided by $n$ tends almost surely to $-(K(c)-K(a))<0$. Hence the posterior mass of $c$, the complement of ${\mathrm{\Theta }}^{\ast }$, tends to zero while the posterior never selects one of $a,b$ over the other.

The relative KL contrast is $K(b)-K(a)={\mathbb{E}}_{P_0}\log \frac{p_a(X)}{p_b(X)}=\frac{2}{5}\log \frac{5}{8}+\frac{3}{5}\log \frac{5}{2}=\frac{1}{5}\log \frac{3125}{512}>0.$ Thus $a$ is the unique KL minimizer.

The posterior odds are $\frac{\mathrm{\Pi }(b\mid X_{1:n})}{\mathrm{\Pi }(a\mid X_{1:n})}={\prod }_{i=1}^n\frac{p_b(X_i)}{p_a(X_i)}\mathrm{.}$ The log factors are bounded and IID, with expectation $-(K(b)-K(a))<0$. By [F3], their sample mean converges almost surely to that negative number, so the odds tend to zero and $\mathrm{\Pi }(b\mid X_{1:n})\to 0$.

The tilted measure has total mass $Q_b(\mathsf{X})={\mathbb{E}}_{P_0}\frac{p_b(X)}{p_a(X)}=\frac{2}{5}\cdot \frac{8}{5}+\frac{3}{5}\cdot \frac{2}{5}=\frac{22}{25}<1.$ This example has a unique KL projection and a finite likelihood tilt that is not a probability law.

The KL loss at $a$ is $K(a)=\frac{1}{10}\log \frac{1}{9}+\frac{9}{10}\log 9=\frac{4}{5}\log 9<\mathrm{\infty }\mathrm{.}$ At $b$, $P_0$ is not absolutely continuous with respect to $P_b$, so $K(b)=+\mathrm{\infty }$. Thus $a$ is the unique minimizer and every KL neighborhood has prior mass at least $1\mathrm{/}2$.

The likelihood tilt at $b$ assigns mass $Q_b(\{0\})=\frac{1}{10}\frac{1}{9\mathrm{/}10}=\frac{1}{9},Q_b(\{1\})=\frac{9}{10}\frac{0}{1\mathrm{/}10}=0.$ Hence $Q_b(\mathsf{X})=1\mathrm{/}9<1$, so $Q_b$ is a finite nonprobability measure.

The density $p_a$ is positive at both sample points, so $K(a)=\mathrm{K}\mathrm{L}(P_0\mathrm{\parallel }{P}_a)<\mathrm{\infty }$. At $b$, $p_b(1)=0$ while $p_0(1)>0$, so $P_0\ll ̸P_b$ and $K(b)=+\mathrm{\infty }$ by [F1]. Therefore $a$ is the unique minimizer.

Since $p_b(0)\mathrm{/}{p}_a(0)={10}^6$ and $p_b(1)\mathrm{/}{p}_a(1)=0$, $Q_b(\mathsf{X})=\frac{1}{1000}\cdot {10}^6+\frac{999}{1000}\cdot 0=1000>1.$ The tilt is a finite nonprobability measure by [F2] and [F3].

Let $c>0$ normalize the density $f(y)=c\mathrm{/}[y^2(\log y{)}^2]$ on $[e,\mathrm{\infty })$. Its normalizing integral is finite and positive, and $\mu :=E_{0}Y=c{\int }_e^{\mathrm{\infty }}[y(\log y{)}^2{]}^{-1}dy=c<\mathrm{\infty }$. For IID $Y_i$ with this density put $X_i=Y_i-\mu$ and $S_n={\sum }_{i=1}^n{X}_i$. These increments are integrable, nonconstant and centered.

For $b\ge e$, direct integration gives $E_0[Y{1}_{\{Y>b\}}]=c\mathrm{/}\log b$ and $P_0(Y>b)\le c\mathrm{/}[b(\log b{)}^2]$. Moreover $E_0[Y^2{1}_{\{Y\le b\}}]=c{\int }_e^b(\log y{)}^{-2}dy=O(b\mathrm{/}(\log b{)}^2)$: split the integral at $\sqrt{b}$, bound the first part by $\sqrt{b}$, and the second by $4b\mathrm{/}(\log b{)}^2$. Put $b_n=n\mathrm{/}(\log n{)}^{3\mathrm{/}2}$ for all sufficiently large $n$. By [F6], the probability that any of the first $n$ observations exceeds $b_n$ is $O((\log n{)}^{-1\mathrm{/}2})\to 0$. The variance of ${\sum }_{i=1}^n{Y}_i{1}_{\{Y_i\le b_n\}}$ is at most $O(n{b}_n\mathrm{/}(\log b_n{)}^2)$ by independence. Dividing this bound by $(n\mathrm{/}\log n{)}^2$ gives $O(b_n\mathrm{/}n)\to 0$. Chebyshev [F4] therefore shows $S_n\mathrm{/}(n\mathrm{/}\log n)=-c\log n\mathrm{/}\log b_n+o_{P_0}(1)\to -c$. Define $a_n=cn\mathrm{/}(2\log n)$ and ${\varphi }_n=1_{\{S_n>-a_n\}}$ for $n\ge 3$, with arbitrary tests for $n<3$. It follows that $P_0^n{\varphi }_n\to 0$.

By [F3], the hitting times $T_j=\inf \{n\ge 1:S_n\ge j\}$ are finite almost surely for every positive integer $j$. Define recursively $N_0=0$ and let $N_j$ be the least integer at least $\max (j,N_{j-1}+1)$ with $P_0(T_j>N_j)\le 2^{-j}$. Such an integer exists since $T_j<\mathrm{\infty }$ almost surely. Put ${ϵ}_j=N_j^{-1}$. Thus ${ϵ}_j>0$ and ${ϵ}_j\to 0$; these constants depend only on the true law, not the realized observations. By [F5], $T_j\le N_j$ eventually almost surely. Also $T_j\to \mathrm{\infty }$ almost surely, since finitely many finite partial sums cannot exceed arbitrarily large levels.

Take sample space $\mathsf{X}=[e,\mathrm{\infty })\bigsqcup \{z\}$ with its disjoint-union Borel sigma-algebra and dominating measure equal to Lebesgue measure plus ${\delta }_z$. Set $p_0(y)=f(y)$ and $p_0(z)=0$. On the continuous part supply $p_{\ast }(y)=f(y)e^{-2y}$ and $p_j(y)=f(y)e^{-y-\mu -{ϵ}_j}$; give each density its remaining mass at $z$. Both continuous masses are strictly between zero and one, so these atom masses are positive, and every density is finite and normalized. Take $\mathrm{\Theta }=\{\ast \}\bigsqcup \{1,2,\dots \}$ with the discrete metric and its Borel sigma-algebra; the supplied density family is jointly measurable. By [F1], $K(\ast )=2\mu <\mathrm{\infty }$ and $K(j)=E_0(Y+\mu +{ϵ}_j)=2\mu +{ϵ}_j$, so ${\mathrm{\Theta }}^{\ast }=\{\ast \}$. All relevant log ratios are integrable because $E_{0}Y<\mathrm{\infty }$.

On the $P_0$-support, the tilt at $j$ has density $r_j(y)=e^{y-\mu -{ϵ}_j}=e^{X-{ϵ}_j}$. Its total mass is infinite, since ${\int }_e^{\mathrm{\infty }}{e}^{y}f(y)dy=\mathrm{\infty }$, but the finite pointwise density makes it sigma-finite by [F2]. Its product density is $e^{S_n-n{ϵ}_j}$. On $\{{\varphi }_n=0\}$ this is at most $e^{-a_n}$, uniformly in $j$, and hence ${\sup }_j{Q}_j^n(1-{\varphi }_n)\le e^{-a_n}\to 0$. For $0<\eta <1$, the set at distance greater than $\eta$ from ${\mathrm{\Theta }}^{\ast }$ is precisely $\{1,2,\dots \}$, so these same tests satisfy both required weak errors. For $\eta \ge 1$ that set is empty and the type-II requirement is vacuous.

Put $r=e^{-1\mathrm{/}4}$ and give the model prior ${\pi }_{\ast }=1\mathrm{/}2$, ${\pi }_j=(1-r)r^{j-1}\mathrm{/}2$. These masses sum to one. Every relative KL neighborhood has mass at least $1\mathrm{/}2$ because it contains $\ast$. On any observed continuous sample, divide the prior-weighted likelihood by ${\prod }_i{p}_{\ast }(Y_i)$ to obtain the alternative numerator $A_n=e^{S_n}{\sum }_{j\ge 1}{\pi }_j{e}^{-n{ϵ}_j}$. It is finite, being at most $e^{S_n}\mathrm{/}2$, and the total relative evidence is $1\mathrm{/}2+A_n>0$. Thus the posterior mass of the alternatives is $A_n\mathrm{/}(1\mathrm{/}2+A_n)$.

Almost surely, for every sufficiently large $j$, step 2.2 gives $T_j{ϵ}_j\le 1$ and $S_{T_j}\ge j$. Consequently $A_{T_j}\ge e^{j-1}{\pi }_j=(1-r)e^{j-1}{r}^{j-1}\mathrm{/}2\to \mathrm{\infty }$. Since $T_j\to \mathrm{\infty }$, the posterior mass outside every discrete neighborhood of radius less than one has limes superior one almost surely. This contradicts the claimed a.s. consistency although both test errors in the refuted statement tend to zero.