Hellinger stability and posterior discretization — Examples and Counterexamples

The uniform prior is the finite atomic measure ${\mu }_0(A):=\mathrm{\mid }A\mathrm{\mid }\mathrm{/}3$ for $A\subseteq D$; it is countably additive on this finite space and has ${\mu }_0(D)=1$, so [F4] makes it a probability measure. The masses of $p$ and $q$ are positive and each sums to one. By [F6], $D$ is Borel and both finite potentials are measurable. By [F3], $e^{-{\mathrm{\Phi }}_P(u)}=3p(u)$ and $e^{-{\mathrm{\Phi }}_Q(u)}=3q(u)$ for every $u\in D$. Thus $Z_{{\mathrm{\Phi }}_P}=\frac{1}{3}{\sum }_{u\in D}3p(u)=1$ and $Z_{{\mathrm{\Phi }}_Q}=\frac{1}{3}{\sum }_{u\in D}3q(u)=1$. By [F1], the two posterior updates have masses $p$ and $q$; they are proper and have finite means.

By [F5], their means are $(-1)(\epsilon \mathrm{/}2)+(1)(\epsilon \mathrm{/}2)=0$ and $(-1)((1-\epsilon )\mathrm{/}2)+(1)((1-\epsilon )\mathrm{/}2)=0$, respectively. Hence their posterior means agree.

Using ${\mu }_0$ as the common dominating measure, the corresponding densities are $3p$ and $3q$. By [F2], $h(P,Q{)}^2=2{(\sqrt{1\mathrm{/}200}-\sqrt{99\mathrm{/}200})}^2+{(\sqrt{99\mathrm{/}100}-1\mathrm{/}10)}^2\mathrm{.}$ Since $99\mathrm{/}100>(9\mathrm{/}10{)}^2$, the central term is greater than $(4\mathrm{/}5{)}^2=16\mathrm{/}25$. Since $99\mathrm{/}200>(7\mathrm{/}10{)}^2$ and $1\mathrm{/}200<(1\mathrm{/}14{)}^2$, each endpoint squared difference is greater than $(7\mathrm{/}10-1\mathrm{/}14{)}^2=(22\mathrm{/}35{)}^2$. Therefore $h(P,Q{)}^2>16\mathrm{/}25+2(22\mathrm{/}35{)}^2=1752\mathrm{/}1225>1$. Equal posterior means thus coexist with Hellinger distance greater than one.

For any $a\in \mathbb{R}$, the intervals $(x-\mathrm{\mid }x-a\mathrm{\mid }\mathrm{/}2,x+\mathrm{\mid }x-a\mathrm{\mid }\mathrm{/}2)$, indexed by $x\mathrm{\ne }a$, cover $\mathbb{R}\setminus \{a\}$ and omit $a$; hence $\mathbb{R}\setminus \{a\}$ is open and every singleton is closed. Therefore $D$ is Borel as a finite union of closed sets, and it is bounded because $\mathrm{\mid }u\mathrm{\mid }=1$ on $D$. The open intervals $(1\mathrm{/}2,3\mathrm{/}2)$ and $(-3\mathrm{/}2,-1\mathrm{/}2)$ intersect $D$ in $\{1\}$ and $\{-1\}$, respectively, so every singleton is relatively open and $D$ has the discrete relative topology. The displayed finite atomic measure is countably additive and its two atom masses sum to one, so it is a Borel probability with ${\mu }_0(D)=1$. Thus $D$ itself is a bounded Borel subset of positive prior mass, as required by the theorem. The real line is a separable Banach space. By [F4], for any open $O\subseteq \mathbb{R}$, ${\mathrm{\Phi }}^{-1}(O)$ intersects each open slice $\{u\}\times \mathbb{R}$ in an open subset of that slice, hence open in $D\times \mathbb{R}$; taking the union over $u\in D$ proves joint continuity. The potential is nonnegative. For $\mathrm{\mid }y\mathrm{\mid },\mathrm{\mid }{y}^{\mathrm{\prime }}\mathrm{\mid }\le R$ and $u\in D$, $\mathrm{\mid }\mathrm{\Phi }(u;y)-\mathrm{\Phi }(u;y^{\mathrm{\prime }})\mathrm{\mid }=\frac{1}{2}\mathrm{\mid }y-y^{\mathrm{\prime }}\mathrm{\mid }\mathrm{\mid }y+y^{\mathrm{\prime }}-2u\mathrm{\mid }\le (R+\mathrm{\mid }u\mathrm{\mid })\mathrm{\mid }y-y^{\mathrm{\prime }}\mathrm{\mid }\mathrm{.}$ Here $\mathrm{\mid }y+y^{\mathrm{\prime }}-2u\mathrm{\mid }\le \mathrm{\mid }y\mathrm{\mid }+\mathrm{\mid }{y}^{\mathrm{\prime }}\mathrm{\mid }+2\mathrm{\mid }u\mathrm{\mid }\le 2R+2\mathrm{\mid }u\mathrm{\mid }$ by the triangle inequality. Thus choose $M_1(r,s)=0$ and $M_2(r,s)=r+s$, which are Borel and nondecreasing in each argument. Then ${\int }_D{e}^{M_1(R,\mathrm{\mid }u\mathrm{\mid })}(1+M_2(R,\mathrm{\mid }u\mathrm{\mid }{)}^2)d{\mu }_0(u)=1+(R+1{)}^2<\mathrm{\infty },$ so [F2] applies and gives the stated Hellinger Lipschitz stability on $[-R,R]$.

By [F1], the evidence is the displayed two-term sum and is positive. The posterior mass ratio is $\frac{{\mathrm{\Pi }}^y(\{1\})}{{\mathrm{\Pi }}^y(\{-1\})}=\frac{e^{-(y-1{)}^2\mathrm{/}2}}{e^{-(y+1{)}^2\mathrm{/}2}}=e^{2y};$ the two masses sum to one, so ${\mathrm{\Pi }}^y(\{1\})=e^{2y}\mathrm{/}(1+e^{2y})$. The verified evidence and the explicit formula determine the whole two-point posterior.

By [F3], $0<{\sum }_{n\ge 1}{n}^{-2}<\mathrm{\infty }$, so $c$ is finite and positive and [F7] makes ${\mu }_0$ a probability with ${\mu }_0(D)=1$. By [F6], $D$ is Borel and has the discrete relative topology, so every real-valued function on $D$, including $\mathrm{\Phi },{\mathrm{\Phi }}_N$, is continuous. The real line with its usual norm is a separable Banach space by [F6]. Thus all $\mathrm{\Phi },{\mathrm{\Phi }}_N$ are finite continuous functions on $D$. They satisfy $\mathrm{\Phi },{\mathrm{\Phi }}_N\ge 0$ and $\mathrm{\mid }{\mathrm{\Phi }}_N(n)-\mathrm{\Phi }(n)\mathrm{\mid }=n\mathrm{/}N=M_2(\mathrm{\mid }n\mathrm{\mid }){\psi }_N$. But ${\int }_D{e}^{M_1(\mathrm{\mid }u\mathrm{\mid })}(1+M_2(\mathrm{\mid }u\mathrm{\mid }{)}^2){\mu }_0(du)={\sum }_{n\ge 1}\frac{c}{n^2}(1+n^2)=+\mathrm{\infty },$ since the second summand contributes $c$ for each $n$. This is precisely the integrability hypothesis used by [F4], and it fails here.

The base evidence is $Z=1$. For each $N$, the approximate evidence $Z_N={\sum }_{n\ge 1}\frac{c}{n^2}{e}^{-n\mathrm{/}N}$ is positive and at most one, so both posterior updates exist. Moreover $Z_N\to 1$: given $ϵ>0$, choose $m$ with ${\sum }_{n>m}c\mathrm{/}{n}^2<ϵ\mathrm{/}2$ by [F3]. The finite sum ${\sum }_{n=1}^m(c\mathrm{/}{n}^2)(1-e^{-n\mathrm{/}N})$ tends to zero by continuity of the exponential [F5], while the remaining tail of $1-Z_N$ is at most ${\sum }_{n>m}c\mathrm{/}{n}^2<ϵ\mathrm{/}2$.

Since $e^{-1}<1$, the number $\alpha :=(1+e^{-1})\mathrm{/}2$ is less than one. From $Z_N\to 1$, for all sufficiently large $N$ we have $Z_N\ge \alpha$. For $n\ge N$, $\frac{e^{-n\mathrm{/}N}}{Z_N}\le \frac{e^{-1}}{\alpha }=\frac{2}{e+1}<1,$ so $1-\sqrt{e^{-n\mathrm{/}N}\mathrm{/}{Z}_N}\ge \kappa :=1-\sqrt{2\mathrm{/}(e+1)}>0$.

The posterior density of the base law relative to ${\mu }_0$ is $1$, and that of the approximation is $e^{-n\mathrm{/}N}\mathrm{/}{Z}_N$. By [F2] and step 3.1, $h(\mathrm{\Pi },{\mathrm{\Pi }}_N{)}^2={\sum }_{n\ge 1}\frac{c}{n^2}{(1-\sqrt{\frac{e^{-n\mathrm{/}N}}{Z_N}})}^2\ge {\kappa }^2{\sum }_{n=N}^{2N-1}\frac{c}{n^2}\ge \frac{{\kappa }^{2}c}{4N},$ because the block has $N$ terms and each $n$ in it satisfies $n<2N$. Therefore $h(\mathrm{\Pi },{\mathrm{\Pi }}_N)\ge (\kappa \sqrt{c}\mathrm{/}2)N^{-1\mathrm{/}2}$, while ${\psi }_N=N^{-1}$. The ratio $h(\mathrm{\Pi },{\mathrm{\Pi }}_N)\mathrm{/}{\psi }_N$ is at least $(\kappa \sqrt{c}\mathrm{/}2)\sqrt{N}$ and is unbounded, so the claimed $O({\psi }_N)$ rate fails.