Hellinger geometry and posterior consistency

By [A1], [F3], and [F4], take finite-valued nonnegative densities $p=dP\mathrm{/}d\mu$ and $q=dQ\mathrm{/}d\mu$ for a common sigma-finite dominator $\mu$. Define $\rho (A):=P(A)+Q(A)$. For disjoint measurable $A_k$, countable additivity of $P,Q$ and regrouping nonnegative series give $\rho ({\bigcup }_k{A}_k)={\sum }_k\rho (A_k)$; also $\rho (\mathrm{\varnothing })=0$ and $\rho (\mathsf{X})=2$. Thus $\rho$ is a finite, hence sigma-finite, measure by [F6]. Additivity of the nonnegative integral [F10] gives $\rho (A)={\int }_A(p+q)d\mu$, with $p+q$ measurable by [F7].

Using $p,q$ against $\mu$, the set $B:=\{p\ge q\}$ is measurable by [F7,F8]. On $B$, $p=(p-q)+q$, so additivity [F10] gives $\delta :=P(B)-Q(B)={\int }_B(p-q)d\mu \ge 0$. On $B^c$, $q=(q-p)+p$ and the total masses are one, hence ${\int }_{B^c}(q-p)d\mu =Q(B^c)-P(B^c)=\delta$. Additivity then gives $\int \mathrm{\mid }p-q\mathrm{\mid }d\mu =2\delta$. For any measurable $A$, the difference on $A\cap B$ lies in $[0,\delta ]$ by monotonicity [F10], while the difference on $A\cap B^c$ lies in $[-\delta ,0]$; thus $\mathrm{\mid }P(A)-Q(A)\mathrm{\mid }\le \delta$, and $A=B$ attains $\delta$. Therefore [F2] gives $\parallel P-Q{\parallel }_{\mathrm{T}\mathrm{V}}=\delta =\frac{1}{2}\int \mathrm{\mid }p-q\mathrm{\mid }d\mu$.

On $\{p+q>0\}$ define $\bar{p}=p\mathrm{/}(p+q)$ and $\bar{q}=q\mathrm{/}(p+q)$, and set both to zero on $\{p+q=0\}$. For $r<0$, $\{\bar{p}>r\}$ is the whole space; for $r\ge 0$, it is $\{p+q>0\}\cap \{p-r(p+q)>0\}$. Thus [F7,F8] make $\bar{p}$ measurable, and the same argument applies to $\bar{q}$. Since $p=q=0$ where $p+q=0$, [F9] gives ${\int }_A\bar{p}d\rho ={\int }_A\bar{p}(p+q)d\mu =P(A)$ and ${\int }_A\bar{q}d\rho =Q(A)$ for every measurable $A$. Hence $\bar{p},\bar{q}$ are finite-valued density representatives of $P,Q$ against $\rho$.

Pointwise $\mathrm{\mid }p-q\mathrm{\mid }=\mathrm{\mid }\sqrt{p}-\sqrt{q}\mathrm{\mid }(\sqrt{p}+\sqrt{q})$. Hölder [F5] therefore gives $\int \mathrm{\mid }p-q\mathrm{\mid }d\mu \le h(P,Q)(\int (\sqrt{p}+\sqrt{q}{)}^{2}d\mu {)}^{1\mathrm{/}2}$. The squared second factor is $2+2\int \sqrt{pq}d\mu \le 4$, since [F5] applied to $\sqrt{p},\sqrt{q}$ bounds their product integral by $1$. Step 1.2 now gives $\parallel P-Q{\parallel }_{\mathrm{T}\mathrm{V}}\le h(P,Q)$.

Write $H_{\lambda }(P,Q{)}^2$ for the defining Hellinger integral computed from supplied densities against a common dominator $\lambda$. Applying [F1] and [F9] to the densities in step 2.1 gives $H_{\rho }(P,Q{)}^2=\int (\sqrt{\bar{p}}-\sqrt{\bar{q}}{)}^{2}d\rho ={\int }_{\{p+q>0\}}(\sqrt{p}-\sqrt{q}{)}^{2}d\mu =H_{\mu }(P,Q{)}^2$, because both original densities vanish where $p+q=0$. For any measure $\alpha$ and finite-valued nonnegative densities $f,g$ of the same finite measure $\mathrm{\Lambda }$ with respect to $\alpha$, they agree $\alpha$-almost everywhere: on $E_m:=\{f\ge g+1\mathrm{/}m\}$, equality of their density integrals gives $0={\int }_{E_m}(f-g)d\alpha \ge \alpha (E_m)\mathrm{/}m$, so $\alpha (E_m)=0$. Since $\{f>g\}={\bigcup }_{m\ge 1}{E}_m$, and the same argument with $f,g$ swapped handles $\{g>f\}$, this proves uniqueness almost everywhere. Thus changing supplied representatives against a fixed dominator does not change the Hellinger integral.

For any other common sigma-finite dominator $\nu$, repeat steps 1.1 and 2.1 using its densities to obtain density representatives ${\bar{p}}_{\nu },{\bar{q}}_{\nu }$ against the same finite measure $\rho =P+Q$. The uniqueness argument in step 3.1 gives ${\bar{p}}_{\nu }=\bar{p}$ and ${\bar{q}}_{\nu }=\bar{q}$ $\rho$-almost everywhere. The same density-change calculation as in step 3.1 therefore identifies the Hellinger integral against $\nu$ with the same canonical value against $\rho$, and hence with that against $\mu$.

Pointwise $(\sqrt{p}-\sqrt{q}{)}^2\le \mathrm{\mid }p-q\mathrm{\mid }$. Integrating by monotonicity [F10] and using [F1] and step 1.2 gives $h(P,Q{)}^2\le \int \mathrm{\mid }p-q\mathrm{\mid }d\mu =2\parallel P-Q{\parallel }_{\mathrm{T}\mathrm{V}}$.

For $n\ge 1$ put $L_n(\theta ,x_{1:n})={\prod }_{i=1}^{n}p(\theta ,x_i)$ and $Z_n(x_{1:n})={\int }_{\mathrm{\Theta }}{L}_n(\theta ,x_{1:n})\mathrm{\Pi }(d\theta )$. The coordinate maps into $\mathrm{\Theta }\times \mathsf{X}$ are measurable by the product sigma-algebra definition [F20]; joint measurability of $p$ and closure under finite products [F23] make $L_n$ product-measurable. For each fixed $\theta$, repeated Tonelli [F6, F21] and $\int p(\theta ,x)d\mu =1$ give $\int L_n(\theta ,\cdot )d{\mu }^n=1$. By [F24], $L_n(\theta ,\cdot )d{\mu }^n$ is a measure. On every measurable rectangle ${\prod }_{i=1}^n{A}_i$, repeated Tonelli and the density identity give mass ${\prod }_i{P}_{\theta }(A_i)$; this is also the rectangle mass of $P_{\theta }^n$ by [F8, F10]. Rectangles generate the product sigma-algebra [F20], and both are probability measures, so [F22], using the whole sample space as a finite-measure exhaustion, identifies these measures. Tonelli over $\mathrm{\Pi }$ and ${\mu }^n$ now gives $\int Z_{n}d{\mu }^n=1$, hence $Z_n<\mathrm{\infty }$ ${\mu }^n$-almost everywhere and $P_0^n$-almost surely.

Fix $0<a<b$, choose measurable $B_a\subseteq \{\theta :D(P_0\mathrm{\parallel }{P}_{\theta })<a\}$ with $\mathrm{\Pi }(B_a)>0$, and let ${\mathrm{\Pi }}_a$ be the normalized restriction of $\mathrm{\Pi }$ to $B_a$. For each $\theta \in B_a$, finite KL gives $P_0\ll P_{\theta }$. Define $r_{\theta }=p_0\mathrm{/}p(\theta ,\cdot )$ where $p(\theta ,\cdot )>0$ and zero elsewhere. Since $P_0(\{p(\theta ,\cdot )=0\})=0$, direct integration of the two supplied densities gives ${\int }_A{r}_{\theta }d{P}_{\theta }=P_0(A)$ for every measurable $A$, so $r_{\theta }$ is a Radon–Nikodym derivative. Define $Z_{\theta }=\log (p(\theta ,\cdot )\mathrm{/}{p}_0)$ where both densities are positive and zero elsewhere; then $Z_{\theta }=-\log r_{\theta }$ $P_0$-almost surely. For $u\ge 1$, [F19] gives $\log u\le u$, so ${\mathbb{E}}_0{Z}_{\theta }^{+}\le {\int }_{\{p(\theta ,x)>p_0(x)\}}p(\theta ,x)d\mu \le 1$. Pinsker [F17] gives $D(P_0\mathrm{\parallel }{P}_{\theta })\ge 0$, and [F5] yields ${\mathbb{E}}_0{Z}_{\theta }^{-}=D(P_0\mathrm{\parallel }{P}_{\theta })+{\mathbb{E}}_0{Z}_{\theta }^{+}\le a+1$. Thus ${\mathbb{E}}_0\mathrm{\mid }{Z}_{\theta }\mathrm{\mid }\le a+2$.

Fix $\eta >0$ with $V_{\eta }\mathrm{\ne }\mathrm{\varnothing }$, and let ${\alpha }_n=P_0^n{\varphi }_n$ and ${\beta }_n={\sup }_{\theta \in V_{\eta }}{P}_{\theta }^n(1-{\varphi }_n)$. The two summable series imply ${\alpha }_n,{\beta }_n\to 0$. Choose $m\ge 1$ with ${\alpha }_m<1\mathrm{/}4$ and ${\beta }_m<1\mathrm{/}4$. For $\mathrm{\ell }=\lfloor n\mathrm{/}m\rfloor$, group the first $\mathrm{\ell }m$ observations into disjoint consecutive blocks of length $m$. By [F8, F10] the block vectors are independent and identically distributed under each fixed IID law; [F9] makes $Y_j:={\varphi }_m(\text{block }j)$ independent $[0,1]$ random variables. Their means are at most $1\mathrm{/}4$ under $P_0$ and at least $3\mathrm{/}4$ under every $P_{\theta }$ with $\theta \in V_{\eta }$. Let ${\tilde{\varphi }}_n=\mathbf{1}\{{\mathrm{\ell }}^{-1}{\sum }_{j=1}^{\mathrm{\ell }}{Y}_j>1\mathrm{/}2\}$ for $\mathrm{\ell }\ge 1$, and $0$ otherwise.

Since $P_0=P_{{\theta }_0}\ll \mu$, $p_0(X_i)>0$ for every sampled coordinate almost surely. On this event define the relative evidence $D_n=Z_n(X_{1:n})\mathrm{/}{\prod }_{i=1}^n{p}_0(X_i)$; it is finite $P_0^n$-almost surely by step 1.1.

Let $u(\theta ,x)=p(\theta ,x)\mathrm{/}{p}_0(x)$ when both densities are positive and $0$ otherwise; the lift of $p_0$ to $\mathrm{\Theta }\times \mathsf{X}$ is measurable by [F2,F20], and the piecewise quotient is measurable by [F23,F25]. Extend $Z(\theta ,x)=\log u(\theta ,x)$ on $u>0$ by $0$ on $u=0$. For every real $s$, its upper threshold is $\{u>\exp (s)\}$ if $s\ge 0$ and $\{u=0\}\cup \{u>\exp (s)\}$ if $s<0$; [F27,F29] and [F25] therefore make $Z$ product-measurable. For each $\theta \in B_a$, [F5] identifies $D(P_0\mathrm{\parallel }{P}_{\theta })=-{\mathbb{E}}_0{Z}_{\theta }$; applying [F6] to the positive and negative parts shows that these section integrals, and hence $\theta \mapsto D(P_0\mathrm{\parallel }{P}_{\theta })$ on $B_a$, are measurable. Tonelli [F6] and the uniform bound in step 1.2 show $G(x):=\int Z(\theta ,x){\mathrm{\Pi }}_a(d\theta )$ is finite for $P_0$-almost every $x$ and integrable, with ${\mathbb{E}}_{0}G=-\kappa$ where $\kappa :=\int D(P_0\mathrm{\parallel }{P}_{\theta }){\mathrm{\Pi }}_a(d\theta )\le a$. For each $\theta \in B_a$, $P_0\{x:p(\theta ,x)=0\}=0$; Tonelli [F6] applied to the measurable indicator of this set implies ${\mathrm{\Pi }}_a\{\theta :p(\theta ,X_i)=0\}=0$ almost surely for each coordinate $i$. Intersecting these probability-one events over all $i$, $\exp ({\sum }_{i=1}^{n}Z(\theta ,X_i))={\prod }_{i=1}^{n}p(\theta ,X_i)\mathrm{/}{p}_0(X_i)$ for ${\mathrm{\Pi }}_a$-almost every $\theta$ and every $n$. Set $G=0$ on the measurable $P_0$-null set where the integral is not finite.

For $\mathrm{\ell }\ge 1$, [F26] writes the exponential of the centered sum as a product; the finite product law [F10] and Tonelli [F6] factor its expectation, and [F12] bounds each factor by $e^{{\lambda }^2\mathrm{/}8}$. Thus $\mathbb{E}\exp \{\lambda {\sum }_{j=1}^{\mathrm{\ell }}(Y_j-\mathbb{E}{Y}_j)\}\le e^{\mathrm{\ell }{\lambda }^2\mathrm{/}8}$. Under $P_0$, rejection requires an upward deviation greater than $\mathrm{\ell }\mathrm{/}4$; Markov [F15] with $\lambda =1$ gives $P_0^n{\tilde{\varphi }}_n\le e^{-\mathrm{\ell }\mathrm{/}8}$. Under every $P_{\theta }$ in $V_{\eta }$, nonrejection requires a downward deviation at least $\mathrm{\ell }\mathrm{/}4$; Markov with $\lambda =-1$ gives $P_{\theta }^n(1-{\tilde{\varphi }}_n)\le e^{-\mathrm{\ell }\mathrm{/}8}$. Since $\mathrm{\ell }\ge n\mathrm{/}m-1$, both errors are at most $e^{1\mathrm{/}8}{e}^{-cn}$ for $c:=1\mathrm{/}(8m)$ and all $n\ge m$.

The variables $G(X_i)$ are IID by [F8, F9] and integrable by step 2.2. The strong law [F11] gives $n^{-1}{\sum }_{i=1}^{n}G(X_i)\to -\kappa$; because $\kappa \le a<b$, eventually ${\sum }_{i}G(X_i)>-nb$. Tonelli and the bound in step 1.2 also give $\int {\sum }_i\mathrm{\mid }{Z}_{\theta }(X_i)\mathrm{\mid }{\mathrm{\Pi }}_a(d\theta )<\mathrm{\infty }$ for $P_0^n$-almost every sample; the exponential is integrable by step 1.1. By step 2.2, the likelihood ratio in $D_n$ agrees with $\exp ({\sum }_i{Z}_{\theta }(X_i))$ for ${\mathrm{\Pi }}_a$-almost every $\theta$, so $D_n\ge \mathrm{\Pi }(B_a)\int \exp ({\sum }_i{Z}_{\theta }(X_i)){\mathrm{\Pi }}_a(d\theta )$. Jensen [F13, F14] bounds this integral below by $\exp ({\sum }_{i}G(X_i))$, giving $D_n\ge \mathrm{\Pi }(B_a)e^{-nb}$ eventually.

On $\{p_0(X_i)>0\text{ for }1\le i\le n\}$, define $A_n:=(1-{\tilde{\varphi }}_n){\int }_{V_{\eta }}{\prod }_{i=1}^n[p(\theta ,X_i)\mathrm{/}{p}_0(X_i)]\mathrm{\Pi }(d\theta )$, and set $A_n=0$ otherwise. The product-density identity in step 1.1 and Tonelli [F6] imply ${\mathbb{E}}_{P_0^n}{A}_n={\int }_{V_{\eta }}{P}_{\theta }^n((1-{\tilde{\varphi }}_n){\mathbf{1}}_{\{p_0(X_i)>0\mathrm{\forall }i\}})\mathrm{\Pi }(d\theta )\le e^{1\mathrm{/}8}{e}^{-cn}$ for $n\ge m$, by step 2.3 and $\mathrm{\Pi }(V_{\eta })\le 1$.

On valid posterior fibres, split the likelihood numerator over the test and its complement. The posterior formula [F7] gives ${\mathrm{\Pi }}_n(V_{\eta }\mid X_{1:n})\le {\tilde{\varphi }}_n+A_n\mathrm{/}{D}_n$. Step 3.1 ensures the denominator is finite and positive for all sufficiently large $n$ almost surely.

Choose $0<a<b<c$ and use step 3.1 with these radii. Since $c>0$ and $c-b>0$, [F27] gives $e^{-c},e^{-(c-b)}\in (0,1)$; [F28] makes the corresponding geometric series finite. For each $r>0$, Markov [F15] gives ${\sum }_{n\ge m}{P}_0^n(e^{nb}{A}_n>r)\le r^{-1}{e}^{1\mathrm{/}8}{\sum }_{n\ge m}{e}^{-n(c-b)}<\mathrm{\infty }$. Borel–Cantelli [F16], first for $r=1\mathrm{/}k$ and then by countability, gives $e^{nb}{A}_n\to 0$ almost surely. The type-I bound from step 2.3 is also summable, so [F16] gives ${\tilde{\varphi }}_n\to 0$ almost surely. Combining these limits with $D_n\ge \mathrm{\Pi }(B_a)e^{-nb}$ from step 3.1 and step 4.1 proves ${\mathrm{\Pi }}_n(V_{\eta }\mid X_{1:n})\to 0$ almost surely.

The consistency conclusion holds simultaneously for all $\eta >0$: intersect the probability-one events for positive rational $q$; for any $\eta >0$ choose rational $q<\eta$, so $V_{\eta }\subseteq V_q$.

For each $\theta$ with finite KL, Pinsker [F17] and the local comparison [F18] give $h(P_0,P_{\theta }{)}^2\le 2\sqrt{D(P_0\mathrm{\parallel }{P}_{\theta })\mathrm{/}2}$, hence $h(P_0,P_{\theta }{)}^4\le 2D(P_0\mathrm{\parallel }{P}_{\theta })$. The parameter map $\theta \mapsto h(P_0,P_{\theta }{)}^2$ is measurable: [F25] makes both square-root sections measurable, [F23] makes their squared difference product-measurable, and Tonelli [F6] makes its $\mu$-integral measurable in $\theta$. Thus every Hellinger ball is measurable. Taking a positive-measure KL subset $B_{{\eta }^4\mathrm{/}2}$ places it inside $\{\theta :h(P_0,P_{\theta })<\eta \}$, proving positive prior mass.

Let $\mathsf{X}=\{0,1\}$ with counting measure, let $\mathrm{\Theta }=\{0,1,2,\dots \}$ with its power-set sigma-algebra, and put ${\delta }_k=(k+2{)}^{-2}$ for $k\ge 1$. Define $P_0=\mathrm{Bernoulli}(1\mathrm{/}2)$ and $P_k=\mathrm{Bernoulli}(1\mathrm{/}2+{\delta }_k)$. Every mass is strictly between zero and one, and the map $k\mapsto P_k$ is injective because the ${\delta }_k$ are distinct. Thus the model is identifiable by [F1]. The parameter and observation spaces are discrete, so every event-probability section is measurable by [F9] and every function on their countable product, including the pointwise Bernoulli density, is jointly measurable. The counting measure is sigma-finite and the displayed densities are normalized, so this is a dominated model by [F8].

Give the true parameter $0$ prior mass zero and set $\mathrm{\Pi }(\{k\})=2^{-k}$ for $k\ge 1$. By [F10] these masses sum to one, so [F12] makes $\mathrm{\Pi }$ a probability. For $\delta ={\delta }_k$, the explicit densities and [F2] give $D(P_0\mathrm{\parallel }{P}_k)=\frac{1}{2}\log \frac{1\mathrm{/}2}{1\mathrm{/}2+\delta }+\frac{1}{2}\log \frac{1\mathrm{/}2}{1\mathrm{/}2-\delta }=-\frac{1}{2}\log (1-4{\delta }^2)$. For $0<x<1$, [F11] gives $0\le -\log (1-x)={\int }_{1-x}^{1}dt\mathrm{/}t\le x\mathrm{/}(1-x)$; hence $0\le D(P_0\mathrm{\parallel }{P}_k)\le 2{\delta }_k^2\mathrm{/}(1-4{\delta }_k^2)\to 0$. Every KL neighborhood of $P_0$ therefore contains some $P_k$ with prior mass $2^{-k}>0$.

Put $d(P_j,P_k)=1$ for distinct model laws and $d(P_j,P_j)=0$. This is a metric: if the endpoints in a triangle inequality differ, at least one adjacent pair differs and the sum of its two distances is at least one; if they agree, the left side is zero. Thus the parameter complement is $V_{1\mathrm{/}2}=\{k\ge 1\}$. For every finite data vector, all $P_k^n(x_{1:n})>0$ and are at most one, so the evidence ${\sum }_{k\ge 1}{2}^{-k}{P}_k^n(x_{1:n})$ is finite and positive. By [F5], the posterior is supported on $\{k\ge 1\}$; consequently ${\mathrm{\Pi }}_n(V_{1\mathrm{/}2}\mid x_{1:n})=1$ for every data vector and every $n$.

Let ${\rho }_k={\sum }_{x\in \{0,1\}}\sqrt{P_0(\{x\})P_k(\{x\})}$. Then $0\le {\rho }_k\le 1$, since $h(P_0,P_k{)}^2=2-2{\rho }_k$ is an integral of a square. The IID product law [F6, F7] and distributivity give product affinity ${\rho }_k^n$, so $h(P_0^n,P_k^n{)}^2=2(1-{\rho }_k^n)\le n2(1-{\rho }_k)=nh(P_0,P_k{)}^2$, since $1-r^n=(1-r){\sum }_{j=0}^{n-1}{r}^j\le n(1-r)$ for $0\le r\le 1$. Also $\mathrm{\parallel }{P}_0-P_k{\mathrm{\parallel }}_{\mathrm{T}\mathrm{V}}={\delta }_k$: the four events in $\{0,1\}$ have probability differences $0,{\delta }_k,-{\delta }_k,0$. Thus [F3] gives $\mathrm{\parallel }{P}_0^n-P_n^n{\mathrm{\parallel }}_{\mathrm{T}\mathrm{V}}\le h(P_0^n,P_n^n)\le \sqrt{2n{\delta }_n}\to 0$.

Write ${\mathrm{\Delta }}_n(x)=P_0^n(\{x\})-P_n^n(\{x\})$ on the finite sample space. Since ${\sum }_x{\mathrm{\Delta }}_n(x)=0$, the positive and negative sums each have magnitude $\mathrm{\parallel }{P}_0^n-P_n^n{\mathrm{\parallel }}_{\mathrm{T}\mathrm{V}}$: the event $\{x:{\mathrm{\Delta }}_n(x)>0\}$ attains the positive sum, and every event sum lies between the negative and positive sums. For any randomized test $0\le {\varphi }_n\le 1$, it follows term-by-term that $\mathrm{\mid }{P}_0^n{\varphi }_n-P_n^n{\varphi }_n\mathrm{\mid }=\mathrm{\mid }{\sum }_x{\varphi }_n(x){\mathrm{\Delta }}_n(x)\mathrm{\mid }\le \mathrm{\parallel }{P}_0^n-P_n^n{\mathrm{\parallel }}_{\mathrm{T}\mathrm{V}}$. Hence ${\sup }_{k\ge 1}{P}_k^n(1-{\varphi }_n)\ge P_n^n(1-{\varphi }_n)\ge 1-P_0^n{\varphi }_n-\mathrm{\parallel }{P}_0^n-P_n^n{\mathrm{\parallel }}_{\mathrm{T}\mathrm{V}}$. If the type-I error tends to zero, step 2.3 makes this uniform type-II lower bound tend to one. No tests with both errors tending to zero exist for this complement, although the model is identifiable and the prior has KL support.

Steps 1.1 and 2.1 establish identifiability and KL support, while steps 2.2, 2.3, and 3.1 establish posterior inconsistency and absence of uniformly consistent tests. This refutes the statement.