IID posterior contraction by entropy and prior mass

If $m=0$, then $S_m=V=0$, so the zero-variance clause holds. Suppose $m\ge 1$, fix one centered variable $Z$ with $\mathrm{\mid }Z\mathrm{\mid }\le b$, and let ${\sigma }^2=\mathbb{E}[Z^2]$. For every integer $k\ge 2$, the power functions are Borel and $x\mapsto e^{\lambda x}$ is continuous by [F9], so their compositions with $Z$ are measurable by [F10]; the almost-sure bound gives integrability of each power. Thus $\mathrm{\mid }\mathbb{E}[Z^k]\mathrm{\mid }\le \mathbb{E}[\mathrm{\mid }Z{\mathrm{\mid }}^k]\le b^{k-2}{\sigma }^2$. For fixed real $\lambda$, the exponential series converges uniformly on $[-b,b]$: its tail is bounded by ${\sum }_{k>N}(\mathrm{\mid }\lambda \mathrm{\mid }b{)}^k\mathrm{/}k!\to 0$. Since $\mathrm{\mid }Z\mathrm{\mid }\le b$ almost surely, this uniform convergence and finite linearity [F2] permit termwise expectation in [F3], and the centered first-order term vanishes.

For $k\ge 2$, $k!\ge 2\cdot 3^{k-2}$, by induction from $2!=2$ and the fact that each next factor is at least $3$. If $\mathrm{\mid }\lambda \mathrm{\mid }b<3$, [F3] and step 1.1 therefore give $\mathbb{E}{e}^{\lambda Z}\le 1+{\sigma }^2{\sum }_{k\ge 2}\frac{\mathrm{\mid }\lambda {\mathrm{\mid }}^k{b}^{k-2}}{k!}\le 1+\frac{{\lambda }^2{\sigma }^2}{2(1-\mathrm{\mid }\lambda \mathrm{\mid }b\mathrm{/}3)}\le \exp \text{\hspace{0.17em}⁣}\left(\frac{{\lambda }^2{\sigma }^2}{2(1-\mathrm{\mid }\lambda \mathrm{\mid }b\mathrm{/}3)}\right),$ where the last inequality follows from $1+x\le e^x$ for $x\ge 0$, which follows termwise from [F3].

For $0<\lambda <3\mathrm{/}b$, repeated use of [F4] gives $e^{\lambda S_m}={\prod }_{i=1}^m{e}^{\lambda Z_i}$. Each $Z_i$ is measurable, scalar multiplication preserves measurability by [F10], and $x\mapsto e^x$ is Borel by [F9], so the factors are measurable by composition. They are nonnegative functions of the independent variables, so [F5] and step 2.1 yield $\mathbb{E}{e}^{\lambda S_m}={\prod }_{i=1}^m\mathbb{E}{e}^{\lambda Z_i}\le \exp \text{\hspace{0.17em}⁣}\left(\frac{{\lambda }^{2}V}{2(1-\lambda b\mathrm{/}3)}\right)\mathrm{.}$

If $V>0$, take $\lambda =t\mathrm{/}(V+bt\mathrm{/}3)$; then $0<\lambda b\mathrm{/}3=bt\mathrm{/}(3V+bt)<1$. The finite sum $S_m$ and its threshold event are measurable by [F10]. Since [F9] makes $e^x$ increasing, on $\{S_m\ge t\}$ one has $e^{\lambda S_m}\ge e^{\lambda t}$. Markov [F6] and step 3.1 give $\mathbb{P}(S_m\ge t)\le \exp \text{\hspace{0.17em}⁣}(\frac{{\lambda }^{2}V}{2(1-\lambda b\mathrm{/}3)}-\lambda t)=\exp \text{\hspace{0.17em}⁣}(-\frac{t^2}{2(V+bt\mathrm{/}3)})\mathrm{.}$

Applying step 4.1 to $-Z_1,\dots ,-Z_m$ gives the lower-tail bound, since this family remains independent, centered, bounded by $b$, and has the same total second moment $V$. If $V=0$, each nonnegative $Z_i^2$ has expectation zero, so [F7] makes every $Z_i=0$ almost surely; their finite sum is then zero almost surely.

Define $k(\theta ,x)=p_0(x)\log (p_0(x)\mathrm{/}{p}_{\theta }(x))$ as $0$ when $p_0=0$ and as $+\mathrm{\infty }$ when $p_0>0,p_{\theta }=0$. The sets $\{p_0=0\}$, $\{p_0>0,p_{\theta }=0\}$, and $\{p_0>0,p_{\theta }>0\}$ are product-measurable; on the last set the logarithm is a difference of measurable logarithms by [F13]–[F14]. Thus $k^{+}$ and $k^{-}$ are product-measurable by [F10]. Where $p_{\theta }>p_0>0$, setting $u=p_0\mathrm{/}{p}_{\theta }\in (0,1)$ gives $-u\log u\le 1\mathrm{/}e$, so $k^{-}\le p_{\theta }\mathrm{/}e$ and $\int k^{-}d\mu \le 1\mathrm{/}e$. If $P_0\ll P_{\theta }$, then $P_0\{p_{\theta }=0\}=0$; [F17] gives $p_0=0$ $\mu$-almost everywhere there. Consequently, for every measurable $A$, the function $r_{\theta }=p_0\mathrm{/}{p}_{\theta }$ on $\{p_{\theta }>0\}$ and $0$ otherwise satisfies ${\int }_A{r}_{\theta }d{P}_{\theta }={\int }_{A\cap \{p_{\theta }>0\}}{p}_{0}d\mu =P_0(A)$. By [F18], it is a Radon–Nikodym derivative, so [F3] identifies $\mathrm{K}\mathrm{L}(P_0\mathrm{\parallel }{P}_{\theta })=\int k(\theta ,x)d\mu$. Conversely, if $P_0\{p_{\theta }=0\}=0$, then every $P_{\theta }$-null set $A$ has $p_{\theta }=0$ $\mu$-almost everywhere on $A$, hence $P_0(A)=0$; thus failure of $P_0\ll P_{\theta }$ implies $P_0\{p_{\theta }=0\}>0$. In that case the same integral is $+\mathrm{\infty }$, again equal to [F3]. Finally, Tonelli [F4] makes the integrals of $k^{+}$ and $k^{-}$ measurable in $\theta$; the latter is finite, so their difference gives a measurable extended-real KL map. For the log-square coordinate, the specified ratio conventions and [F10], [F13], [F14] give a nonnegative product-measurable integrand, whose section integral is measurable by [F4]. Hence $K_n\in \mathcal{T}$.

Set $Y_{\theta ,i}:=p_{\theta }(X_i)\mathrm{/}{p}_0(X_i)$ when $p_0(X_i)>0$ and $0$ otherwise. This is a measurable function of $X_i$ by [F10, F14]. Since $P_0\{p_0=0\}={\int }_{\{p_0=0\}}{p}_{0}d\mu =0$, change of density using [F1, F2] gives ${\mathbb{E}}_{P_0}{Y}_{\theta ,i}={\int }_{\{p_0>0\}}{p}_{\theta }d\mu \le 1$. For each fixed $\theta$, the $Y_{\theta ,i}$ are independent measurable transforms of the IID coordinates; [F6, F7, F12] therefore give ${\mathbb{E}}_{P_0^n}{\prod }_i{Y}_{\theta ,i}\le 1$. Joint measurability and Tonelli [F4] imply ${\mathbb{E}}_{P_0^n}{D}_n\le 1$, so [F15] gives $D_n<\mathrm{\infty }$ almost surely.

By [A1, F3], every $\theta \in K_n$ satisfies $P_0\ll P_{\theta }$, so $P_0\{p_{\theta }=0\}=0$. For each fixed $\theta$, the IID sample therefore has ${\prod }_i{p}_{\theta }(X_i)>0$ almost surely. Apply Tonelli [F4] to the zero-product indicator on $K_n\times {\mathsf{X}}^n$ with the product of the prior conditioned on K_n and the n-fold sampling law: for $P_0^n$-almost every sample, the conditional prior measure of parameters with zero product is zero. Hence the product likelihood is positive for prior-almost every $\theta \in K_n$, and [F17] implies its integral over the positive-mass set $K_n$ is positive. On the full-probability event where each $p_0(X_i)>0$, this implies $D_n>0$.

Write ${\bar{\mathrm{\Pi }}}_n=\mathrm{\Pi }(\cdot \mid K_n)$ and define the jointly measurable extended-real function $\mathrm{\ell }(\theta ,x)$ by the ratio conventions in the Statement: it is zero on $\{p_0=0\}$, equals $\log (p_{\theta }\mathrm{/}{p}_0)$ where $p_0,p_{\theta }>0$, and is $-\mathrm{\infty }$ where $p_0>0,p_{\theta }=0$; the stated arithmetic and Borel-composition facts [F10, F13, F14] justify this measurability. This avoids assuming a common $P_0$-full set on which all densities in an uncountable family are positive. For every $\theta \in K_n$, finite KL gives $P_0\ll P_{\theta }$, so $\mathrm{\ell }(\theta ,\cdot )$ is finite $P_0$-almost everywhere; the local second-moment condition gives ${\mathbb{E}}_{P_0}\mathrm{\ell }(\theta ,X_1{)}^2\le {ϵ}_n^2$. Holder [F16] on the probability space yields ${\mathbb{E}}_{P_0}\mathrm{\mid }\mathrm{\ell }(\theta ,X_1)\mathrm{\mid }\le {ϵ}_n$, and the directional convention [F3] gives ${\mathbb{E}}_{P_0}\mathrm{\ell }(\theta ,X_1)=-\mathrm{K}\mathrm{L}(P_0\mathrm{\parallel }{P}_{\theta })\ge -{ϵ}_n^2$. Tonelli [F4] applied to the positive and negative parts therefore makes $Z(x):=\int \mathrm{\ell }(\theta ,x){\bar{\mathrm{\Pi }}}_n(d\theta )$ finite almost everywhere, measurable after assigning zero on the null exceptional set, and integrable with ${\mathbb{E}}_{P_0}Z(X_1)\ge -{ϵ}_n^2$. Tonelli also gives ${\mathbb{E}}_{P_0}\int \mathrm{\ell }(\theta ,X_1{)}^2{\bar{\mathrm{\Pi }}}_n(d\theta )\le {ϵ}_n^2$; for $P_0$-almost every $x$, Jensen [F5] for $z\mapsto z^2$ gives $Z(x{)}^2\le \int \mathrm{\ell }(\theta ,x{)}^2{\bar{\mathrm{\Pi }}}_n(d\theta )$. Hence ${\mathbb{E}}_{P_0}Z(X_1{)}^2\le {ϵ}_n^2$.

Set $Z_i:=Z(X_i)$. By [F6] and [F7], the $Z_i$ are IID; [F8] and step 1.4 imply $\mathrm{Var}(Z_i)\le {ϵ}_n^2$, so the centered variables are square-integrable and integrable. For $i\mathrm{\ne }j$, the centered variables are independent measurable transforms of independent coordinates, so [F12] factors their product expectation and their covariance is zero. Applying [F11] to their finite sum therefore gives $\mathrm{Var}(n^{-1}{\sum }_i{Z}_i)=n^{-2}{\sum }_i\mathrm{Var}(Z_i)\le {ϵ}_n^2\mathrm{/}n$.

The mean of $n^{-1}{\sum }_i{Z}_i$ is at least $-{ϵ}_n^2$ and its variance is at most ${ϵ}_n^2\mathrm{/}n$, by independence in step 2.1. Thus the event $n^{-1}{\sum }_i{Z}_i<-2{ϵ}_n^2$ forces a deviation from its mean of at least ${ϵ}_n^2$. Chebyshev [F9] yields $P_0^n\text{\hspace{0.17em}⁣}\{n^{-1}{\sum }_i{Z}_i<-2{ϵ}_n^2\}\le \frac{1}{n{ϵ}_n^2}\mathrm{.}$

By Tonelli [F4] and the local second-moment bound, for $P_0^n$-almost every sample each of the finitely many functions $\theta \mapsto \mathrm{\ell }(\theta ,X_i)$ is square-integrable under ${\bar{\mathrm{\Pi }}}_n$; Cauchy–Schwarz [F16] makes their sum integrable. Step 1.3 and the $P_0^n$-full event where each $p_0(X_i)>0$ ensure that $\exp ({\sum }_i\mathrm{\ell }(\theta ,X_i))={\prod }_i{p}_{\theta }(X_i)\mathrm{/}{p}_0(X_i)$ for ${\bar{\mathrm{\Pi }}}_n$-almost every $\theta$. Step 1.2 bounds the integrated exponential by the finite $D_n\mathrm{/}\mathrm{\Pi }(K_n)$, so Jensen [F5] applies to that sum and the exponential. On the complementary event, it gives $D_n\ge \mathrm{\Pi }(K_n)\int \exp \text{\hspace{0.17em}⁣}({\sum }_i\mathrm{\ell }(\theta ,X_i)){\bar{\mathrm{\Pi }}}_n(d\theta )\ge \mathrm{\Pi }(K_n)\exp \text{\hspace{0.17em}⁣}\left({\sum }_i{Z}_i\right)\ge e^{-(C+2)n{ϵ}_n^2}\mathrm{.}$ Combining with step 3.1 proves the probability bound.

Since $p,q$ are measurable, [F8] makes $\sqrt{p},\sqrt{q}$ measurable and [F7] makes their difference and absolute value measurable. Their unit integrals put both roots in $L^2(\mu )$. The triangle inequality in [F2] gives symmetry, nonnegativity, and the triangle inequality for $h$. If $h(P,Q)=0$, [F5] gives $\sqrt{p}=\sqrt{q}$ almost everywhere, hence $p=q$ almost everywhere and $P=Q$. Conversely, put $g=p-q\in L^1(\mu )$ by [F6]. If $P=Q$, then for every measurable $A$, $\int g{\mathbf{1}}_{A}d\mu =\int p{\mathbf{1}}_{A}d\mu -\int q{\mathbf{1}}_{A}d\mu =P(A)-Q(A)=0$ by [F6]. For each integer $m\ge 1$, the set $E_m:=\{g\ge 1\mathrm{/}m\}$ is measurable by [F8]. Define $u_m:=g{\mathbf{1}}_{E_m}$; it is nonnegative and measurable by [F7], and linearity gives $\int u_{m}d\mu =0$. By [F5], $u_m=0$ almost everywhere; since $u_m\ge 1\mathrm{/}m$ on $E_m$, $\mu (E_m)=0$. Applying the same argument to $(q-p){\mathbf{1}}_{\{q-p\ge 1\mathrm{/}m\}}$ shows $\mu (\{g\le -1\mathrm{/}m\})=0$. Since $\{g\mathrm{\ne }0\}$ is the countable union of these null sets, $p=q$ almost everywhere and $h(P,Q)=0$. Thus $h$ is a metric on these laws.

Put $g:=p-q$ and $A:=\{x:g(x)\ge 0\}$, which is measurable by [F7, F8]. Since $\int gd\mu =0$, the positive and negative parts have equal integral, so ${\int }_{A}gd\mu =\frac{1}{2}\int \mathrm{\mid }g\mathrm{\mid }d\mu$. For every measurable $B$, positivity of $g$ on $A$ and nonpositivity on $A^c$, together with monotonicity [F9], give ${\int }_{A^c}g\le {\int }_{B}g\le {\int }_{A}g$; equality in the absolute supremum is attained at $A$ (or $A^c$). Thus [F4] gives $\parallel P-Q{\parallel }_{\mathrm{T}\mathrm{V}}=\frac{1}{2}{d}_1(P,Q)$.

The functions in the pointwise inequality are measurable by [F7, F8]. If $p\ge q$, then $(\sqrt{p}-\sqrt{q}{)}^2\le p-q$, because $0\le (\sqrt{p}-\sqrt{q})\mathrm{/}(\sqrt{p}+\sqrt{q})\le 1$ when the denominator is positive; the case $q\ge p$ is symmetric, and when both vanish the inequality is zero on both sides. Monotonicity [F9] and [F1] give $h(P,Q{)}^2\le d_1(P,Q)$.

By [F7, F8], the displayed products and sums of square roots are measurable. Factor $\mathrm{\mid }p-q\mathrm{\mid }=\mathrm{\mid }\sqrt{p}-\sqrt{q}\mathrm{\mid }(\sqrt{p}+\sqrt{q})$. By [F3], $d_1(P,Q)\le h(P,Q)\parallel \sqrt{p}+\sqrt{q}{\parallel }_2$. Also $\parallel \sqrt{p}+\sqrt{q}{\parallel }_2^2=2+2\int \sqrt{pq}d\mu \le 4$, again by [F3] and the unit $L^2$ norms. Hence $d_1(P,Q)\le 2h(P,Q)$, and step 1.2 gives $\parallel P-Q{\parallel }_{\mathrm{T}\mathrm{V}}\le h(P,Q)$.

Put $\eta :=1\mathrm{/}128$ and, for integers $j\ge 0$, set $a_j:=(M+j){ϵ}_n$ and $S_{n,j}:=\{p\in {\mathcal{F}}_n:a_j<d(p,p_0)\le a_j+{ϵ}_n\}$. These shells cover every $p$ with $d(p,p_0)>M{ϵ}_n$. The selected metrics are bounded on probability densities: $h^2\le 2$ by [F1], $d_1\le 2$ by [F17] and the unit-mass densities, and in the $d_2$ branch only, [A2] and [F17] give $d_2^2\le \int (p+p_0{)}^{2}d\mu \le 2B\int (p+p_0)d\mu =4B$. Thus only finitely many shells are nonempty for fixed $n$. For each nonempty shell choose a maximal subset $Q_{n,j}\subseteq S_{n,j}$ whose distinct points are more than $r_j:=\eta a_j$ apart. Such a finite maximal set exists: the attainable cardinalities of finite $r_j$-separated subsets form a nonempty bounded subset of $\mathbb{N}$, so a largest cardinality is attained. Since $a_j\ge M{ϵ}_n$ and $M\ge 128$, $r_j\ge {ϵ}_n$; its size is at most $D({ϵ}_n,{\mathcal{F}}_n,d)\le e^{n{ϵ}_n^2}$. Maximality puts every shell member within $r_j$ of a center, and the triangle inequality gives $d(q,p_0)>(1-\eta )a_j$ for each center. Only finitely many choices are made for each $n$, so no arbitrary-index choice axiom is used to form the nets.

Now let $d=d_1$. For a center $q$ in shell $j$, put $s:=d_1(q,p_0)>(1-\eta )a$, $r=\eta a$, and $f(x):=\mathrm{sign}(q(x)-p_0(x))$, with $\mathrm{sign}(0)=1$. By [F15, F16], $f$ is measurable, so its sample mean and threshold event are measurable. Then $\mathrm{\mid }f\mathrm{\mid }\le 1$, $P_{q}f-P_{0}f=s$, and every $p$ with $d_1(p,q)\le r$ satisfies $P_{p}f-P_{0}f\ge \mathrm{\Delta }:=s-r>(1-2\eta )a$. Test by whether the sample mean of $f(X_i)$ is at least $P_{0}f+\mathrm{\Delta }\mathrm{/}2$. Under either law, the centered summands have variance at most one and absolute value at most two; also $\mathrm{\Delta }\le s\le 2$. Bernstein [F4] with $t=n\mathrm{\Delta }\mathrm{/}2$, total variance at most $n$ and bound $b=2$ gives exponent at least $n{\mathrm{\Delta }}^2\mathrm{/}[8(1+\mathrm{\Delta }\mathrm{/}3)]\ge 3n{\mathrm{\Delta }}^2\mathrm{/}40$, so each error is at most $\exp (-3n{\mathrm{\Delta }}^2\mathrm{/}40)$. Thus this branch has $c_1=3(1-2\eta {)}^2\mathrm{/}40$.

Now let $d=d_2$ and assume [A2]. At center $q$, put $s:=\parallel q-p_0{\parallel }_2>(1-\eta )a$, $r=\eta a$, and $f:=q-p_0$. This is measurable by [F15]. Then $P_{q}f-P_{0}f=s^2$. Since $r\mathrm{/}s<\eta \mathrm{/}(1-\eta )<1\mathrm{/}2$, every $p$ with $\parallel p-q{\parallel }_2\le r$ satisfies, by [F10], $P_{p}f-P_{0}f\ge \mathrm{\Delta }:=s^2-rs\ge s^2\mathrm{/}2$. Also $\mathrm{\Delta }\le s^2$, ${\mathbb{E}}_{R}f(X{)}^2\le B{s}^2$ for $R=P_0$ or $P$ by [A2] and monotonicity [F17], and $\mathrm{\mid }f(X)-Rf\mathrm{\mid }\le 2B$ under either law because $\mathrm{\mid }f\mathrm{\mid }\le B$. Testing the sample mean at threshold $P_{0}f+\mathrm{\Delta }\mathrm{/}2$ and applying [F4] with total variance at most $nB{s}^2$, bound $b=2B$, and $t=n\mathrm{\Delta }\mathrm{/}2$ gives exponent at least $3n{s}^2\mathrm{/}(128B)\ge 3n{a}^2\mathrm{/}(512B)$. Thus each error is at most $\exp (-3n{a}^2\mathrm{/}(512B))$ and $c_2=3\mathrm{/}(512B)$.

Fix a nonempty shell, write $a:=a_j$, $r:=\eta a$, and let $q\in Q_{n,j}$. First consider $d=h$. Write $s_0=(1-\eta )a$ and $\rho :=\int \sqrt{p_{0}q}d\mu =1-h(P_0,Q{)}^2\mathrm{/}2$. By [F10], $0\le \rho \le 1$; the affinity integrand is measurable by [F15, F16]. Since $h(P_0,Q)>s_0$, this gives $\rho \le 1-s_0^2\mathrm{/}2$. By step 1.1 and nonemptiness, $s_0^2\le 2$. Set $m:=\lceil 2\log (8)\mathrm{/}{s}_0^2\rceil$. For $u=s_0^2\mathrm{/}2\in [0,1]$, $e^{-u}-(1-u)$ has value zero at $u=0$ and derivative $1-e^{-u}\ge 0$ by monotonicity of the exponential; hence $1-u\le e^{-u}$. On a block of $m$ observations use $T=\mathbf{1}\{{\prod }_{i=1}^{m}q(X_i)>{\prod }_{i=1}^m{p}_0(X_i)\}$. The products and comparison event are measurable by [F15, F16]. Tonelli on the finite product space gives product affinity ${\rho }^m$, and ${\rho }^m\le (1-u{)}^m\le e^{-mu}\le 1\mathrm{/}8$. Its type-I error under $P_0^m$ is at most this affinity, since on $T=1$ the product density $p_0^{\otimes m}$ is at most $\sqrt{p_0^{\otimes m}{q}^{\otimes m}}$; its type-II error under $Q^m$ is at most the same bound by the symmetric inequality on $T=0$.

If $p$ lies in the Hellinger ball $h(P,Q)\le r$, then Tonelli on the measurable product densities gives $h(P^m,Q^m{)}^2=2(1-\rho (P,Q{)}^m)\le mh(P,Q{)}^2$, since $1-z^m=(1-z)(1+z+\cdots +z^{m-1})\le m(1-z)$ for $0\le z\le 1$. By [F2], $\parallel P^m-Q^m{\parallel }_{\mathrm{T}\mathrm{V}}\le h(P^m,Q^m)\le \sqrt{m}r$. Since $a\le \sqrt{2}$ for a nonempty Hellinger shell, $s_0^2\le 2$, $m\le 8\mathrm{/}{s}_0^2$, and $m{r}^2\le 8{\eta }^2\mathrm{/}(1-\eta {)}^2<1\mathrm{/}64$, so $\sqrt{m}r<1\mathrm{/}8$. Indeed, [F11] gives $e^3>1+3+9\mathrm{/}2>8$, hence $\log 8<3$; also $1-\eta >1\mathrm{/}2$. For the event $\{T=0\}$, the change in probability from $Q^m$ to $P^m$ is at most TV, so the type-II error is at most $1\mathrm{/}8+1\mathrm{/}8=1\mathrm{/}4$ uniformly over this ball.