IID posterior contraction by entropy and prior mass — Examples

Put ${ϵ}_n:=2\sqrt{\log n\mathrm{/}n}$ for $n\ge 3$. For two parameters, $d_1(p_{\theta },p_{{\theta }^{\mathrm{\prime }}})=2\mathrm{\mid }\theta -{\theta }^{\mathrm{\prime }}\mathrm{\mid }$. Hence an ${ϵ}_n$-packing of ${\mathcal{F}}_n$ has at most $1+2(b-a)\mathrm{/}{ϵ}_n$ points, so $\log D({ϵ}_n,{\mathcal{F}}_n,d_1)\le \frac{1}{2}\log n+O(1)\le 4\log n=n{ϵ}_n^2$ for all sufficiently large $n$. Since ${\mathrm{\Theta }}_n=[a,b]$, its prior complement has mass zero. Also ${ϵ}_n\to 0$ and $n{ϵ}_n^2=4\log n\to \mathrm{\infty }$.

For $g(\theta ):=\mathrm{K}\mathrm{L}(P_0\mathrm{\parallel }{P}_{\theta })$, use [F5] to write $g(\theta )=c_0-{\theta }_0\log \theta -(1-{\theta }_0)\log (1-\theta )$, with $c_0={\theta }_0\log {\theta }_0+(1-{\theta }_0)\log (1-{\theta }_0)$. The chain, sum, and scalar rules [F6], together with [F4], give $g({\theta }_0)=g^{\mathrm{\prime }}({\theta }_0)=0$ and $g^{\mathrm{\prime }\mathrm{\prime }}(\theta )=\frac{{\theta }_0}{{\theta }^2}+\frac{1-{\theta }_0}{(1-\theta {)}^2}$; the reciprocal-power derivative in this computation is [F7]. These formulas show that $g$ and $g^{\mathrm{\prime }}$ are differentiable on $(0,1)$; hence they are continuous there by [F8], so the mean value theorem hypotheses hold on the relevant subintervals of $[a,b]$. The explicit bound $\mathrm{\mid }{g}^{\mathrm{\prime }\mathrm{\prime }}(\theta )\mathrm{\mid }\le L:={\theta }_0\mathrm{/}{a}^2+(1-{\theta }_0)\mathrm{/}(1-b{)}^2$ holds there. For $\theta \mathrm{\ne }{\theta }_0$, apply the mean-value theorem [F3] first to $g$ and then to $g^{\mathrm{\prime }}$ to obtain $\mathrm{\mid }g(\theta )\mathrm{\mid }\le L\mathrm{\mid }\theta -{\theta }_0{\mathrm{\mid }}^2$. Put $C_1:=L$. For the log-square term, apply [F3] to $\log \theta$ and $\log (1-\theta )$; these functions are continuous on $[a,b]$ by [F5], and the chain rule [F6] and derivative bounds $1\mathrm{/}\theta \le 1\mathrm{/}a$ and $1\mathrm{/}(1-\theta )\le 1\mathrm{/}(1-b)$ give $\int {[\log \frac{p_{\theta }}{p_0}]}^2{p}_{0}d\mu \le C_2\mathrm{\mid }\theta -{\theta }_0{\mathrm{\mid }}^2$, for example with $C_2=L$. At $\theta ={\theta }_0$, both contrasts are zero by [F2, F5], so both bounds also hold.

Choose $c>0$ with $c^2\max (C_1,C_2)\le 1$. For all sufficiently large $n$, the interval $[{\theta }_0-c{ϵ}_n,{\theta }_0+c{ϵ}_n]$ lies in $[a,b]$ and is contained in the theorem's local KL/log-square set. Its uniform prior mass is $2c{ϵ}_n\mathrm{/}(b-a)$. Since $-\log {ϵ}_n=\frac{1}{2}\log n+O(\log \log n)$ while $n{ϵ}_n^2=4\log n$, there is a fixed $C>0$ such that this mass is at least $e^{-Cn{ϵ}_n^2}$ for large $n$.

All hypotheses of [F1] now hold with $d=d_1$; consequently the posterior mass of $\{\theta :d_1(p_{\theta },p_0)>M{ϵ}_n\}$ tends to zero in $P_0^n$-probability for sufficiently large fixed $M$. The rate furnished by this global-entropy theorem is $2M\sqrt{\log n\mathrm{/}n}$ in the full density $L^1$ norm; this example makes no minimax-optimality claim.

Put $t_n:=\log n$. For $n\ge 3$, $t_n>1$, and ${ϵ}_n^2=\frac{\sqrt{t_n}}{n},n{ϵ}_n^2=\sqrt{t_n}⟶\mathrm{\infty }\mathrm{.}$ The integral formula [F5] gives $\log n\le 2\sqrt{n}$ for $n\ge 1$, hence ${ϵ}_n^2\le \sqrt{2}{n}^{-3\mathrm{/}4}\to 0$. Thus ${ϵ}_n\to 0$ and the rate conditions in [F1] hold.

For $j=0,\dots ,\lfloor 1\mathrm{/}{ϵ}_n\rfloor$, set ${\theta }_j=j{ϵ}_n$. These parameters lie in $[0,1]$, and $d_1(p_{{\theta }_j},p_{{\theta }_k})=2\mathrm{\mid }{\theta }_j-{\theta }_k\mathrm{\mid }>{ϵ}_n(j\mathrm{\ne }k)\mathrm{.}$ Therefore $D({ϵ}_n,{\mathcal{F}}_n,d_1)\ge \lfloor 1\mathrm{/}{ϵ}_n\rfloor +1>1\mathrm{/}{ϵ}_n$ and $\log D({ϵ}_n,{\mathcal{F}}_n,d_1)>\log (1\mathrm{/}{ϵ}_n)=\frac{1}{2}{t}_n-\frac{1}{4}\log t_n\mathrm{.}$ For $t\ge 1$, [F5] gives $\log t\le 2\sqrt{t}$. If $t_n\ge 64$, then $\frac{1}{2}{t}_n-\frac{1}{4}\log t_n\ge \frac{1}{2}{t}_n-\frac{1}{2}\sqrt{t_n}>\sqrt{t_n}=n{ϵ}_n^2$. Hence the entropy premise fails for all sufficiently large $n$.

The sieve is the full parameter space, so $\mathrm{\Pi }({\mathrm{\Theta }}_n^c)=0$. Its local KL/log-square set contains ${\theta }_0$, where $\mathrm{K}\mathrm{L}(P_0\mathrm{\parallel }{P}_{{\theta }_0})=0$ and $\log (p_{{\theta }_0}\mathrm{/}{p}_0)=0$; consequently its prior mass is $1$, which exceeds $e^{-Cn{ϵ}_n^2}$ for every $C>0$.

For any data vector $x_{1:n}$, the likelihood at the sole prior atom is $p_{{\theta }_0}^{\otimes n}(x_{1:n})=2^{-n}>0$, so the evidence is also $2^{-n}$. The normalization in [F1] therefore gives $\mathrm{\Pi }(\cdot \mid x_{1:n})={\delta }_{{\theta }_0}$. For every $M>0$, its mass on $\{\theta :d_1(p_{\theta },p_0)>M{ϵ}_n\}$ is exactly zero for every sample, and thus it contracts in $P_0^n$-probability under the IID data law [F4, F6] despite the failed entropy condition.

The chosen $K$ is fixed and $1\mathrm{/}(2\alpha +1)>0$, so ${ϵ}_n\to 0$ and $n{ϵ}_n^2=K^2{n}^{1\mathrm{/}(2\alpha +1)}\to \mathrm{\infty }$. For all sufficiently large $n$, the assumed entropy bound gives $\log D({ϵ}_n,\mathcal{F},h)\le C_H{K}^{-1\mathrm{/}\alpha }{n}^{1\mathrm{/}(2\alpha +1)}\le K^2{n}^{1\mathrm{/}(2\alpha +1)}=n{ϵ}_n^2,$ because $K^{2+1\mathrm{/}\alpha }\ge C_H$. The full-space sieve has $\mathrm{\Pi }({\mathrm{\Theta }}_n^c)=0$, and the local prior-mass premise is exactly the theorem's remaining prior condition.

All hypotheses of [F3] now hold eventually with the Hellinger metric and the positive sequence $({ϵ}_n)$. Hence for sufficiently large fixed $M$, the posterior mass outside the Hellinger ball of radius $MK{n}^{-\alpha \mathrm{/}(2\alpha +1)}$ tends to zero in $P_0^n$-probability. This is a sufficient-condition rate calculation only; it does not prove the class's entropy bound or assert a minimax lower bound.