Gaussian-process posterior contraction — Examples

The RKHS of the point-mass prior is $\{0\}$; its approximation cost at $w_0=0$ is zero. Also $\mathrm{Pr}(\mathrm{\mid }W\mathrm{\mid }<ϵ)=1$ for every $ϵ>0$, so the explicitly defined concentration function satisfies ${\varphi }_0(ϵ)=0$.

For $n$ sufficiently large, ${ϵ}_n<1\mathrm{/}2$, hence the uniform prior assigns mass $2{ϵ}_n$ to $(1-{ϵ}_n,1+{ϵ}_n)$. Therefore $-\log (2{ϵ}_n)=\frac{1}{2}\log n-\frac{1}{2}\log \log \log n-\log 2,$ while $n{ϵ}_n^2=\log \log n$. Along $n_m=\lceil e^{e^m}\rceil$, the first quantity grows on the order of $e^m$ and the second on the order of $m$, so their ratio is unbounded.

Since $\log \log n\to \mathrm{\infty }$ and $\log \log n\mathrm{/}n\to 0$, the rate has $n{ϵ}_n^2\to \mathrm{\infty }$ and ${ϵ}_n\to 0$ while the scale interval mass has negative logarithm not $O(n{ϵ}_n^2)$ by step 1.2. Thus the Gaussian concentration condition does not establish this particular denominator-mass check; it does not establish failure of posterior contraction. Any separate posterior-contraction argument requires a specified sampling likelihood and its own hypotheses; a local bound on the scale-prior density alone is insufficient.

Since $\mathrm{\mid }{A}_k\mathrm{\mid }=k^{-4}$, $\mathrm{\parallel }{w}_k{\mathrm{\parallel }}_2^2=k^2\mathrm{\mid }{A}_k\mathrm{\mid }=k^{-2}\to 0$. The normalizer is $Z_k=1-k^{-4}+e^k{k}^{-4}$, so $p_k$ is a positive unit-mass density.

The Hellinger affinity is ${\int }_0^1\sqrt{p_k}dx=\frac{1-k^{-4}+k^{-4}{e}^{k\mathrm{/}2}}{\sqrt{Z_k}}\mathrm{.}$ Since $Z_k\ge e^k\mathrm{/}{k}^4$, this is at most $k^2{e}^{-k\mathrm{/}2}+k^{-2}$. The exponential series gives $e^{k\mathrm{/}2}\ge (k\mathrm{/}2{)}^3\mathrm{/}3!=k^3\mathrm{/}48$, hence $k^2{e}^{-k\mathrm{/}2}\le 48\mathrm{/}k\to 0$.

By [F1], $h^2(P_k,P_0)\to 2$, and nonnegativity of Hellinger distance gives $h(P_k,P_0)\to \sqrt{2}$.

The intervals $I_k$ are disjoint subsets of $[1\mathrm{/}2,3\mathrm{/}2]$, since ${\delta }_{k+1}={\delta }_k^{k+1}$ and ${\delta }_{k+1}+{\delta }_{k+1}^4<{\delta }_k$. The series $S={\sum }_{k\ge k_0}\sqrt{{\delta }_k}$ is positive and finite, bounded by ${\sum }_{k\ge k_0}{e}^{-k\mathrm{/}2}$ using $k!\ge k$ and [F3]. Define ${\mathrm{\Pi }}_{\sigma }=\frac{1}{2}{U}_{[1\mathrm{/}2,3\mathrm{/}2]}+{\sum }_{k\ge k_0}{m}_k{U}_{I_k},m_k=\frac{\sqrt{{\delta }_k}}{2S}\mathrm{.}$ By [F2] this is a probability measure with a measurable Lebesgue density bounded below by $1\mathrm{/}2$ on the required interval. Its spike densities need not be bounded above near $1$.

The zero Gaussian law is tight and centered, with support $\{0\}$, covariance RKHS $\{0\}$ and ${\varphi }_0(ϵ)=0$ for every positive $ϵ$. For all sufficiently large $n$ define ${ϵ}_n=\sqrt{\log \log \log n\mathrm{/}n}$, choosing arbitrary positive initial terms. Then ${ϵ}_n\to 0$, $n{ϵ}_n^2\to \mathrm{\infty }$ and the concentration condition holds. Put $n_k=\lceil k^4{\delta }_k^{-2}\rceil$. Since $\log n_k=2k!+O(\log k)$, $\log \log \log n_k=O(\log k)$ and tends to infinity; hence ${ϵ}_{n_k}\mathrm{/}{\delta }_k=O(\sqrt{\log k}\mathrm{/}{k}^2)\to 0$.

Put $S_n={\sum }_i{Y}_i^2$. By [F1] the likelihood ratio relative to true scale $1$ is $L_n(\sigma )=\exp \{-n\log \sigma +(1-{\sigma }^{-2})S_n\mathrm{/}2\}\mathrm{.}$ On $E_k=\{\mathrm{\mid }{S}_{n_k}-n_k\mathrm{\mid }\le k\sqrt{n_k}\}$, [F4] gives $P_0(E_k^c)\to 0$. Write $q=S_{n_k}\mathrm{/}{n}_k$. Maximizing the log likelihood over all positive $\sigma$ gives its maximum $n_k(q-1-\log q)\mathrm{/}2$ at ${\sigma }^2=q$. Since $\mathrm{\mid }q-1\mathrm{\mid }\le k\mathrm{/}\sqrt{n_k}$, Taylor's formula gives ${\sup }_{\sigma \in [1\mathrm{/}2,3\mathrm{/}2]}{L}_{n_k}(\sigma )\le e^{C{k}^2}$ for a constant independent of $k$.

Uniformly on $I_k$, $\mathrm{\mid }\sigma -1\mathrm{\mid }\le 2{\delta }_k$. The function $d(\sigma )=-\log \sigma +(1-{\sigma }^{-2})\mathrm{/}2$ satisfies $d(1)=d^{\mathrm{\prime }}(1)=0$ and has bounded second derivative near $1$, while $\mathrm{\mid }1-{\sigma }^{-2}\mathrm{\mid }\le C\mathrm{\mid }\sigma -1\mathrm{\mid }$. On $E_k$ this gives $\log L_{n_k}(\sigma )\ge -C{n}_k{\delta }_k^2-Ck\sqrt{n_k}{\delta }_k\ge -C^{\mathrm{\prime }}{k}^4\mathrm{.}$ Thus the posterior denominator is at least $m_k{e}^{-C^{\mathrm{\prime }}{k}^4}$ on $E_k$.

Fix $M>0$. Since ${ϵ}_{n_k}\mathrm{/}{\delta }_k\to 0$, the set $J_k=\{\mathrm{\mid }\sigma -1\mathrm{\mid }\le M{ϵ}_{n_k}\}$ misses $I_k$ and all earlier spikes for large $k$. Its prior mass is at most $M{ϵ}_{n_k}+{\sum }_{j>k}{m}_j$. The later sum is $O(\sqrt{{\delta }_{k+1}})$: successive square-root terms have ratio at most $1\mathrm{/}2$ for $k\ge k_0$, so [F3] bounds the tail by twice its first term. Steps 2.1, 3.1 therefore give ${\mathrm{\Pi }}_{\sigma }(J_k\mid Y^{n_k})\le e^{C{k}^2+C^{\mathrm{\prime }}{k}^4}\frac{M{ϵ}_{n_k}+O(\sqrt{{\delta }_{k+1}})}{m_k}\mathrm{.}$ Both terms tend to zero: ${ϵ}_{n_k}\mathrm{/}\sqrt{{\delta }_k}=O(e^{-k!\mathrm{/}2}\sqrt{\log k}\mathrm{/}{k}^2)$ and $\sqrt{{\delta }_{k+1}\mathrm{/}{\delta }_k}=e^{-kk!\mathrm{/}2}$ dominate the polynomial exponent.

On events with probability tending to one, step 4.1 makes posterior mass within $M{ϵ}_{n_k}$ tend to zero for every fixed $M$. Thus its expected mass outside that radius tends to one, by boundedness and $P_0(E_k^c)\to 0$. The mean function is identically its truth, so this is exactly failure for the combined empirical-mean-plus-scale distance. All original scale-prior and Gaussian concentration assumptions are satisfied, refuting the unrestricted rate claim.

Every prior path is constant, so $Z_W={\int }_0^1{e}^{Z}dx=e^Z$ and $p_W(x)=1=p_0(x)$. Thus the posterior distribution of the density itself is the point mass at $p_0$ for every data vector; its posterior mass outside every positive Hellinger ball is exactly zero.

The RKHS approximation term at $w_0=0$ is zero because the zero vector belongs to every Hilbert space. For $0<ϵ\le 1$, [F1] gives $\mathrm{Pr}(\mathrm{\parallel }W{\mathrm{\parallel }}_{\mathrm{\infty }}<ϵ)=\mathrm{Pr}(\mathrm{\mid }Z\mathrm{\mid }<ϵ)\ge 2\phi (1)ϵ$, hence ${\varphi }_0(ϵ)\le -\log (2\phi (1)ϵ)$. For ${ϵ}_n=\sqrt{\log n\mathrm{/}n}$, $n{ϵ}_n^2=\log n$ and $-\log {ϵ}_n\le \frac{1}{2}\log n$ for large n, so ${\varphi }_0({ϵ}_n)\le n{ϵ}_n^2$ eventually.

The density family is jointly measurable and all normalizers are positive and finite. Step 1.2 verifies the concentration premise; applying the log-Gaussian branch of [F2] gives the stated rate, while step 1.1 gives the stronger exact posterior conclusion.

Differentiation gives ${\mathrm{\Psi }}^{\mathrm{\prime }}(u)=\mathrm{\Psi }(u)(1-\mathrm{\Psi }(u))$, so $\mathrm{\mid }{\mathrm{\Psi }}^{\mathrm{\prime }}(u)\mathrm{\mid }\le 1\mathrm{/}4$ and $\mathrm{\mid }{\mathrm{\Psi }}^{\mathrm{\prime }}(u)\mathrm{/}[\mathrm{\Psi }(u)(1-\mathrm{\Psi }(u))]\mathrm{\mid }=1$. The true response probability is $\mathrm{\Psi }(0)=1\mathrm{/}2\in (0,1)$, and $w_0=0$ is in the support of the scalar standard-normal prior.

Here $L^2(G)$ is isometric to $\mathbb{R}$ and $\mathrm{\parallel }W{\mathrm{\parallel }}_{L^2(G)}=\mathrm{\mid }Z\mathrm{\mid }$. The RKHS approximation cost at zero is zero. By [F1], for $0<ϵ\le 1$, $\mathrm{Pr}(\mathrm{\mid }Z\mathrm{\mid }<ϵ)\ge 2\phi (1)ϵ$, so ${\varphi }_0({ϵ}_n)\le -\log (2\phi (1){ϵ}_n)$. With ${ϵ}_n=\sqrt{\log n\mathrm{/}n}$, this is at most $n{ϵ}_n^2=\log n$ for all sufficiently large n.

The joint Bernoulli density relative to $G\otimes \mathrm{\#}$ is bounded by one, and the constant-process evaluation is jointly measurable. Steps 1.1–1.2 verify the remaining link, truth-support and concentration conditions; applying the binary-regression branch of [F2] yields expected posterior contraction in $L^2(G)$.

The support of $W$ is the line of constant functions, which contains $w_0$. The RKHS approximation cost at zero is zero, and $\mathrm{\parallel }W{\mathrm{\parallel }}_{\mathrm{\infty }}=\mathrm{\mid }Z\mathrm{\mid }$. For $ϵ\le 1$, [F1] gives $\mathrm{Pr}(\mathrm{\parallel }W{\mathrm{\parallel }}_{\mathrm{\infty }}<ϵ)\ge 2\phi (1)ϵ$, so ${\varphi }_0(ϵ)\le -\log (2\phi (1)ϵ)$.

At ${ϵ}_n=2\sqrt{\log n\mathrm{/}n}$, $n{ϵ}_n^2=4\log n$ while $-\log (2\phi (1){ϵ}_n)=\frac{1}{2}\log n-\frac{1}{2}\log \log n+O(1)$, so the concentration inequality holds for large n. Also, for ${ϵ}_n<1\mathrm{/}2$, the uniform scale prior assigns mass $2{ϵ}_n$ to $(1-{ϵ}_n,1+{ϵ}_n)$; its negative log is $O(\log n)=O(n{ϵ}_n^2)$.

The scale prior is independent and has density one, the true scale lies in the interior of its fixed positive interval, and every design point is $x_0$. Steps 1.1–1.2 verify the concentration and scale-local-mass checks, so the sharper fixed-design conclusion follows from [F2]: this Gaussian law is nonzero and the stated rate already satisfies ${\rho }_n={ϵ}_n$.