Gaussian-process posterior contraction in density and regression models

The separable-support reduction [F5] leaves the inherited norm, law and support point unchanged. For the first two branches use the rate $r_n={ϵ}_n$; for the universal fixed-design branch use $r_n={\rho }_n$. The concentration function is nonincreasing in its radius, because its feasible RKHS approximation set grows and centered small-ball probabilities increase. Thus ${\varphi }_{w_0}(r_n)\le {\varphi }_{w_0}({ϵ}_n)\le n{ϵ}_n^2\le n{r}_n^2$. Both choices satisfy $r_n\to 0$ and $n{r}_n^2\to \mathrm{\infty }$. By [F8], the support point lies in the RKHS closure.

Now suppose $W=0$ almost surely and its scale density $g$ is bounded above by $U<\mathrm{\infty }$ on $\mathrm{\mid }\sigma -{\sigma }_0\mathrm{\mid }<\delta$ within $[a,b]$. The support assumption forces $w_0=0$. Put $S_n={\sum }_i{Y}_i^2$, $q=S_n\mathrm{/}(n{\sigma }_0^2)$, and $L_n(\sigma )=({\sigma }_0\mathrm{/}\sigma {)}^n\exp [(1-{\sigma }_0^2\mathrm{/}{\sigma }^2)S_n\mathrm{/}(2{\sigma }_0^2)]$. On $E_R=\{\mathrm{\mid }{S}_n-n{\sigma }_0^2\mathrm{\mid }\le R\sqrt{n}\}$, write $z=(\sigma -{\sigma }_0)\mathrm{/}{\sigma }_0$. Compactness of the positive scale interval and Taylor's formula give constants $c,C>0$ independent of $n,R$ with $-Cn(\sigma -{\sigma }_0{)}^2-CR\sqrt{n}\mathrm{\mid }\sigma -{\sigma }_0\mathrm{\mid }\le \log L_n(\sigma )\le -cn(\sigma -{\sigma }_0{)}^2+CR\sqrt{n}\mathrm{\mid }\sigma -{\sigma }_0\mathrm{\mid }\mathrm{.}$ For verification, its deterministic part is $n[-\log (1+z)+(1-(1+z{)}^{-2})\mathrm{/}2]$, negative away from zero, with second derivative $-2$ at zero; the remaining term is $(S_n-n{\sigma }_0^2)(1-(1+z{)}^{-2})\mathrm{/}(2{\sigma }_0^2)$, bounded by $CR\sqrt{n}\mathrm{\mid }z\mathrm{\mid }$. The residual-energy tail in [F15] with zero mean shift gives ${\mathrm{lim\hspace{0.17em}sup}}_n{P}_0(E_R^c)\le 2e^{-c_0{R}^2}$ for fixed $R$.

In log-Gaussian density estimation, evaluation measurability and Tonelli [F6] make $w\mapsto Z_w$ measurable. On its full-prior positive-finite set, $(w,x)\mapsto e^{w(x)}\mathrm{/}{Z}_w$ is a jointly measurable density family. The $2r_n$ supremum ball has directional KL at most $2e^{4r_n}{r}_n^2$ and log-square at most $16r_n^2$ by [F3]. Choose a fixed sufficiently large $k$. Its prior mass is at least $e^{-n{r}_n^2}$ by [F10], so [F1] at radius $k{r}_n$ gives a denominator at least $e^{-An{r}_n^2}$ for fixed $A=1+2k^2$ except with probability tending to zero.

In binary regression, [F3] places the $2r_n$ prior ball in a directional KL/log-square set of radius $k^2{r}_n^2$ for a fixed sufficiently large $k$. Its mass is at least $e^{-n{r}_n^2}$ by [F10], so [F1] gives the same type of denominator bound. The joint Bernoulli densities relative to $G\otimes \mathrm{\#}$ are bounded by one, as required for the density-$L^2$ tests in [F2].

For the universal fixed-design branch take $r_n={\rho }_n$. On $\mathrm{\parallel }w-w_0{\mathrm{\parallel }}_{\mathrm{\infty }}<2r_n$ and $\mathrm{\mid }\sigma -{\sigma }_0\mathrm{\mid }<r_n$, [F3] bounds average directional KL and log-square by $K{r}_n^2$ for fixed $K$. Independence and [A2], together with [F10], give joint local prior mass at least $c{r}_n{e}^{-n{r}_n^2}$. Since $r_n\ge \sqrt{\log n\mathrm{/}n}$, eventually $\log (1\mathrm{/}{r}_n)\le \frac{1}{2}\log n\le \frac{1}{2}n{r}_n^2$; hence this mass is at least $e^{-C_{0}n{r}_n^2}$ for fixed $C_0$. The independent-observation denominator lemma in [F4] applies.

For each branch choose $C>1$ larger than its denominator exponent; eventually $e^{-Cn{r}_n^2}<1\mathrm{/}2$. Apply the prior sieve theorem [F10] using step 1.1 to get measurable $B_n$ with entropy at radius $3r_n$ at most $6Cn{r}_n^2$ and tail at most $e^{-Cn{r}_n^2}$. The same theorem supplies the local prior mass used in steps 2.1, 2.2, 2.3.

On $E_R$ integrate the likelihood over $\mathrm{\mid }\sigma -{\sigma }_0\mathrm{\mid }\le 1\mathrm{/}\sqrt{n}$ within $[a,b]$. This interval has length at least $1\mathrm{/}\sqrt{n}$ for large $n$, even at an endpoint; [A2] and step 1.2 bound its integral below by $c_R\mathrm{/}\sqrt{n}$ for some $c_R>0$. On the local neighborhood where $g\le U$, substitute $u=\sqrt{n}(\sigma -{\sigma }_0)$ in the numerator outside $M{ϵ}_n$. Dividing by that lower bound gives at most $C_R{\int }_{\mathrm{\mid }u\mathrm{\mid }>M\sqrt{n}{ϵ}_n}{e}^{-c{u}^2+CR\mathrm{\mid }u\mathrm{\mid }}du\to 0$, since $n{ϵ}_n^2\to \mathrm{\infty }$. Outside the local neighborhood, step 1.2 bounds $L_n$ by $e^{-c^{\mathrm{\prime }}n}$ for large $n$, whose integral against the entire probability prior is at most $e^{-c^{\mathrm{\prime }}n}$; after dividing by the denominator this also tends to zero. First let $n\to \mathrm{\infty }$ and then $R\to \mathrm{\infty }$. This proves the sharper degenerate-prior contraction in probability for every fixed $M>0$.

In the density branch intersect $B_n$ with the measurable positive-finite-normalizer set, changing its prior tail by zero. From a radius-$3r_n$ cover, select a valid center for every cover ball meeting that intersection; finite selection doubles the radius to $6r_n$ without increasing the number. The pairwise norm comparison [F3] sends this to a Hellinger cover of radius at most $7r_n$ for large $n$. Thus any $14r_n$-separated set has size at most $e^{6Cn{r}_n^2}$. Taking packing radius at least $\max \{14,\sqrt{6C}\}{r}_n$, [F2] gives tests with null and uniform sieve alternative errors at most $e^{-Tn{r}_n^2}$ for any fixed desired $T>A$, by increasing the alternative multiple. The denominator and sieve tail now satisfy the posterior split in [F4], proving contraction in probability.

In binary regression the norm-ball and sieve cover map by [F3] to density-$L^2$ balls and covers of constant times $r_n$. Taking packing radius at least twice the cover radius and at least $\sqrt{6C}{r}_n$ gives packing entropy at most its squared radius times $n$. By [F2] and [F4], the tests, denominator and sieve tail prove contraction in probability. Since the joint Bernoulli densities satisfy $d_2(q_w,q_{w_0})=\sqrt{2}\mathrm{\parallel }\mathrm{\Psi }(w)-f_0{\mathrm{\parallel }}_{L^2(G)}$, this is the stated binary distance.

In fixed-design regression take the product sieve $B_n\times [a,b]$. Its prior tail is at most $e^{-Cn{r}_n^2}$, and its function component has the radius-$3r_n$ cover from step 2.4. The local fixed-design testing lemma [F14] gives tests outside $\mathrm{\parallel }w-w_0{\mathrm{\parallel }}_n+\mathrm{\mid }\sigma -{\sigma }_0\mathrm{\mid }>M{r}_n$ with errors at most $e^{-Tn{r}_n^2}$ for any fixed $T$ by choosing fixed $M$ large. Choose $T$ and $C$ above the denominator exponent in step 2.3. The posterior split [F4] proves the universal ${\rho }_n$ contraction in probability.

If the Gaussian law is not ${\delta }_0$, [F13] supplies a continuous linear functional with nonzero Gaussian variance. Indeed, otherwise every member of a countable norming family would vanish almost surely, forcing $\mathrm{\parallel }W\mathrm{\parallel }=0$ almost surely. By [F7] there is a fixed constant $D$ such that $\log (1\mathrm{/}{ϵ}_n)\le n{ϵ}_n^2+D$. Put $t_n=n{ϵ}_n^2\to \mathrm{\infty }$; then $\log n=\log t_n+2\log (1\mathrm{/}{ϵ}_n)\le 3t_n+2D$ eventually, since $\log t_n\le t_n$. Consequently ${\rho }_n\le C_1{ϵ}_n$ for fixed $C_1$. The universal conclusion in step 3.3 therefore proves the sharper ${ϵ}_n$ conclusion after enlarging $M$ by $C_1$.

In each proved branch the posterior tail is a random variable $T_n\in [0,1]$ converging to zero in probability. For $\eta >0$, $E{T}_n\le \eta +P(T_n>\eta )$; first take the limit superior and then $\eta \downarrow 0$. Thus all asserted expectation limits follow. The universal fixed-design rate requires no upper bound on the scale density; the sharper degenerate alternative uses only the explicitly local upper bound.