Weak nonparametric Bernstein–von Mises limits

For each finite $N$, orthonormality [F2] gives $\mathrm{\parallel }{f}_N{\mathrm{\parallel }}_H^2={\sum }_{0\le l\le N}{\sum }_{k\in K_l}{\sigma }_l^2{\xi }_{lk}^2\mathrm{.}$ The nonnegative partial sums increase to the full coefficient-square sum, so [F4] gives its expectation as the limit of the finite expectations, ${\sum }_{l\ge 0}{\sum }_{k\in K_l}{\sigma }_l^2\mathbb{E}{\xi }_{lk}^2=m_2{\sum }_{l\ge 0}{2}^l{2}^{-2l(\alpha +1\mathrm{/}2)}=m_2{\sum }_{l\ge 0}{2}^{-2\alpha l}<\mathrm{\infty },$ since the final series is geometric with ratio $2^{-2\alpha }<1$. If the coefficient-square sum were infinite on an event of probability $p>0$, its expectation would be at least $Rp$ for every $R>0$, contradicting finiteness. Thus that sum is finite almost surely.

On the probability-one event where that sum is finite, for $M>N$ orthonormality gives $\mathrm{\parallel }{f}_M-f_N{\mathrm{\parallel }}_H^2={\sum }_{N<l\le M}{\sum }_{k\in K_l}{\sigma }_l^2{\xi }_{lk}^2\mathrm{.}$ The right side tends to zero uniformly over $M>N$ as $N\to \mathrm{\infty }$, since it is a tail of a convergent nonnegative series. Thus $(f_N)$ is Cauchy in $H$ and has a limit $f$ by completeness [F1]. When $H=L^2([0,1])$, completeness is supplied by [F5] under [A1]. The Cauchy event is measurable because it is a countable combination of conditions on the measurable distances $\mathrm{\parallel }{f}_M-f_N{\mathrm{\parallel }}_H$. On that event the pointwise limit is measurable as a limit into the separable metric space $H$; setting the limit equal to zero on the measurable complement gives a Borel $H$-valued random element.

The same monotone-convergence calculation in [F4] gives $\mathbb{E}\mathrm{\parallel }f{\mathrm{\parallel }}_H^2=\mathbb{E}{\sum }_{l\ge 0}{\sum }_{k\in K_l}{\sigma }_l^2{\xi }_{lk}^2=m_2{\sum }_{l\ge 0}{2}^{-2\alpha l}<\mathrm{\infty }\mathrm{.}$ This proves both the stated second-moment formula and the claimed well-definedness.

Nonnegativity, symmetry and the triangle inequality for ${\beta }_{BL}$ follow by applying these properties to the difference of the two test integrals and then taking the supremum. If ${\beta }_{BL}(\nu ,\rho )=0$, fix a nonempty closed $F$ and put $g_m=(1-md(\cdot ,F){)}_{+}$. The function $g_m\mathrm{/}m$ is an admissible test, so $\int g_{m}d\nu =\int g_{m}d\rho$. By [F4], these integrals decrease to $\nu (F)$ and $\rho (F)$, respectively. The two probabilities therefore agree on every closed set, including the empty set. Closed sets form a pi-system generating the Borel sigma-algebra, and both total masses are one, so [F6] gives $\nu =\rho$. Thus ${\beta }_{BL}$ is a metric; this argument uses no choice.

Assume ${\beta }_{BL}({\mu }_n,\mu )\to 0$, and fix a nonempty closed set $F\subseteq S$. For every integer $m\ge 1$, the function $g_m(x)=(1-md(x,F){)}_{+}$ satisfies $0\le g_m\le 1$ and ${\mathrm{Lip}}_d(g_m)\le m$. Hence $g_m\mathrm{/}m$ is admissible in the definition of ${\beta }_{BL}$, so $\int g_{m}d{\mu }_n\to \int g_{m}d\mu$. Since $1_F\le g_m$, it follows that ${\mathrm{lim\hspace{0.17em}sup}}_n{\mu }_n(F)\le \int g_{m}d\mu$ for every $m$.

Now assume ${\mu }_n\Rightarrow \mu$, and put $\mathcal{A}=\{\mu \}\cup \{{\mu }_n:n\ge 1\}$. Any sequence chosen from $\mathcal{A}$ either has a member repeated infinitely often, yielding a constant convergent subsequence, or has a subsequence of the form $({\mu }_{n_j})$ with $n_j\to \mathrm{\infty }$, which converges weakly to $\mu$. Therefore $\mathcal{A}$ is relatively sequentially compact for weak convergence.

The functions $g_m$ decrease pointwise to $1_F$ and are bounded by the integrable constant function $1$ under the probability $\mu$. By [F4], $\int g_{m}d\mu \to \mu (F)$, so the closed-set bound holds; it is trivial for $F=\mathrm{\varnothing }$. The Portmanteau theorem [F2] gives weak convergence in the sense of [F1].

By [F5], $S$ is Polish; therefore [F3] applies under [A1] and $\mathcal{A}$ is tight. Thus for every $\epsilon >0$ there is a compact $K\subseteq S$ such that ${\sup }_{\nu \in \mathcal{A}}\nu (K^c)<\epsilon$.

Fix $\eta >0$. Compactness gives a finite $\eta \mathrm{/}8$-net $x_1,\dots ,x_q$ of $K$. Quantize each value $g(x_i)\in [-1,1]$ into finitely many bins of width at most $\eta \mathrm{/}2$. There are finitely many resulting value patterns for the bounded 1-Lipschitz test class; choose one representative $h_r$ from each nonempty pattern. If $g$ and $h_r$ share a pattern, then for $x\in K$ a net point $x_i$ gives $\mathrm{\mid }g(x)-h_r(x)\mathrm{\mid }\le 2d(x,x_i)+\mathrm{\mid }g(x_i)-h_r(x_i)\mathrm{\mid }<\eta$.

For any admissible $g$, choose the representative $h_r$ from its pattern. The integral difference is bounded by $\mid \int gd{\mu }_n-\int gd\mu \mid \le {\max }_r\mid \int h_{r}d{\mu }_n-\int h_{r}d\mu \mid +2\eta +2{\mu }_n(K^c)+2\mu (K^c)\mathrm{.}$ The maximum tends to zero by [F1], because there are finitely many bounded continuous representatives, and the last two terms are at most $4\epsilon$ by step 2.2. Taking the supremum over $g$, then the limit superior in $n$, gives ${\mathrm{lim\hspace{0.17em}sup}}_n{\beta }_{BL}({\mu }_n,\mu )\le 2\eta +4\epsilon$. Letting $\eta ,\epsilon \downarrow 0$ proves the reverse implication.

For each coordinate $j$, the pair $({\theta }_j,X_j)$ is a measurable function of the independent pair $({\xi }_j,Z_j)$. Thus the coordinate pairs are independent: finite rectangle probabilities factor by [F1] and [F2]. Fix finite $J$. For any cylinder event $B$ depending on finitely many observations outside $J$, the random vector $({\theta }_J,X_J)$ is independent of $B$ by that same factorization. The events in $\sigma (X_{J^c})$ for which this independence identity holds form a lambda-system containing the outside-coordinate cylinder pi-system; [F3] extends it to all outside-data events. Hence conditioning on $X_{J^c}$ adds no information about ${\theta }_J$ once $X_J$ is known.

For finite $J$, the joint prior density of ${\theta }_J$ is the product of ${\sigma }_j^{-1}\phi (t_j\mathrm{/}{\sigma }_j)$, and the likelihood of $X_J=x_J$ is proportional to ${\prod }_{j\in J}{e}^{-n(x_j-t_j{)}^2\mathrm{/}2}$. Each one-dimensional normalizer is at most $1$ and is positive: the Gaussian factor is bounded by $1$, is strictly positive everywhere, and the prior density integrates to $1$. Thus the normalized product density is ${\prod }_{j\in J}{p}_{j,n}(t_j\mid x_j)$. Integrating it over rectangles in ${\theta }_J$ and $X_J$ verifies the conditional-law identity for these coordinates. The prior is a Borel probability on $L^2$ by [L1] under [A1], and [A2] supplies the full posterior kernel; step 1.1 extends the finite-coordinate identity to the full observation sequence.

The product of the one-coordinate conditional densities in step 2.1 is exactly the stated finite-dimensional posterior marginal, so the posterior factors by coordinates for every finite index set $J$.

Fix $(l,k)$ and abbreviate $\sigma ={\sigma }_l$, ${\theta }_0={\theta }_{0,lk}$, $r={\theta }_0\mathrm{/}\sigma$, and $\lambda =\sqrt{n}\sigma$. Under the true law write $X={\theta }_0+n^{-1\mathrm{/}2}Z$, where $Z$ has density $(2\pi {)}^{-1\mathrm{/}2}{e}^{-z^2\mathrm{/}2}$. If $B_{lk}(X)=\int (\theta -{\theta }_0{)}^2{\mathrm{\Pi }}_n(d{\theta }_{lk}\mid X)$, change variables $v=\sqrt{n}(\theta -{\theta }_0)$ in [F2] to obtain $n{B}_{lk}(X)=\frac{N_{\lambda }(Z)}{D_{\lambda }(Z)},D_{\lambda }(z)=\frac{1}{\lambda }{\int }_{\mathbb{R}}{e}^{-v^2\mathrm{/}2+zv}\phi (r+v\mathrm{/}\lambda )dv,N_{\lambda }(z)=\frac{1}{\lambda }{\int }_{\mathbb{R}}{v}^2{e}^{-v^2\mathrm{/}2+zv}\phi (r+v\mathrm{/}\lambda )dv\mathrm{.}$ All integrands are nonnegative and the denominator is positive. The posterior kernel is the $L^2$ model supplied by [F2] under [A2].

Set $h=(\tau -M)\mathrm{/}2>0$. If $\lambda \le S_0$ for a fixed $S_0>0$, then $\mathrm{\mid }r+v\mathrm{/}\lambda \mathrm{\mid }<\tau$ whenever $\mathrm{\mid }v\mathrm{\mid }\le h\lambda$. Substituting $v=\lambda t$ and pairing $t$ with $-t$ gives $D_{\lambda }(z)\ge c_{\phi }{\int }_{-h}^h{e}^{-{\lambda }^2{t}^2\mathrm{/}2+z\lambda t}dt=2c_{\phi }{\int }_0^h{e}^{-{\lambda }^2{t}^2\mathrm{/}2}\cosh (z\lambda t)dt\ge 2h{c}_{\phi }{e}^{-h^2{\lambda }^2\mathrm{/}2}\ge d_0:=2h{c}_{\phi }{e}^{-h^2{S}_0^2\mathrm{/}2}>0,$ because $\cosh (x)\ge 1$ and $e^{-{\lambda }^2{t}^2\mathrm{/}2}\ge e^{-h^2{\lambda }^2\mathrm{/}2}$ on $[0,h]$. Completing the square in the standard normal density gives $\mathbb{E}{e}^{Zv}=e^{v^2\mathrm{/}2}$; hence, by the substitution $u=r+v\mathrm{/}\lambda$, $\mathbb{E}{N}_{\lambda }(Z)=\frac{1}{\lambda }\int v^2\phi (r+v\mathrm{/}\lambda )dv={\lambda }^2\int (u-r{)}^2\phi (u)du\le 2{\lambda }^2(m_2+M^2)\mathrm{.}$ Here [F4] justifies the exchange of expectation and the nonnegative $v$-integral. Therefore $\mathbb{E}{B}_{lk}(X)\le C{\lambda }^2\mathrm{/}n=C{\sigma }_l^2\le C{S}_0^2\mathrm{/}n$ in this range.

If $\lambda \ge S_0$, let $R=h{S}_0$. For $\mathrm{\mid }v\mathrm{\mid }\le R$ one has $\mathrm{\mid }r+v\mathrm{/}\lambda \mathrm{\mid }<\tau$, so $D_{\lambda }(z)\ge \frac{c_{\phi }{e}^{z^2\mathrm{/}2}}{\lambda }{I}_R(z),N_{\lambda }(z)\le \frac{C_{\phi }{e}^{z^2\mathrm{/}2}}{\lambda }J(z),$ where $I_R(z)={\int }_{-R}^R{e}^{-(v-z{)}^2\mathrm{/}2}dv$ and $J(z)={\int }_{\mathbb{R}}{v}^2{e}^{-(v-z{)}^2\mathrm{/}2}dv=\sqrt{2\pi }(1+z^2)$. Thus $n{B}_{lk}(X)\le (C_{\phi }\mathrm{/}{c}_{\phi })J(Z)\mathrm{/}{I}_R(Z)$. For $\mathrm{\mid }z\mathrm{\mid }\le R$, integrate $I_R$ over the half-interval nearest $z$; for $\mathrm{\mid }z\mathrm{\mid }>R$, integrate over the half-interval adjacent to the nearer endpoint. These choices give the common lower bound $I_R(z)\ge \frac{R}{2}\exp (-\frac{R^2}{2}-\frac{(\mathrm{\mid }z\mathrm{\mid }-R\mathrm{/}2{)}_{+}^2}{2})\mathrm{.}$ Consequently $\mathbb{E}[J(Z)\mathrm{/}{I}_R(Z)]$ is finite: after multiplying by the standard normal density, its integrand is bounded on $\mathrm{\mid }z\mathrm{\mid }\le R\mathrm{/}2$ and is at most a constant times $(1+z^2)e^{-R\mathrm{\mid }z\mathrm{\mid }\mathrm{/}2}$ on $\mathrm{\mid }z\mathrm{\mid }>R\mathrm{/}2$. Hence $\mathbb{E}{B}_{lk}(X)\le C\mathrm{/}n$ also when $\lambda \ge S_0$, with $C$ independent of $(l,k,n)$.

The two ranges in steps 2.1 and 2.2 cover every $\lambda >0$; since ${\sigma }_l^2\le S_0^2\mathrm{/}n$ in the first range, there is a single constant $C$ for which ${\sup }_{l,k}{\mathbb{E}}_{f_0}^{(n)}{B}_{lk}(X)\le C\mathrm{/}n$.

By [F1], the posterior squared loss is the nonnegative series ${\sum }_{l,k}{w}_l^{\delta }{B}_{lk}(X)$. Apply monotone convergence [F3] to its finite partial sums and use step 3.1 to obtain ${\mathbb{E}}_{f_0}^{(n)}\int \mathrm{\parallel }f-f_0{\mathrm{\parallel }}_{H(\delta )}^2{\mathrm{\Pi }}_n(df\mid X)={\sum }_{l,k}{w}_l^{\delta }{\mathbb{E}}_{f_0}^{(n)}{B}_{lk}(X)\le \frac{C}{n}{\sum }_{l,k}{w}_l^{\delta }\mathrm{.}$ The trace on the right is finite for $\delta >1\mathrm{/}2$ by [F1], proving the bound.

Fix one coordinate $j=(l,k)$ and write $r={\theta }_{0,j}\mathrm{/}{\sigma }_j$. Under the true law, $X_j\mathrm{/}{\sigma }_j=r+Z_j\mathrm{/}(\sqrt{n}{\sigma }_j)\to r$ in probability, and $b_n=(\sqrt{n}{\sigma }_j{)}^{-1}\to 0$. By [F1], under [A1], the posterior density of $U_n=\sqrt{n}({\theta }_j-X_j)$ is $p_n(u\mid X_j)=\frac{e^{-u^2\mathrm{/}2}\phi (X_j\mathrm{/}{\sigma }_j+b_{n}u)}{c_n(X_j)},c_n(X_j)={\int }_{\mathbb{R}}{e}^{-v^2\mathrm{/}2}\phi (X_j\mathrm{/}{\sigma }_j+b_{n}v)dv\mathrm{.}$ For any deterministic sequence $x_n\mathrm{/}{\sigma }_j\to r$, boundedness of $\phi$ and [F2] give $c_n(x_n)\to \phi (r)\sqrt{2\pi }>0$.

Along such a deterministic sequence, $p_n(u\mid x_n)\to (2\pi {)}^{-1\mathrm{/}2}{e}^{-u^2\mathrm{/}2}$ pointwise and is eventually bounded by a constant multiple of $e^{-u^2\mathrm{/}2}$. A second application of [F2] gives convergence in $L^1(du)$, hence convergence in total variation. Since $X_j\mathrm{/}{\sigma }_j\to r$ in probability, if the total-variation convergence in probability failed, some subsequence would have error probability bounded below. Select a further subsequence for which $\mathbb{P}(\mathrm{\mid }{X}_j\mathrm{/}{\sigma }_j-r\mathrm{\mid }>2^{-q})\le 2^{-q}$ at its $q$th term; [F3] bounds each union of tail events by the corresponding geometric tail, which tends to zero, so the further subsequence converges almost surely to $r$, contradicting the deterministic conclusion.

For finite nonempty $J$, [F1] makes the scaled posterior and its Gaussian target product measures. Telescope the difference of their product densities one coordinate at a time. Integrating the absolute value of each term gives the $L^1$ distance of that coordinate pair, since every other coordinate density integrates to $1$. Thus the joint $L^1$ distance is at most the sum of the finitely many coordinate $L^1$ distances, which tends to zero in probability by step 2.1. For each measurable event $A$, the absolute difference of its probabilities is at most this $L^1$ distance, so the total-variation conclusion follows.

The positive continuous function $g$ is Borel measurable, since inverse images of open sets are open. For an integer $R\ge 1$, substitute $x=\sqrt{2}t$ in the continuous compact-interval integral. The bridge in [F1] and then increasing truncation in [F2] give ${\int }_{\mathbb{R}}{e}^{-x^2\mathrm{/}2}dx=\sqrt{2}{\lim }_{R\to \mathrm{\infty }}{\int }_{-R\mathrm{/}\sqrt{2}}^{R\mathrm{/}\sqrt{2}}{e}^{-t^2}dt=\sqrt{2\pi }$. Thus [F2] makes $\gamma$ a probability. For $v>0$, compact substitutions in the same way show that its affine pushforward has the asserted half-line integrals of $g_{m,v}$; limits of compact intervals give the half-lines, and [F5] gives equality of the measures. For $v=0$ the affine map is constant, giving the Dirac law directly.

The CDF of $N(m,v)$ for $v>0$ is continuous, since $g_{m,v}$ is bounded and the integral over an interval of length $h$ is at most $\mathrm{\mid }h\mathrm{\mid }\mathrm{/}\sqrt{2\pi v}$. It is strictly increasing because the positive continuous density has a positive minimum on every nondegenerate compact interval. For $Z$ with density $g$, $g^{\mathrm{\prime }}(x)=-xg(x)$, and integration by parts gives ${\int }_{-R}^R\mathrm{\mid }x\mathrm{\mid }g(x)dx=2(g(0)-g(R))$ and ${\int }_{-R}^R{x}^{2}g(x)dx={\int }_{-R}^{R}g(x)dx-2Rg(R)$. The exponential series shows $g(R)\to 0$ and $Rg(R)\to 0$. Increasing truncation gives $\mathbb{E}\mathrm{\mid }Z\mathrm{\mid }<\mathrm{\infty }$ and $\mathbb{E}{Z}^2=1$; symmetry gives $\mathbb{E}Z=0$. Expanding $m+\sqrt{v}Z$ gives the claimed mean and variance, also for $v=0$.

Let $U,V$ be independent centered density normal variables with variances $a,b>0$. Their sum has density $h(x)={\int }_{\mathbb{R}}{g}_{0,a}(u)g_{0,b}(x-u)du$. Indeed, [F4] expresses $P(U+V\le x)$ as the nonnegative double integral of ${\mathbf{1}}_{u+t\le x}{g}_{0,a}(u)g_{0,b}(t)$; for each fixed $u$, a translation in the continuous half-line integral followed by Tonelli gives ${\int }_{-\mathrm{\infty }}^{x}h(y)dy$. These half-line probabilities determine the law by [F5]. Completing the square gives $\frac{u^2}{a}+\frac{(x-u{)}^2}{b}=\frac{a+b}{ab}{(u-\frac{ax}{a+b})}^2+\frac{x^2}{a+b}\mathrm{.}$ The compact substitutions and increasing limits in step 1.1 therefore give $h(x)=g_{0,a+b}(x)$. If either variance is zero the corresponding variable is zero almost surely and the same conclusion is immediate. Scaling a coordinate by a negative coefficient gives the same density as scaling by its absolute value, by the symmetry of $g$. Finite induction, using mutual independence to separate the last coordinate from the preceding tuple, proves $S_m\sim N(0,v_m)$ with $v_m={\sum }_{j\le m}{c}_j^2$.

If $v=0$, every coefficient is zero, so convergence in probability implies $S=0$ almost surely. Suppose $v>0$. Then $v_m\to v$ and eventually $v_m>0$. Write $\mathrm{\Phi }$ for the continuous CDF of $\gamma$. For any $x\in \mathbb{R}$ and $\epsilon >0$, convergence in probability and step 3.1 give $\mathrm{\Phi }((x-\epsilon )\mathrm{/}\sqrt{v_m})-P(\mathrm{\mid }{S}_m-S\mathrm{\mid }>\epsilon )\le P(S\le x)\le \mathrm{\Phi }((x+\epsilon )\mathrm{/}\sqrt{v_m})+P(\mathrm{\mid }{S}_m-S\mathrm{\mid }>\epsilon )\mathrm{.}$ First let $m\to \mathrm{\infty }$, then $\epsilon \downarrow 0$. Continuity from step 2.1 gives $P(S\le x)=\mathrm{\Phi }(x\mathrm{/}\sqrt{v})$ for every $x$. By [F5], $S$ has law $N(0,v)$. Almost sure convergence implies convergence in probability by dominated convergence on the indicators of $\{\mathrm{\mid }{S}_m-S\mathrm{\mid }>\epsilon \}$, and $L^2$ convergence implies it by ${\epsilon }^{2}P(\mathrm{\mid }{S}_m-S\mathrm{\mid }>\epsilon )\le \mathbb{E}\mathrm{\mid }{S}_m-S{\mathrm{\mid }}^2$.