Small-ball MAP and Cameron–Martin geometry — Examples and Counterexamples

By [F1], the prior and noise have positive continuous Gaussian densities proportional to $\exp (-\frac{1}{2}{u}^T{C}^{-1}u)$ and $\exp (-\frac{1}{2}(y-Au{)}^T{\mathrm{\Gamma }}^{-1}(y-Au))$, respectively. For any $w$, $w^T{\mathrm{\Gamma }}^{-1}w\ge 0$: write $w=\mathrm{\Gamma }v$, so $w^T{\mathrm{\Gamma }}^{-1}w=v^T\mathrm{\Gamma }v\ge 0$. Thus ${\mathrm{\Phi }}_y\ge 0$ and $e^{-{\mathrm{\Phi }}_y}\le 1$, so the evidence is finite by [F6] and the fact that the prior is a probability. Write $f_0$ for the prior density. The function $f_0$ is positive and continuous at $0$, and $e^{-{\mathrm{\Phi }}_y}$ is also positive and continuous there. On a sufficiently small open ball $O$ about $0$, there are constants $c_0,c_1>0$ with $f_0\ge c_0$ and $e^{-{\mathrm{\Phi }}_y}\ge c_1$. By [F5], $\lambda (O)>0$; [F6] then gives $N(0,C)(O)={\int }_O{f}_{0}d\lambda \ge c_0\lambda (O)>0$ and $Z\ge c_{1}N(0,C)(O)>0$. The posterior update [F1] is therefore defined. Its Lebesgue density is proportional to the product of the prior density and likelihood, hence to $e^{-I_y(u)}$. Put $Q=C^{-1}+A^T{\mathrm{\Gamma }}^{-1}A$. For $v\mathrm{\ne }0$, $v^{T}Qv=v^T{C}^{-1}v+(Av{)}^T{\mathrm{\Gamma }}^{-1}(Av)>0$: if $z=C^{-1}v$, then $v^T{C}^{-1}v=z^{T}Cz>0$, and the second term is nonnegative by the preceding argument. Thus $Q$ is positive definite. Completing the square gives the unique minimizer $m=Q^{-1}{A}^T{\mathrm{\Gamma }}^{-1}y$, and the posterior is the nondegenerate Gaussian $N(m,Q^{-1})$.

For every $r>0$, apply [F2] to this posterior and the loss $L_r(x)={\mathbf{1}}_{\{\mathrm{\parallel }x{\mathrm{\parallel }}_2>r\}}$. Its sublevel sets are the closed radius-$r$ ball and the whole space, so it is bounded, symmetric and quasi-convex. Thus every ball of radius $r$ has posterior mass at most the ball centered at $m$, and the supremum in [F3] is attained at $m$. Consequently $m$ is a strong small-ball MAP.

Let $q$ be the positive continuous posterior density. By [F5], every closed Euclidean ball has positive finite Lebesgue measure; the inclusions $B(z,r)\subseteq \bar{B}(z,r)\subseteq B(z,2r)$ and the open-ball result in [F5] give the volume bounds, while [F4] makes the volumes of same-radius balls equal. For any fixed $z\in {\mathbb{R}}^d$, the absolute difference between $\lambda (\bar{B}(z,r){)}^{-1}\mathrm{\Pi }(\bar{B}(z,r))$ and $q(z)$ is at most ${\sup }_{\mathrm{\parallel }x-z{\mathrm{\parallel }}_2\le r}\mathrm{\mid }q(x)-q(z)\mathrm{\mid }$, by [F6]; this tends to zero by continuity. The same holds at any $v$, and the common ball volume cancels; hence $\mathrm{\Pi }(\bar{B}(z,r))\mathrm{/}\mathrm{\Pi }(\bar{B}(v,r))\to q(z)\mathrm{/}q(v)$. Since $q$ is proportional to $e^{-I_y}$, the finite-dimensional small-ball ratio agrees with the density and Onsager–Machlup comparison.

In [F1], take $H_0=H_1={\mathrm{\ell }}^2$, $w_j=1$, and ${\lambda }_j=1\mathrm{/}(j(j+1))$. The finite sums telescope as ${\sum }_{j=1}^N{\lambda }_j={\sum }_{j=1}^N(1\mathrm{/}j-1\mathrm{/}(j+1))=1-1\mathrm{/}(N+1)\le 1$, so the required infinite sum is finite. Thus [F1] applies to the stated independent standard normals and proves that $W$ is a centered Borel Gaussian random element.

Suppose that $\lambda$ is a nonzero locally finite translation-invariant Borel measure, where locally finite means that every point has a neighborhood of finite measure. At zero choose $R>0$ with $\lambda (B(0,R))<\mathrm{\infty }$. Put $r=R\mathrm{/}8$ and $U_j=B((R\mathrm{/}2)e_j,r)$. For $j\mathrm{\ne }k$ the distance between the centers is $R\mathrm{/}\sqrt{2}>2r$, so these balls are pairwise disjoint, and every $U_j$ lies in $B(0,R)$ because $R\mathrm{/}2+r<R$. Translation invariance gives a common finite mass $m=\lambda (U_j)$. If $m>0$, finite additivity and monotonicity give $Nm\le \lambda (B(0,R))$ for every $N$, a contradiction. Hence $m=0$. This covers in particular every measure finite on all bounded Borel sets, since that condition implies local finiteness.

Translation invariance and step 1.2 make every radius-$r$ ball have measure zero. Let $D$ be the set of finitely supported rational sequences. [F3] makes $D$ countable: it is the countable union, over support lengths $N$, of the countable set ${\mathbb{Q}}^N$ padded with zeros. It is dense in ${\mathrm{\ell }}^2$: for any $x\in {\mathrm{\ell }}^2$ and $ϵ>0$, choose $N$ so that the squared tail sum is less than ${ϵ}^2\mathrm{/}4$, then approximate the first $N$ coordinates by rationals with squared total error less than ${ϵ}^2\mathrm{/}4$; the resulting padded vector is within $ϵ$ of $x$. The radius-$r$ balls centered at $D$ therefore form a countable cover of ${\mathrm{\ell }}^2$. By [F2], $\lambda ({\mathrm{\ell }}^2)=0$, and [F4] makes every Borel subset null. This contradicts that $\lambda$ is nonzero. Hence no such reference measure exists, refuting the statement; in particular, the displayed Gaussian has no density relative to one.

For $k=1,2,3,4$, put $a_k={\int }_{[0,1]}{\theta }^{k}d\pi$. These bounded continuous functions are integrable because $0\le {\theta }^k\le 1$ on the support of the probability $\pi$. For each $N\ge 1$, partition $[0,1]$ into $I_{j,N}=[j\mathrm{/}N,(j+1)\mathrm{/}N)$ for $j<N-1$ and $I_{N-1,N}=[(N-1)\mathrm{/}N,1]$. The simple lower and upper step functions with values $(j\mathrm{/}N{)}^k$ and $((j+1)\mathrm{/}N{)}^k$ on $I_{j,N}$ bound ${\theta }^k$ pointwise; [F3] gives $\pi (I_{j,N})=1\mathrm{/}N$, and [F4] therefore gives $L_{N,k}:=N^{-(k+1)}{\sum }_{j=0}^{N-1}{j}^k\le a_k\le N^{-(k+1)}{\sum }_{j=1}^N{j}^k=:U_{N,k}\mathrm{.}$ Their difference is $U_{N,k}-L_{N,k}=1\mathrm{/}N$. Also the binomial identity telescopes to $N^{k+1}=(k+1){\sum }_{j=0}^{N-1}{j}^k+{\sum }_{i=0}^{k-1}\left(\genfrac{}{}{0px}{}{k+1}{i}\right){\sum }_{j=0}^{N-1}{j}^i\mathrm{.}$ For each $i<k$, ${\sum }_{j=0}^{N-1}{j}^i\le N^{i+1}$, so after division by $N^{k+1}$ all terms in the second sum tend to zero. Hence $L_{N,k}\to 1\mathrm{/}(k+1)$, and the squeeze gives $a_k=1\mathrm{/}(k+1)$. Also ${\int }_{[0,1]}1d\pi =1$.

On $[0,1]$, $p(\theta )={\theta }^2(1-\theta )\ge 0$. Its derivative is $p^{\mathrm{\prime }}(\theta )=\theta (2-3\theta )$, so its maximum is $p(2\mathrm{/}3)=4\mathrm{/}27<1$. Therefore $p$ and $1-p$ define valid conditional probabilities for $Y=1$ and $Y=0$.

By [F1] and [F5], the evidence for $Y=1$ is $Z={\int }_{[0,1]}p(\theta )d\pi =\int {\theta }^{2}d\pi -\int {\theta }^{3}d\pi =\frac{1}{3}-\frac{1}{4}=\frac{1}{12}>0,$ using step 1.1. The posterior formula [F1] gives density $p\mathrm{/}Z=12{\theta }^2(1-\theta )$ relative to $\pi$. Since $\pi$ has Lebesgue density ${\mathbf{1}}_{[0,1]}$, the posterior Lebesgue density is $q(\theta )=12{\theta }^2(1-\theta )$ on $[0,1]$, extended by zero on $\mathbb{R}\setminus [0,1]$.

On $(0,1)$, $q^{\mathrm{\prime }}(\theta )=12\theta (2-3\theta )$, so $q$ increases up to $2\mathrm{/}3$ and decreases after it; $q(0)=q(1)=0$. Thus $q_{\max }=q(2\mathrm{/}3)>0$ is the unique density maximum. For any center $x\in \mathbb{R}$ and $r>0$, the posterior density relative to $\pi$ and [F3] give $m_r(x)\le q_{\max }\pi ([x-r,x+r]\cap [0,1])\le 2r{q}_{\max }$. For $0<r<1\mathrm{/}3$, $[2\mathrm{/}3-r,2\mathrm{/}3+r]\subseteq [0,1]$ and its $\pi$-mass is $2r$. Continuity of $q$ then gives $\mid \frac{m_r(2\mathrm{/}3)}{2r}-q_{\max }\mid \le {\sup }_{\mathrm{\mid }\theta -2\mathrm{/}3\mathrm{\mid }\le r}\mathrm{\mid }q(\theta )-q_{\max }\mathrm{\mid }⟶0.$ Hence $1\ge \frac{m_r(2\mathrm{/}3)}{{\sup }_{x\in \mathbb{R}}{m}_r(x)}\ge \frac{m_r(2\mathrm{/}3)}{2r{q}_{\max }}⟶1.$ By [F2], $2\mathrm{/}3$ is a strong small-ball MAP.

By [F1], [F5], and steps 1.1 and 2.1, the posterior mean is ${\int }_{[0,1]}\theta q(\theta )d\pi =12(\int {\theta }^{3}d\pi -\int {\theta }^{4}d\pi )=12(\frac{1}{4}-\frac{1}{5})=\frac{3}{5}\mathrm{.}$ Since $3\mathrm{/}5\mathrm{\ne }2\mathrm{/}3$, the posterior mean differs from the strong small-ball MAP. Continuity at $3\mathrm{/}5$ gives $m_r(3\mathrm{/}5)\mathrm{/}(2r)\to q(3\mathrm{/}5)$, while step 3.1 gives ${\sup }_x{m}_r(x)\mathrm{/}(2r)\to q_{\max }$. Consequently $\frac{m_r(3\mathrm{/}5)}{{\sup }_x{m}_r(x)}⟶\frac{q(3\mathrm{/}5)}{q(2\mathrm{/}3)}=\frac{216\mathrm{/}125}{16\mathrm{/}9}=\frac{243}{250}<1,$ so the posterior mean is not a strong small-ball MAP.