Linear Gaussian inverse posteriors and Tikhonov

By [F1], choose an orthonormal eigenbasis of $C$ and order it so the positive eigenvalues are ${\lambda }_1,\dots ,{\lambda }_r$. Let $U$ have the corresponding eigenvectors as columns and set $\mathrm{\Lambda }=\mathrm{diag}({\lambda }_1,\dots ,{\lambda }_r)$. The zero eigenspace is the orthogonal complement of these columns, so $C=U\mathrm{\Lambda }{U}^T$ and $\mathrm{Ran}(C)=\mathrm{Ran}(U)$. When $r=0$, the eigenbasis gives $C=0$.

If $r>0$, set $P=U{\mathrm{\Lambda }}^{-1}{U}^T$. The decomposition in step 1.1 gives $CP=PC=U{U}^T$, hence $CPC=C$ and $PCP=P$; also $CP$ and $PC$ are symmetric. Thus $P$ satisfies all four equations in [F5], so uniqueness gives $P=C^{†}$. If $r=0$, the zero matrix satisfies the four equations for $C=0$, and uniqueness gives $C^{†}=0$.

Suppose $r>0$. By [F2], the square root is $C^{1\mathrm{/}2}=U{\mathrm{\Lambda }}^{1\mathrm{/}2}{U}^T$. By [F3], $X$ has the law of $m+U{\mathrm{\Lambda }}^{1\mathrm{/}2}{U}^T{Z}_n$ for a standard-normal vector $Z_n$. For every $a\in {\mathbb{R}}^r$, the scalar $a^T{U}^T{Z}_n=(Ua{)}^T{Z}_n$ is centered normal with variance $\mathrm{\parallel }Ua{\mathrm{\parallel }}_2^2=\mathrm{\parallel }a{\mathrm{\parallel }}_2^2$, so $U^T{Z}_n$ has law $N_r(0,I_r)$. Therefore $X$ has the stated representation. When $r=0$, [F3] makes $X=m$ almost surely.

If $r>0$, let $x=m+U{\mathrm{\Lambda }}^{1\mathrm{/}2}z$ be any point of $m+\mathrm{Ran}(C)$. The continuous map $t\mapsto m+U{\mathrm{\Lambda }}^{1\mathrm{/}2}t$ sends $z$ to $x$, so the preimage of every open ball about $x$ contains an open ball about $z$. By [F4], $N_r(0,I_r)$ assigns positive measure to every nonempty open ball. The representation in step 2.2 therefore gives positive ${\mu }_X$-mass to every neighborhood of $x$. Conversely, the law in step 2.2 is concentrated on the finite-dimensional closed affine subspace $m+\mathrm{Ran}(C)$; every point outside it has a neighborhood of zero mass. Hence the support is exactly that subspace. If $r=0$, the point-mass law from step 2.2 has support $\{m\}$.

For $v\in {\mathbb{R}}^m\setminus \{0\}$, $v^{T}Sv=(A^{T}v{)}^T{C}_0(A^{T}v)+v^T\mathrm{\Gamma }v>0$, since the first term is nonnegative and the second is positive. Thus $S\succ 0$ and $S^{-1}$ exists.

Let $T$ be the finite index set with state indices $s_1,\dots ,s_n$ and observation indices $o_1,\dots ,o_m$. Define $f_{s_i}=X_i$ and $f_{o_j}=(AX{)}_j$. Every finite linear combination of these coordinates is a linear projection of $X$, so it is normal by [F1] and $f$ is a Gaussian process by [F3]. Its covariance blocks are $K_{SS}=C_0$, $K_{SO}=C_0{A}^T$, and $K_{OO}=A{C}_0{A}^T$.

For each fixed $y$, ${\mathrm{\Phi }}_y$ is finite, continuous, and nonnegative on ${\mathbb{R}}^n$, and $(x,y)\mapsto {\mathrm{\Phi }}_y(x)$ is jointly continuous. The prior ${\mu }_0$ is a probability on the separable Banach space ${\mathbb{R}}^n$, and the envelope $M_1(r)=0$ satisfies $\int e^{M_1(\mathrm{\parallel }x{\mathrm{\parallel }}_2)}{\mu }_0(dx)=1$. Therefore [F9] gives $0<Z(y)<\mathrm{\infty }$ for every $y$, and [F8] defines ${\mathrm{\Pi }}^y$. The likelihood density from [F7] is a positive constant independent of $x$ times $e^{-{\mathrm{\Phi }}_y(x)}$; its constant cancels in the normalized update.

The vector $f$ is a measurable function of $X$, so [F4] and [F5] make $\eta$ independent of $f$. The observed covariance is $K_{OO}+\mathrm{\Gamma }=S\succ 0$ by step 1.1. Applying [F6] with test list $S$ and observation list $O$ gives the displayed Gaussian conditional-law version for every $y$.

Suppose $r=\mathrm{rank}(C_0)>0$. By [F2], write $C_0=U\mathrm{\Lambda }{U}^T$ and represent $X$ as $m_0+L\xi$, where $L=U{\mathrm{\Lambda }}^{1\mathrm{/}2}$ and $\xi \sim N_r(0,I_r)$. Put $B=AL$ and $d=y-A{m}_0$. By the pushforward formula for ${\mu }_0$ and the likelihood in [F7], the density of the normalized update in these coordinates is proportional, for every $y$, to $\exp \text{\hspace{0.17em}⁣}(-\frac{1}{2}[{\xi }^T\xi +(d-B\xi {)}^T{\mathrm{\Gamma }}^{-1}(d-B\xi )])\mathrm{.}$ This is a pointwise fixed-data calculation; it does not replace the all-data version convention in step 1.3.

Set $Q=I_r+B^T{\mathrm{\Gamma }}^{-1}B$ and $h=B^T{\mathrm{\Gamma }}^{-1}d$. Then $Q\succ 0$, and expansion gives ${\xi }^T\xi +(d-B\xi {)}^T{\mathrm{\Gamma }}^{-1}(d-B\xi )=(\xi -Q^{-1}h{)}^{T}Q(\xi -Q^{-1}h)+d^T{\mathrm{\Gamma }}^{-1}d-h^T{Q}^{-1}h\mathrm{.}$ Thus the update in step 2.2 is the Gaussian law of $\xi$ with mean $Q^{-1}h$ and covariance $Q^{-1}$, so the update of $X=m_0+L\xi$ has mean $m_0+L{Q}^{-1}{B}^T{\mathrm{\Gamma }}^{-1}d$ and covariance $L{Q}^{-1}{L}^T$.

Since $S=\mathrm{\Gamma }+B{B}^T$, multiplication gives $Q{B}^T=B^T{\mathrm{\Gamma }}^{-1}S$, hence $Q^{-1}{B}^T{\mathrm{\Gamma }}^{-1}=B^T{S}^{-1}$. Also $Q(I_r-B^T{S}^{-1}B)=Q-Q{B}^T{S}^{-1}B=I_r$, so $Q^{-1}=I_r-B^T{S}^{-1}B$. Substituting these identities into step 3.1 yields $m_0+L{Q}^{-1}{B}^T{\mathrm{\Gamma }}^{-1}d=m_0+C_0{A}^T{S}^{-1}d$ and $L{Q}^{-1}{L}^T=C_0-C_0{A}^T{S}^{-1}A{C}_0\mathrm{.}$ These are exactly the mean and covariance in the Statement.

If $r=0$, [F2] gives $C_0=0$ and $X=m_0$ almost surely. Reweighting a point mass by the positive likelihood leaves that point mass, while the displayed formulas reduce to mean $m_0$ and covariance $0$. This also matches the conditional version from step 2.1 and proves the rank-zero boundary. Thus the fixed-data update and the stated all-data conditional version have the displayed law in every rank case.

If $r=0$, [F2] gives $C_0=0$ and $S_0=\{m_0\}$. The posterior is the point mass at $m_0$ by [F1], so its mean and unique MAP point are $m_0$. The objective $J_y$ has only this admissible point, hence it has the same unique minimizer on $S_0$.

Suppose $r>0$. By [F2], choose $U,\mathrm{\Lambda }$ with $C_0=U\mathrm{\Lambda }{U}^T$, $C_0^{†}=U{\mathrm{\Lambda }}^{-1}{U}^T$, and write $L=U{\mathrm{\Lambda }}^{1\mathrm{/}2}$. The map $z\mapsto m_0+Lz$ is a bijection from ${\mathbb{R}}^r$ onto $S_0$. For $x=m_0+Lz$, $(x-m_0{)}^T{C}_0^{†}(x-m_0)=z^{T}z$.

Let $W$ be the non-negative square root of ${\mathrm{\Gamma }}^{-1}$. Since $\mathrm{\Gamma }\succ 0$, $W$ is symmetric positive definite and $W^{T}W={\mathrm{\Gamma }}^{-1}$ by [F4]. Set $D=WAL$ and $b=W(y-A{m}_0)$. Substituting $x=m_0+Lz$ into the objective in the Statement gives $J_y(m_0+Lz)=\mathrm{\parallel }Dz-b{\mathrm{\parallel }}_2^2+\mathrm{\parallel }z{\mathrm{\parallel }}_2^2$. This is the Tikhonov objective in [F5] with $\lambda =1$, so [F6] gives the unique minimizer $\widehat{z}=(D^{T}D+I_r{)}^{-1}{D}^{T}b$.

By the Gaussian representation in [F2], the prior coordinate $z$ has standard Gaussian density. The likelihood is proportional to $\exp (-\frac{1}{2}\mathrm{\parallel }Dz-b{\mathrm{\parallel }}_2^2)$ by [F7]. Thus the posterior density in $z$ is proportional to $\exp [-\frac{1}{2}(\mathrm{\parallel }Dz-b{\mathrm{\parallel }}_2^2+\mathrm{\parallel }z{\mathrm{\parallel }}_2^2)]$. Its precision matrix $H=I_r+D^{T}D$ is positive definite, and completing the square gives a Gaussian density with mean $H^{-1}{D}^{T}b=\widehat{z}$ and covariance $H^{-1}$. This density has its unique maximum at its mean. The affine bijection in step 1.2 has constant positive Jacobian in orthonormal coordinates on $S_0$, so its image has the unique MAP point $\widehat{x}=m_0+L\widehat{z}$. Its posterior mean is also $m_0+L\widehat{z}$, and step 2.1 makes this point the unique minimizer of $J_y$ on $S_0$.