Gaussian sequence inversion and exact risk

Assume the Axiom of Countable Choice. Let $X$ and $Y$ be real Hilbert spaces and let $K:X\to Y$ be a compact linear operator of infinite rank, viewed as an inverse model with exact datum $Kx$. Here compact means that the image of every bounded subset has compact closure. Write $N(K)=\{x\in X:Kx=0\}$ and $H_0=\overline{\mathrm{ran}K}$. Fix a positive singular system $(s_j,v_j,u_j{)}_{j\in {\mathbb{N}}_{>0}}$ for $K$, where ${\mathbb{N}}_{>0}=\{1,2,3,\dots \}$, $s_1\ge s_2\ge \cdots >0$, $s_j\to 0$, $K{v}_j=s_j{u}_j$, $(v_j)$ is a complete orthonormal basis of $N(K{)}^{\perp }$, and $(u_j)$ is a complete orthonormal basis of $H_0$. A complete orthonormal basis is an orthonormal family whose closed linear span is the whole indicated subspace. These are the compact SVD conventions; in particular, this model concerns minimum-norm representatives in $N(K{)}^{\perp }$ and does not estimate a kernel component.

For a truth $x\in N(K{)}^{\perp }$, let ${\theta }_j=⟨x,v_j{⟩}_X$. The complete-basis coefficient map identifies $N(K{)}^{\perp }$ bijectively and isometrically with ${\mathrm{\ell }}^2({\mathbb{N}}_{>0},\mathbb{R})$, so every $\theta \in {\mathrm{\ell }}^2({\mathbb{N}}_{>0},\mathbb{R})$ is a model parameter. For $ϵ>0$, take the functional white-noise experiment on the separable Hilbert space $H_0$ with observations $Y_x(g)=⟨Kx,g{⟩}_Y+ϵW(g)$, where $W$ is centered, has covariance $\mathbb{E}[W(g)W(h)]=⟨g,h{⟩}_Y$, and has independent standard-normal values on every finite orthonormal tuple. The countable complete basis makes $H_0$ separable. Define ${\xi }_j=W(u_j)$ and $Z_j=Y_x(u_j)\mathrm{/}{s}_j$. The sequence sample space is ${\mathbb{R}}^{{\mathbb{N}}_{>0}}$ with the product Borel sigma-algebra, meaning the sigma-algebra generated by its coordinate projections. The coordinate law is the pushforward of the functional experiment under $y\mapsto (y(u_j)\mathrm{/}{s}_j{)}_{j\in {\mathbb{N}}_{>0}}$; this map is measurable because each coordinate is a cylinder evaluation followed by scalar multiplication. The compact SVD expansion gives $⟨Kx,u_j{⟩}_Y=s_j{\theta }_j$, and the positive-noise white-noise coordinate theorem gives the independent standard-normal coordinates ${\xi }_j$. Hence the normalized coordinates satisfy $Z_j={\theta }_j+ϵs_j^{-1}{\xi }_j,j\in {\mathbb{N}}_{>0}\mathrm{.}$

For the zero-noise boundary $ϵ=0$, define the sequence experiment separately by the point mass at $\theta$, so $Z_j={\theta }_j$. This boundary is not an instance of the positive-noise functional experiment.

Compact linear inverse model between Hilbert spaces

Compact-operator singular system and expansion

Complete orthonormal systems and Hilbert bases

A complete orthonormal basis identifies a Hilbert space isometrically with ${\mathrm{\ell }}^2$

Square-summable families and ${\mathrm{\ell }}^2$

Functional isonormal white-noise experiment

Independent Gaussian coordinates in a white-noise experiment

The Axiom of Countable Choice

Litao Ding and Peter Mathé, Minimax rates for statistical inverse problems under general source conditions, §1, equations (1)–(4), PDF pp. 1–2

Iain M. Johnstone, Gaussian estimation: Sequence and wavelet models, Chapter 3, equations (3.63)–(3.67), printed p. 78; §3.8 begins on p. 77

The deterministic square commutes: ${\mathrm{\Phi }}_X(x)=(⟨x,v_j⟩{)}_j$ and ${\mathrm{\Phi }}_{H_0}(y)=(⟨y,u_j⟩{)}_j$ are the coefficient isometries, and $D_s(\theta )=(s_j{\theta }_j{)}_j$ is well defined because $s_j\le s_1$. The compact SVD expansion gives ${\mathrm{\Phi }}_{H_0}(Kx)=D_s({\mathrm{\Phi }}_X(x))$. The noisy coordinates add $ϵ{\xi }_j$ before division by $s_j$; the diagram is not a claim that white noise is an $H_0$-valued sample and is not used as a proof.

Let $Q\in [0,\mathrm{\infty })$ and let $a=(a_j{)}_{j\in {\mathbb{N}}_{>0}}$ be a nondecreasing sequence of positive real numbers, so $0<a_1\le a_2\le \cdots$. The statistical ellipsoid of radius $Q$ and weights $a$ in the Gaussian sequence inverse model is ${\mathrm{\Theta }}_a(Q)=\{\theta \in {\mathrm{\ell }}^2({\mathbb{N}}_{>0},\mathbb{R}):{\sum }_{j=1}^{\mathrm{\infty }}{a}_j^2{\theta }_j^2\le Q^2\}\mathrm{.}$ No assumption that $a_j\to \mathrm{\infty }$ is included. In particular, this definition does not assert that the ellipsoid is compact in ${\mathrm{\ell }}^2$.

Gaussian sequence inverse model

Square-summable sequence convention

Litao Ding and Peter Mathé, Minimax rates for statistical inverse problems under general source conditions, Definition 1, equation (4), PDF p. 1

Iain M. Johnstone, Gaussian estimation: Sequence and wavelet models, Chapter 3, ellipsoid definition (3.4), printed p. 58; §3.1 begins on p. 57

Input a normalized sequence observation $z=(z_j{)}_{j\in {\mathbb{N}}_{>0}}\in {\mathbb{R}}^{{\mathbb{N}}_{>0}}$ from the Gaussian sequence inverse model and a cutoff $D\in {\mathbb{N}}_0$, where ${\mathbb{N}}_0=\{0,1,2,\dots \}$. Output the finite-support estimate ${\widehat{\theta }}_D(z)\in {\mathrm{\ell }}^2({\mathbb{N}}_{>0},\mathbb{R})$ defined by ${\widehat{\theta }}_{D,j}(z)=\{\begin{array}{ll}{z}_j,& 1\le j\le D,\\ 0,& j>D\mathrm{.}\end{array}$ For $D=0$ the output is the zero sequence.

Gaussian sequence inverse model

Litao Ding and Peter Mathé, Minimax rates for statistical inverse problems under general source conditions, equation (11), PDF p. 3

Iain M. Johnstone, Gaussian estimation: Sequence and wavelet models, §3.2, equation (3.12), printed p. 61

Assume Countable Choice and let $ϵ\ge 0$. In the Gaussian sequence inverse model with positive singular numbers $(s_j{)}_{j\in {\mathbb{N}}_{>0}}$, let $Q\ge 0$ and let $(a_j{)}_{j\in {\mathbb{N}}_{>0}}$ be positive and nondecreasing. For each $\theta \in {\mathrm{\Theta }}_a(Q)$, define the squared sequence risk of the cutoff estimator at $D\in {\mathbb{N}}_0$ by $R_{\theta }(D):={\mathbb{E}}_{\theta }\text{\hspace{0.17em}⁣}[\mathrm{\parallel }{\widehat{\theta }}_D(Z)-\theta {\mathrm{\parallel }}_{{\mathrm{\ell }}^2}^2]\mathrm{.}$ Then its exact worst-case squared sequence risk is ${\sup }_{\theta \in {\mathrm{\Theta }}_a(Q)}{R}_{\theta }(D)=\frac{Q^2}{a_{D+1}^2}+{ϵ}^2{\sum }_{j=1}^D{s}_j^{-2},$ where the sum is $0$ for $D=0$. The supremum is attained by the one-coordinate truth ${\theta }^{\star }=(Q\mathrm{/}{a}_{D+1})e_{D+1}$, with ${\theta }^{\star }=0$ when $Q=0$.

Litao Ding and Peter Mathé, Minimax rates for statistical inverse problems under general source conditions, Lemma 1 and its complete proof, PDF pp. 2–3

Iain M. Johnstone, Gaussian estimation: Sequence and wavelet models, §3.2, equations (3.11)–(3.12), printed pp. 60–61

This is expected squared loss in the Gaussian sequence experiment, over a weighted ellipsoid of coefficient sequences. It is not a deterministic worst-case error bound for arbitrary norm-bounded data perturbations.