Adaptive spectral regularization by SOLIT

Assume Countable Choice. Let $({\xi }_j{)}_{j\ge 1}$ be given real random variables on a probability space such that every finite subfamily is independent and each variable has density $(2\pi {)}^{-1\mathrm{/}2}{e}^{-z^2\mathrm{/}2}$. Let $a_j\ge 0$ be deterministic with $v:={\sum }_j{a}_j<\mathrm{\infty }$, and put $u:={\sup }_j{a}_j$ and $w:={\sum }_j{a}_j^2$. The nonnegative series $Q:={\sum }_j{a}_j{\xi }_j^2$ is finite almost surely. Define $N:=\sqrt{Q}$ on that event and set $N=0$ on its null complement. Then $\mathbb{E}{N}^2=v,\mathbb{E}{N}^4=v^2+2w\le 3v^2\mathrm{.}$ For every $t>0$, $\mathbb{P}\{N^2>v+2\sqrt{wt}+2ut\}\le e^{-t},\mathbb{P}\{N>\sqrt{v}+\sqrt{2ut}\}\le e^{-t}\mathrm{.}$ The upper quantile $q(t):=\inf \{z\ge 0:\mathbb{P}(N>z)\le e^{-t}\}$ is finite and satisfies $\mathbb{P}(N>q(t))\le e^{-t},q(t)\le \sqrt{v}+\sqrt{2ut}\mathrm{.}$ These conclusions include $v=0$, in which case $N=q(t)=0$ almost surely. For a Gaussian series in a fixed orthonormal coordinate system with coordinate variances $a_j$, $N$ is its norm.

Countable Choice

Monotone convergence for nonnegative integrals

Markov's inequality

Spokoiny and Willrich, Bootstrap tuning in ordered model selection, Appendix D

Assume Countable Choice. Let $T:X\to Y$ be an injective compact linear operator of infinite rank between separable real Hilbert spaces (each with a countable dense subset), and assume $T$ is Hilbert–Schmidt, meaning that for its positive singular values $s_j$ the sum ${\sum }_{j\ge 1}{s}_j^2$ is finite. Put $S=T^{\ast }T$, ${\lambda }_j=s_j^2$, and ${\lambda }_1=\mathrm{\parallel }S\mathrm{\parallel }$, with the positive eigenvalues listed with multiplicity in nonincreasing order. Let $\phi :[0,\mathrm{\infty })\to [0,\mathrm{\infty })$ be continuous and strictly increasing with $\phi (0)=0$, and take its inverse on its range. Assume the paper's convexity condition on the spectral range: $x\mapsto x{\phi }^{-1}(\sqrt{x})$ is convex for $0\le x\le \phi ({\lambda }_1{)}^2$. Define the source class ${\mathcal{W}}_{\phi }(\rho ):=\{\phi (S)w:w\in X,\mathrm{\parallel }w{\mathrm{\parallel }}_X\le \rho \}$ for $\rho >0$; equivalently, it is the source ball with singular-value index $\psi (t)=\phi (t^2)$.

An ordered spectral filter is a family of functions $q_{\alpha }:[0,{\lambda }_1]\to [0,\mathrm{\infty })$ indexed by a set $\mathcal{A}\subset (0,\mathrm{\infty })$ with $0$ as an accumulation point, such that for constants $C_q^{\mathrm{\prime }}<\mathrm{\infty }$ and $C_q^{\mathrm{\prime }\mathrm{\prime }}=1$,

and $q_{\alpha }(x)\le q_{\tilde{\alpha }}(x)$ whenever $\tilde{\alpha }<\alpha$, for every $x\in [0,{\lambda }_1]$. The index function $\phi$ is a qualification of this filter if for some $C_{\phi }<\mathrm{\infty }$,

For $\alpha \in \mathcal{A}$, define the variance function

For $\sigma >0$, a decreasing candidate sequence $({\alpha }_m{)}_{m\in {\mathbb{N}}_0}$ in $\mathcal{A}$ has geometric variance growth if $V_{T,q}({\alpha }_0)>0$ and there are constants $1<{\theta }_1\le {\theta }_2<\mathrm{\infty }$ such that

Set $v_m:={\sigma }^2{V}_{T,q}({\alpha }_m)$ and

the operator norm of the estimator-noise covariance in the singular basis. The bound $q_{{\alpha }_m}({\lambda }_j)\le C_q^{\mathrm{\prime }}\mathrm{/}{\alpha }_m$ and ${\sum }_j{\lambda }_j<\mathrm{\infty }$ make each $u_m$ and $v_m$ finite. The finite candidate set ends at

The geometric lower bound $v_m\ge v_0{\theta }_1^m$ makes this index finite.

The singular-value source ball convention and spectral multiplier for an index function

Compact spectral multipliers, source powers, and regularizing filters

Positive singular systems and the compact SVD

Countable Choice for the upstream countable singular-system construction

Li and Werner, Adaptive minimax optimality in statistical inverse problems via SOLIT, §§3.1.1–3.2.1

Clason, Regularization of Inverse Problems, complete lecture notes, Chapter 5 §§5.1–5.2

The source class agrees with the earlier singular-value convention because its multiplier at a singular value $s_j$ is $\psi (s_j)=\phi (s_j^2)=\phi ({\lambda }_j)$. The variance function is finite for every fixed $\alpha \in \mathcal{A}$: the filter bound gives $q_{\alpha }({\lambda }_j)\le C_q^{\mathrm{\prime }}\mathrm{/}\alpha$ and ${\sum }_j{\lambda }_j<\mathrm{\infty }$.

Assume Countable Choice and the positive-noise Gaussian sequence model with $\sigma >0$, a fixed complete positive singular system $(s_j,v_j,u_j)$ for an injective compact Hilbert–Schmidt operator $T:X\to Y$, and a truth with coordinates ${\theta }_j=⟨x,v_j{⟩}_X$ in ${\mathrm{\ell }}^2$. Write ${\lambda }_j=s_j^2$ and let ${\xi }_j$ be the independent standard-normal noise coordinates. Let $q_{\alpha }$ be nonnegative and satisfy $0\le {\lambda }_j{q}_{{\alpha }_m}({\lambda }_j)\le 1$, $q_{{\alpha }_m}({\lambda }_j)\le C_q^{\mathrm{\prime }}\mathrm{/}{\alpha }_m$, and $q_{{\alpha }_{m_1}}({\lambda }_j)\le q_{{\alpha }_{m_2}}({\lambda }_j)$ whenever $m_1<m_2$. For each candidate $m$, form the finite sums

The conclusion below includes the existence of their limit ${\widehat{x}}_m$ in $L^2(\mathrm{\Omega };X)$. Put $f_m:=\mathbb{E}{\widehat{x}}_m$, $B_m:=\mathrm{\parallel }{f}_m-x{\mathrm{\parallel }}_X$, and $v_m:=\mathbb{E}\mathrm{\parallel }{\widehat{x}}_m-f_m{\mathrm{\parallel }}_X^2$. Then ${\widehat{x}}_m$ is a measurable $X$-valued random element with finite second moment and

If $m_1<m_2$, then $B_{m_2}\le B_{m_1}$, $\mathrm{\parallel }{f}_{m_2}-f_{m_1}{\mathrm{\parallel }}_X\le B_{m_1}$, and

The ordered-filter bounds and candidate-grid ordering

The Gaussian sequence coordinate law

Independent standard-normal singular coordinates

Li and Werner, Adaptive minimax optimality in statistical inverse problems via SOLIT, Lemma 3.6 and §§3.1.1–3.2.1

Clason, Regularization of Inverse Problems, complete lecture notes, Chapter 5 §5.1

The $L^2$-limit definition is needed because white noise is not an $Y$-valued observation in infinite dimension. The estimator itself is $X$-valued because its filtered noise covariance has finite trace.

Assume Countable Choice. Let $T:X\to Y$ be an injective compact Hilbert–Schmidt operator of infinite rank between separable real Hilbert spaces, let $S=T^{\ast }T$, and let $(s_j,v_j,u_j)$ be a fixed positive singular system with ${\lambda }_j=s_j^2$. Let $Z$ be a real isonormal Gaussian process on $Y$, meaning that $g\mapsto Z(g)$ is linear into $L^2$, is centered, has covariance $\mathbb{E}[Z(g)Z(h)]=⟨g,h{⟩}_Y$, and has independent standard-normal values on every finite orthonormal tuple. In the weak observation experiment $Y_x(g)=⟨Tx,g{⟩}_Y+\sigma Z(g)$ with known noise standard deviation $\sigma >0$, use the ordered filter $q_{\alpha }$, the geometric-variance candidates ${\alpha }_0>\cdots >{\alpha }_{m_{\max }}$, and variances $v_m={\sigma }^2{\sum }_j{\lambda }_j{q}_{{\alpha }_m}({\lambda }_j{)}^2$ from the preceding definition. For each candidate index $m$, the regularized estimator ${\widehat{x}}_m$ is the $L^2$ limit of its finite singular-coordinate sums; its $j$th coordinate is ${\lambda }_j{q}_{{\alpha }_m}({\lambda }_j){\theta }_j+\sigma s_j{q}_{{\alpha }_m}({\lambda }_j){\xi }_j$, where ${\theta }_j=⟨x,v_j{⟩}_X$ and ${\xi }_j=Z(u_j)$.

Take as input the observed sequence $y=(y_j{)}_{j\ge 1}$, where $y_j:=Y_x(u_j)$. For each candidate $m$ and each $N=1,2,\dots$, compute

If these partial sums converge in $X$, define ${\widehat{x}}_m(y)$ to be their limit; otherwise set ${\widehat{x}}_m(y)=0_X$. This specifies an $X$-valued output for every input sequence. The quantities $v_{m_1,m_2}$ and $z_{m_1,m_2}(x)$ are determined by the known operator, filter, noise level and Gaussian coordinate law.

For $0\le m_1<m_2\le m_{\max }$, set

and let $N_{m_1,m_2}$ be the norm of the centered estimator difference divided by $\sigma$. For $x>0$, define its upper quantile by

Given fixed $\beta >0$ and $\gamma >0$, for $0\le m<m_{\max }$ put $x_m:=2(1+\gamma )\log (v_{m+1}\mathrm{/}{v}_0)$, and for each pair $m_1<m_2$ put ${\eta }_{m_1,m_2}:=\sigma z_{m_1,m_2}(x_{m_1})+\beta \sqrt{v_{m_1,m_2}}$. Accept an index $m_1$ exactly when $\mathrm{\parallel }{\widehat{x}}_{m_1}(y)-{\widehat{x}}_{m_2}(y){\mathrm{\parallel }}_X\le {\eta }_{m_1,m_2}$ for every $m_2$ with $m_1<m_2\le m_{\max }$, and output the smallest accepted index $\widehat{m}$ and parameter $\widehat{\alpha }={\alpha }_{\widehat{m}}$. For $m_1=m_{\max }$, accept without comparisons, including when $m_{\max }=0$. This rule uses only the observed sequence, the known operator and filter, the noise level, the fixed candidate grid, and $\beta ,\gamma$; it does not use $x$ or its source index $\phi$.

Ordered filters, source conditions, and candidate-grid variance

The positive-noise SVD sequence experiment

The weak isonormal observation process

Independent Gaussian coordinates of white noise

Well-definedness and bias–variance properties of the regularized estimators

Li and Werner, Adaptive minimax optimality in statistical inverse problems via SOLIT, §§3.2.1–3.2.2

Spokoiny and Willrich, Bootstrap tuning in ordered model selection, §§2.1–2.4

The arrows show the order of the finite procedure: form the regularized estimates, compare each candidate with every finer one using the two-index threshold, then return the smallest candidate that passes all of its comparisons. The diagram illustrates the algorithm only; the proofs of its properties use the coordinate definitions and inequalities.

The constant $\gamma$ controls the strength of the noise quantiles, while $\beta$ controls the bias tolerance. The quantile definition remains meaningful when a pairwise Gaussian difference is degenerate; then its upper quantile is zero.

Assume Countable Choice. Let $T:X\to Y$ be an injective compact Hilbert–Schmidt operator of infinite rank between separable real Hilbert spaces. Write $A:=T^{\ast }T$, and list its positive eigenvalues with multiplicity as ${\lambda }_1\ge {\lambda }_2\ge \cdots >0$, where ${\lambda }_1=\mathrm{\parallel }A\mathrm{\parallel }$ and ${\sum }_j{\lambda }_j<\mathrm{\infty }$. Observe the weak Gaussian experiment $Y_x(g)=⟨Tx,g{⟩}_Y+\sigma Z(g),g\in Y,$ where $\sigma >0$ is known and $Z:Y\to L^2(\mathrm{\Omega })$ is centered and linear with covariance $\mathbb{E}[Z(g)Z(h)]=⟨g,h{⟩}_Y$, and has independent standard-normal values on every finite orthonormal tuple.

Let $\phi :[0,\mathrm{\infty })\to [0,\mathrm{\infty })$ be continuous and strictly increasing with $\phi (0)=0$. Assume $u\mapsto u{\phi }^{-1}(\sqrt{u})$ is convex for $0\le u\le \phi ({\lambda }_1{)}^2$, where ${\phi }^{-1}$ is the inverse on the range of $\phi$. For $\rho >0$, define the source ball ${\mathcal{W}}_{\phi }(\rho ):=\{\phi (A)w:w\in X,\mathrm{\parallel }w{\mathrm{\parallel }}_X\le \rho \}\mathrm{.}$ Assume $x\in {\mathcal{W}}_{\phi }(\rho )$. Let $q_{\alpha }:[0,{\lambda }_1]\to [0,\mathrm{\infty })$ be an ordered filter indexed by $\alpha \in \mathcal{A}\subset (0,\mathrm{\infty })$, with $0$ an accumulation point of $\mathcal{A}$, such that for finite $C_q^{\mathrm{\prime }},C_{\phi }>0$ and every $t\in [0,{\lambda }_1]$ and $\alpha \in \mathcal{A}$, $0\le \alpha q_{\alpha }(t)\le C_q^{\mathrm{\prime }},0\le t{q}_{\alpha }(t)\le 1,$ $q_{\alpha }(t)\le q_{\tilde{\alpha }}(t)$ whenever $\tilde{\alpha }<\alpha$, and ${\sup }_{0\le t\le {\lambda }_1}\phi (t)(1-t{q}_{\alpha }(t))\le C_{\phi }\phi (\alpha )\mathrm{.}$

Define $\mathrm{\Sigma }(t):=\mathrm{\#}\{j:{\lambda }_j\ge t\}$. Assume there is a continuous $S:(0,{\lambda }_1]\to [0,\mathrm{\infty })$ with ${\mathrm{lim\hspace{0.17em}inf}}_{\alpha \downarrow 0}S(\alpha )\mathrm{/}\mathrm{\Sigma }(\alpha )>0$, $\alpha S(\alpha )\to 0$, and ${\alpha }^{-1}{\int }_0^{\alpha }\mathrm{\Sigma }(t)dt\le C_{S}S(\alpha )$ for some $C_S>0$ for every candidate $\alpha \in \mathcal{A}\cap (0,{\lambda }_1]$.

Let ${\alpha }_0>{\alpha }_1>\cdots$ be candidates in $\mathcal{A}\cap (0,{\lambda }_1]$, with the initial candidate ${\alpha }_0$ fixed independently of $\sigma$, and set $v_m:={\sigma }^2{\sum }_{j\ge 1}{\lambda }_j{q}_{{\alpha }_m}({\lambda }_j{)}^2,$ and assume $v_0>0$ and $1<{\theta }_1\le v_m\mathrm{/}{v}_{m-1}\le {\theta }_2<\mathrm{\infty }$ for $m\ge 1$. Set $m_{\max }:=\min \{m\in {\mathbb{N}}_0:v_m\ge 1\}$. Fix a positive singular system $(s_j,v_j,u_j)$ with $T{v}_j=s_j{u}_j$ and ${\lambda }_j=s_j^2$, set ${\theta }_j=⟨x,v_j{⟩}_X$ and ${\xi }_j=Z(u_j)$, and define ${\widehat{x}}_m$ as the $L^2(\mathrm{\Omega };X)$ limit with coordinates $⟨{\widehat{x}}_m,v_j{⟩}_X={\lambda }_j{q}_{{\alpha }_m}({\lambda }_j){\theta }_j+\sigma s_j{q}_{{\alpha }_m}({\lambda }_j){\xi }_j\mathrm{.}$ For $m_1<m_2$, define the pairwise variance $v_{m_1,m_2}:={\sigma }^2{\sum }_{j\ge 1}{\lambda }_j(q_{{\alpha }_{m_2}}({\lambda }_j)-q_{{\alpha }_{m_1}}({\lambda }_j){)}^2\mathrm{.}$ Let $G_{m_1,m_2}:=({\widehat{x}}_{m_2}-\mathbb{E}{\widehat{x}}_{m_2})-({\widehat{x}}_{m_1}-\mathbb{E}{\widehat{x}}_{m_1})$ be the centered estimator difference, and for $a>0$ set $z_{m_1,m_2}(a):=\inf \{z\ge 0:\mathbb{P}(\mathrm{\parallel }{G}_{m_1,m_2}{\mathrm{\parallel }}_X\mathrm{/}\sigma >z)\le e^{-a}\}\mathrm{.}$ For fixed $\beta ,\gamma >0$, define ${\eta }_{m_1,m_2}:=\sigma z_{m_1,m_2}(2(1+\gamma )\log (v_{m_1+1}\mathrm{/}{v}_0))+\beta \sqrt{v_{m_1,m_2}}$ for $m_1<m_2$. Let $\widehat{m}$ be the smallest $m_1\in \{0,\dots ,m_{\max }\}$ such that $\mathrm{\parallel }{\widehat{x}}_{m_1}-{\widehat{x}}_{m_2}{\mathrm{\parallel }}_X\le {\eta }_{m_1,m_2}$ for every $m_2$ with $m_1<m_2\le m_{\max }$, and set $\widehat{\alpha }={\alpha }_{\widehat{m}}$. Define $m_{\ast }$ as the smallest $m$ for which every pair $m\le m_1<m_2\le m_{\max }$ satisfies $\mathrm{\parallel }\mathbb{E}{\widehat{x}}_{m_1}-\mathbb{E}{\widehat{x}}_{m_2}{\mathrm{\parallel }}_X^2\le {\beta }^2{v}_{m_1,m_2},$ and put ${\mathcal{R}}_{m_{\ast }}:=\mathbb{E}\mathrm{\parallel }{\widehat{x}}_{m_{\ast }}-x{\mathrm{\parallel }}_X^2$. Write $M:=m_{\max }$, $k:=m_{\ast }$, $b_m:=\mathrm{\parallel }\mathbb{E}{\widehat{x}}_m-x{\mathrm{\parallel }}_X$, and $R_m:=b_m^2+v_m$, so $R_k={\mathcal{R}}_{m_{\ast }}$. Put $D_0:=0$, and for $k\ge 1$ put $D_k:=(1+\beta )\sqrt{v_k}+\sqrt{4(1+\gamma )u_k\log (v_k\mathrm{/}{v}_0)},u_m:={\sigma }^2{\sup }_j{\lambda }_j{q}_{{\alpha }_m}({\lambda }_j{)}^2\mathrm{.}$ If $k<M$, define $p_m:=\min \{1,(M-m+1)(v_0\mathrm{/}{v}_m{)}^{2(1+\gamma )}\}(k<m\le M),P_k:=\min \{1,(M-k)(v_0\mathrm{/}{v}_{k+1}{)}^{2(1+\gamma )}\},$ $A_k:=\sqrt{3}{\sum }_{m=k+1}^M{v}_m\sqrt{p_m},B_k:=2\sqrt{3}{R}_k(v_0\mathrm{/}{v}_k{)}^{1+\gamma }{\sum }_{m=k+1}^M\sqrt{M-m+1}(v_k\mathrm{/}{v}_m{)}^{\gamma }\mathrm{.}$ If $k=M$, set $P_k=A_k=B_k=0$. Then the unchanged pairwise-quantile selector satisfies the finite oracle inequality $\mathbb{E}\mathrm{\parallel }{\widehat{x}}_{\widehat{m}}-x{\mathrm{\parallel }}_X^2\le (\sqrt{R_k}+D_k{)}^2+\min \{(b_k\sqrt{P_k}+\sqrt{A_k}{)}^2,B_k\}\mathrm{.}$ In particular, put $\delta :=\frac{1}{2(1+\gamma )},K:=\frac{{\theta }_1}{{\theta }_1-1}+\frac{1}{1-{\theta }_1^{-\gamma }},W_M:=\{\begin{array}{ll}{v}_0{M}^{\delta },& M\ge 1,\\ 0,& M=0.\end{array}$ Then $A_k\le \sqrt{3}K{W}_M$ and $B_k\le \frac{2\sqrt{3}\sqrt{M}}{{\theta }_1^{\gamma }-1}(v_0\mathrm{/}{v}_k{)}^{1+\gamma }{R}_k,$ and a rate-oriented consequence is $\mathbb{E}\mathrm{\parallel }{\widehat{x}}_{\widehat{m}}-x{\mathrm{\parallel }}_X^2\le (\sqrt{R_k}+D_k{)}^2+2b_k^2+2\sqrt{3}K{W}_M\mathrm{.}$ When $M=0$ the selected index is identically zero and its risk is exactly $R_0$. When $k=0$, the first term in both displayed oracle bounds is $R_0$ because $D_0=0$; when $k=M$, the late-selection term vanishes. The bounds retain the pairwise quantiles in the stated rule.

For the rate conclusions as ${\sigma }^2\downarrow 0$, assume additionally that ${\lambda }_1\le e^{-1}$. In both rate regimes let $\phi (t)=t^{\nu }(-\log t{)}^{-\tau }$ for $t\in (0,{\lambda }_1]$, extend it by $\phi (0)=0$, take the natural logarithm, and assume the displayed $\phi$ satisfies the index-function, convexity, and filter-qualification hypotheses above; here either $\nu >0,\tau \in \mathbb{R}$ or $\nu =0,\tau >0$.

If ${\lambda }_j\asymp j^{-a}$ with $a>1$, $S(\alpha )\asymp {\alpha }^{-1\mathrm{/}a}$, and $v_m\ge c_v{\sigma }^{2}S({\alpha }_m)\mathrm{/}{\alpha }_m$ for some $c_v>0$, then for some constant $C>0$, ${\sup }_{x\in {\mathcal{W}}_{\phi }(\rho )}\mathbb{E}\mathrm{\parallel }{\widehat{x}}_{\widehat{m}}-x{\mathrm{\parallel }}_X^2\le C({\sigma }^2{)}^{\frac{2\nu }{2\nu +1\mathrm{/}a+1}}(-\log {\sigma }^2{)}^{-\frac{2\tau (1+1\mathrm{/}a)}{2\nu +1+1\mathrm{/}a}}\mathrm{.}$ If instead ${\lambda }_j\asymp j^{-a}{e}^{-b{j}^{\vartheta }}$ with $a\in \mathbb{R}$, $b>0$, and $0<\vartheta \le \frac{1}{2}$, assume also $S(\alpha )\asymp (-\log \alpha {)}^{1\mathrm{/}\vartheta }$ and $v_m\ge c_v{\sigma }^{2}S({\alpha }_m)\mathrm{/}((-\log {\alpha }_m){\alpha }_m)$ for some $c_v>0$; then for some $C>0$, ${\sup }_{x\in {\mathcal{W}}_{\phi }(\rho )}\mathbb{E}\mathrm{\parallel }{\widehat{x}}_{\widehat{m}}-x{\mathrm{\parallel }}_X^2\le C({\sigma }^2{)}^{\frac{2\nu }{2\nu +1}}(-\log {\sigma }^2{)}^{\frac{2\nu \mathrm{/}\vartheta -2\tau }{2\nu +1}}\mathrm{.}$ The severe-rate assertion requires exactly $0<\vartheta \le \frac{1}{2}$; the source does not establish adaptive-minimax optimality of the polynomial-smoothness severe rate. For spectral cut-off filters, the uniform geometric-variance candidate condition may fail when $\vartheta >1$. When $\nu =0$, both displayed rates simplify to $(-\log {\sigma }^2{)}^{-2\tau }$.

Gaussian series norm tails and fourth moments

Ordered filter, source index, qualification, and geometric-variance grid

The SOLIT selector and its two-index threshold

Risk decomposition, monotone bias, and pairwise variance bound

The positive-noise sequence model and minimum-norm coordinates

Weak Gaussian white-noise observations

Independent Gaussian coordinates of white noise

Li and Werner, Adaptive minimax optimality in statistical inverse problems via SOLIT, Theorems 3.4, 3.8, 4.1 and 4.3

Spokoiny and Willrich, Bootstrap tuning in ordered model selection, Theorem 2.3 and Appendix A

Clason, Regularization of Inverse Problems, complete lecture notes, Chapter 5 §5.2