Minimax estimation over inverse source classes

Put $g(x)=e^{-x^2}$. By [F10] its mixed improper Riemann integral is $\sqrt{\pi }$, and [F15] identifies its two one-sided improper integrals, so each equals $\sqrt{\pi }\mathrm{/}2$. The half-line bridge [F11] gives ${\int }_{[0,\mathrm{\infty })}gd{\lambda }_1=\sqrt{\pi }\mathrm{/}2$. The map $T(x)=-x$ is a $C^1$ diffeomorphism from $(0,\mathrm{\infty })$ to $(-\mathrm{\infty },0)$, so [F12] applied to $g$ gives ${\int }_{(-\mathrm{\infty },0)}gd{\lambda }_1={\int }_{(0,\mathrm{\infty })}gd{\lambda }_1$. By [F13] the point $0$ contributes zero, and [F14] on the disjoint decompositions $(0,\mathrm{\infty })\bigsqcup \{0\}=[0,\mathrm{\infty })$ and $(-\mathrm{\infty },0)\bigsqcup [0,\mathrm{\infty })=\mathbb{R}$ therefore gives ${\int }_{\mathbb{R}}{e}^{-x^2}d{\lambda }_1=\sqrt{\pi }$. Applying [F12] with $T(x)=\sqrt{2}x$ and $f(y)=e^{-y^2\mathrm{/}2}$ now gives ${\int }_{\mathbb{R}}{e}^{-x^2\mathrm{/}2}d{\lambda }_1=\sqrt{2}\sqrt{\pi }$, hence ${\int }_{\mathbb{R}}\varphi (x)d{\lambda }_1=1$.

Write $b=\min \{r,\tau \}$ and $u=b\mathrm{/}\tau \in [0,1]$. This location family is the scalar Gaussian coordinate experiment described in [F1]. Under the two-point prior, the likelihood ratio at observation $y$ is $p_{b,\tau }(y)\mathrm{/}{p}_{-b,\tau }(y)=\exp (2by\mathrm{/}{\tau }^2)$ by [F8]. Hence the posterior mean is $b\tanh (by\mathrm{/}{\tau }^2)$ and the posterior variance is $b^2{\mathrm{sech}}^2(by\mathrm{/}{\tau }^2)$ by [F5]–[F6]; for any action $a$, its conditional squared loss is the square of its distance from this posterior mean plus that posterior variance. Averaging this identity over any action kernel from [F4] leaves the posterior variance as a lower bound, including the degenerate case $b=0$.

The density in the Statement is positive by [F8], continuous by [F9], and measurable by [F3]. Applying [F12] to $T(x)=\theta +\sqrt{2}\tau x$ gives its Lebesgue integral as $(\tau \sqrt{2\pi }{)}^{-1}\sqrt{2}\tau {\int }_{\mathbb{R}}{e}^{-x^2}d{\lambda }_1=1$ by step 1.1, so by [F2]–[F3] it defines the scalar sampling probability on the stated Borel sample space.

Define $I(u):={\int }_{\mathbb{R}}\varphi (x)\mathrm{sech}(ux)d{\lambda }_1(x)$. Since $\cosh t\ge 1$, the integrand lies between $0$ and $\varphi (x)$, whose integral is $1$ by step 1.1; monotonicity [F17] makes $I(u)$ finite. Since $\cosh$ is even and increasing on $[0,\mathrm{\infty })$ by [F6], $\mathrm{sech}(ux)\ge \mathrm{sech}(x)$ for every real $x$ and $0\le u\le 1$. Therefore $I(u)\ge I(1)$ by [F17], while $e^{-u^2\mathrm{/}2}\ge e^{-1\mathrm{/}2}$ by [F9].

For $x\ge 0$, [F6] gives $0\le \tanh x$ and $(\tanh x{)}^{\mathrm{\prime }}\le 1$ because $\cosh x\ge 1$; with $\tanh 0=0$, [F16] yields $\tanh x\le x$. Set $g(x)=e^{-x^2\mathrm{/}2}\cosh x$. Differentiation using [F6] and [F16] gives $g^{\mathrm{\prime }}(x)=g(x)(\tanh x-x)\le 0$, so [F16] implies $g(x)\le g(0)=1$. Evenness extends $\cosh x\le e^{x^2\mathrm{/}2}$ to all real $x$. Consequently $\varphi (x)\mathrm{sech}x\ge (2\pi {)}^{-1\mathrm{/}2}{e}^{-x^2}$, and step 1.1 plus [F17] gives $I(1)\ge 1\mathrm{/}\sqrt{2}$.

The prior predictive density, after $x=y\mathrm{/}\tau$, is ${\tau }^{-1}\varphi (x)e^{-u^2\mathrm{/}2}\cosh (ux)$, where $\varphi (x)=(2\pi {)}^{-1\mathrm{/}2}{e}^{-x^2\mathrm{/}2}$, by [F8]. Multiplying by the posterior variance from step 1.2 and applying the full-line Lebesgue affine substitution [F12] to this nonnegative measurable integrand gives Bayes risk $B(r,\tau )=b^2{e}^{-u^2\mathrm{/}2}I(u)$, with $I$ defined in step 2.2.

Steps 3.1, 2.2 and 2.3 give $B(r,\tau )\ge b^2{e}^{-1\mathrm{/}2}\mathrm{/}\sqrt{2}=b^2\mathrm{/}\sqrt{2e}=T(r,\tau )\mathrm{/}\sqrt{2e}$, where $e=\exp (1)$ by [F7] and the exponential laws [F8]–[F9]. Since the prior is supported in $[-r,r]$, the supremum risk of every rule is at least its prior-averaged risk; taking the infimum over rules and using [F4] yields $M(r,\tau )\ge B(r,\tau )$.

The constant estimator $0$ has maximum risk $r^2$. For the identity estimator, [F1] gives $Y-\theta =\tau \xi$ with $\mathbb{E}{\xi }^2=1$, so its risk is ${\tau }^2$ at every parameter; hence their better maximum risk is $T(r,\tau )=b^2$. This proves the projection-risk claim together with step 4.1.

Write $w_j=⟨w,v_j{⟩}_X$. By [F4] and the compact spectral multiplier [F5], the coefficient of $\mathrm{\mid }K{\mathrm{\mid }}^{\nu }w$ at $v_j$ is $s_j^{\nu }{w}_j$; the kernel component of $w$ contributes zero.

Conversely, take any $\theta \in {\mathrm{\ell }}^2$ with ${\sum }_j{s}_j^{-2\nu }{\theta }_j^2\le {\rho }^2$ and put $w_j=s_j^{-\nu }{\theta }_j$. This sequence is square-summable with norm at most $\rho$; the coefficient isometry in [F1] gives a unique $w\in N(K{)}^{\perp }$ with these coefficients.

For every $x=\mathrm{\mid }K{\mathrm{\mid }}^{\nu }w$ with $\mathrm{\parallel }w{\mathrm{\parallel }}_X\le \rho$, step 1.1 and Bessel's inequality for the orthonormal family $(v_j)$ give ${\sum }_{j\ge 1}{s}_j^{-2\nu }⟨x,v_j{⟩}_X^2={\sum }_{j\ge 1}{w}_j^2\le \mathrm{\parallel }w{\mathrm{\parallel }}_X^2\le {\rho }^2$. Thus every source element has coefficient sequence in the stated ellipsoid.

Set $x=\mathrm{\mid }K{\mathrm{\mid }}^{\nu }w$ for the vector constructed in step 1.2. By [F5] its $j$th coefficient is $s_j^{\nu }{w}_j={\theta }_j$; [F1] makes equality of the coefficient sequences equality of vectors in $N(K{)}^{\perp }$. Since $\mathrm{\parallel }w{\mathrm{\parallel }}_X\le \rho$, [F2] gives $x\in X_{\nu ,\rho }(K)$. This proves the reverse inclusion.

If $s_{j+1}=s_j$, their inverse powers are equal. Otherwise $0<s_{j+1}<s_j$; by [F7], $f(t)=t^{-\nu }$ is continuous on $[s_{j+1},s_j]$ and differentiable on its interior, with $f^{\mathrm{\prime }}(t)=-\nu t^{-\nu -1}<0$. Thus [F8] gives $f(s_j)-f(s_{j+1})=f^{\mathrm{\prime }}(c)(s_j-s_{j+1})<0$ for some $c\in (s_{j+1},s_j)$. In both cases $s_{j+1}^{-\nu }\ge s_j^{-\nu }$. Hence $a_j=s_j^{-\nu }$ is positive and nondecreasing; by [F3] the right-hand set is a weighted statistical ellipsoid. Steps 2.1 and 2.2 prove the two set inclusions, completing the identification.

If $Q=0$, then ${\mathrm{\Theta }}_a(Q)=\{0\}$. The zero estimator has minimax risk zero, and the $D=0$ cutoff expression is also zero, so the asserted inequalities hold. Hence assume $Q>0$.

Put ${\tau }_j^2={ϵ}^2{s}_j^{-2}$ and let $\mathcal{C}$ be the set of sequences $r_j\ge 0$ with ${\sum }_j{a}_j^2{r}_j\le Q^2$. For $r\in \mathcal{C}$, define $J(r)={\sum }_j\min (r_j,{\tau }_j^2)$. Since $a_j\ge a_1>0$, one has ${\sum }_j{r}_j\le Q^2\mathrm{/}{a}_1^2$, so $J(r)$ is finite.

As $s_j\to 0$ and $ϵ>0$, ${\tau }_j^2\to \mathrm{\infty }$. Choose $N\ge 1$ so that ${\tau }_j^2>Q^2\mathrm{/}{a}_1^2$ for $j>N$. Every feasible $r$ then has $r_j<{\tau }_j^2$ on that tail. Replacing its tail by ${\tilde{r}}_{N+1}=({\sum }_{j>N}{a}_j^2{r}_j)\mathrm{/}{a}_{N+1}^2$ at coordinate $N+1$ and zero thereafter preserves the weighted budget and does not decrease $J$, because $a_j\ge a_{N+1}$ for $j>N$ and ${\sum }_{j>N}{r}_j\le {\tilde{r}}_{N+1}<{\tau }_{N+1}^2$. The finite-dimensional feasible set in coordinates $1,\dots ,N+1$ is nonempty (it contains zero), closed, and bounded (each coordinate is at most $Q^2\mathrm{/}{a}_j^2$). The finite sum $J$ is continuous there, so [F9] gives a finite-support maximizer $r^{\ast }$.

For $k\in \mathbb{N}$, put $c_k:=(k+3)!$. By induction, $c_k\ge 6\cdot 3^k$, with strict inequality for $k\ge 1$: the base case is $c_0=3!=6$, and if $c_k\ge 6\cdot 3^k$, then $c_{k+1}=(k+4)c_k\ge 4c_k\ge 24\cdot 3^k>18\cdot 3^k=6\cdot 3^{k+1}$, using $(k+4)\ge 4$ and $c_k>0$. Thus for every $N\ge 4$, ${\sum }_{n=0}^N\frac{1}{n!}\le \frac{5}{2}+\frac{1}{6}{\sum }_{k=0}^{N-3}{3}^{-k}-\frac{1}{72},$ because the $n=4$ term is strictly below its comparison term by $1\mathrm{/}18-1\mathrm{/}24=1\mathrm{/}72$ and all other terms from $n\ge 3$ are bounded by the induction. For $k=0$ one has $(1\mathrm{/}3{)}^0=3^0=1$. For $k\ge 1$, the product and reciprocal laws in [F7] give $3^k(1\mathrm{/}3{)}^k=(3(1\mathrm{/}3){)}^k=1$, hence $(1\mathrm{/}3{)}^k=(3^k{)}^{-1}=3^{-k}$. Applying the geometric-series formula in [F7] at ratio $1\mathrm{/}3$ therefore gives ${\sum }_{k\ge 0}{3}^{-k}=1\mathrm{/}(1-1\mathrm{/}3)=3\mathrm{/}2$. By [F7], the left partial sums tend to $e$ and these geometric sums tend to $3\mathrm{/}2$; [F7] therefore gives $e\le \frac{11}{4}-\frac{1}{72}<\frac{11}{4}$. Hence $2\sqrt{2e}<\sqrt{22}<\frac{121}{25}=(\frac{11}{5}{)}^2$.

The compact-SVD coordinate realization uses Countable Choice [A1]. For each $D$, the first $D$ coordinate evaluations are measurable in the product Borel data space [F1], and their finite-support embedding into ${\mathrm{\ell }}^2$ is continuous; thus the cutoff rule [F3] is measurable, including the constant rule at $D=0$. Its squared loss is measurable by [F10].

Among the finite-support maximizers choose one, $r^{\ast }$, with the longest initial saturated segment, and let $D$ be its length; this choice exists because step 1.3 bounds the support by $N+1$. If $P=\{j:r_j^{\ast }\ge {\tau }_j^2\}$ had a later saturated index $j>D+1$, then $i=D+1$ is the first gap before such an index and $r_i^{\ast }<{\tau }_i^2$. Set $\delta ={\tau }_i^2-r_i^{\ast }>0$. Since $s_j$ is nonincreasing, ${\tau }_i^2\le {\tau }_j^2$, so $\delta \le {\tau }_i^2\le {\tau }_j^2\le r_j^{\ast }$. Replacing $r_i^{\ast }$ by $r_i^{\ast }+\delta$ and $r_j^{\ast }$ by $r_j^{\ast }-\delta$ does not increase the budget, since $a_i\le a_j$, and does not decrease $J$, since the gain at $i$ is $\delta$ and the loss at $j$ is at most $\delta$. It therefore gives a maximizer whose initial saturated segment has length at least $D+1$, a contradiction. Thus $P=\{1,\dots ,D\}$ for some $D\ge 0$.

Let the prior choose independently, for each $j\le N+1$, the two values $\pm b_j$ with equal probability, where $b_j=\min \{\sqrt{r_j^{\ast }},{\tau }_j\}$, and set all later coordinates to zero. Every point in its finite support belongs to ${\mathrm{\Theta }}_a(Q)$ because ${\sum }_{j\le N+1}{a}_j^2{b}_j^2\le {\sum }_j{a}_j^2{r}_j^{\ast }\le Q^2$. For any such truth $\eta$, relative to the zero-truth product law $P_0$, the likelihood ratio is $L_{\eta }(z)={\prod }_{j=1}^{N+1}\exp ({\eta }_j{z}_j\mathrm{/}{\tau }_j^2-{\eta }_j^2\mathrm{/}(2{\tau }_j^2))$. For a cylinder determined by a finite coordinate set $F$, split $F$ into active coordinates $F\cap \{1,\dots ,N+1\}$ and inactive coordinates. On each active coordinate, multiplying the zero-mean Gaussian density by its factor in $L_{\eta }$ gives the shifted Gaussian density; inactive coordinates retain their zero-truth density, and active likelihood factors outside $F$ integrate to one. Finite-coordinate independence therefore gives equality of the two probability measures on every such cylinder, and the product density integrates to one. These cylinders form a generating pi-system, so [F8] gives equality on the full product Borel sigma-algebra and proves that $L_{\eta }$ is the likelihood ratio for the entire experiment. The finite product prior and this factorized likelihood give a coordinatewise posterior.

The admissible cutoff rule from step 1.5 has minimax risk no greater than its worst-case risk by [F6]. For each $D$ this risk is the exact expression in [F4]; taking the infimum over $D$ gives $R_{\ast }(ϵ)\le R_{\mathrm{c}\mathrm{u}\mathrm{t}}(ϵ)$.

Fix $r\in \mathcal{C}$ and $m\ge N+1$, and truncate $r$ after coordinate $m$; the truncation remains feasible. The segment from $r^{\ast }$ to this finite-support truncation is feasible, and its finite sum $J$ is maximized at its $r^{\ast }$ endpoint. Its right derivative there is therefore nonpositive. Since $P=\{1,\dots ,D\}$ by step 2.1, coordinates $j>D$ are strictly below their caps at $r^{\ast }$ and contribute their direction increments; coordinates $j\le D$ with $r_j^{\ast }>{\tau }_j^2$ contribute zero, while those with $r_j^{\ast }={\tau }_j^2$ contribute $\min (r_j-{\tau }_j^2,0)$. Consequently ${\sum }_{D<j\le m}(r_j-r_j^{\ast })+{\sum }_{\{j\le D:r_j^{\ast }={\tau }_j^2\}}\min (r_j-{\tau }_j^2,0)\le 0$. Letting $m\to \mathrm{\infty }$ is legitimate because $\sum r_j$ and $\sum r_j^{\ast }$ converge. Rearranging gives ${\sum }_{j>D}{r}_j\le {\sum }_{j>D}{r}_j^{\ast }+{\sum }_{\{j\le D:r_j^{\ast }={\tau }_j^2\}}({\tau }_j^2-r_j{)}_{+}\le {\sum }_{j>D}{r}_j^{\ast }+{\sum }_{j=1}^D{\tau }_j^2$.

For fixed observed data, the randomized rule draws its action from the parameter-independent kernel in [F6]; thus additionally conditioning on that action leaves the posterior from step 2.2 unchanged. For each active coordinate, conditional on the data and action, squared error averaged over this posterior is its posterior variance plus a nonnegative squared displacement from the posterior mean. Averaging over the action kernel and then over the data preserves the posterior-variance lower bound. Thus every rule's prior-averaged risk is at least the sum of the scalar Bayes risks. Each scalar prior is supported at $\pm \min \{\sqrt{r_j^{\ast }},{\tau }_j\}$, so [F5] bounds that sum below by ${\sum }_{j\le N+1}\min (r_j^{\ast },{\tau }_j^2)\mathrm{/}\sqrt{2e}=J_{\ast }\mathrm{/}\sqrt{2e}$. The supremum risk dominates the supported prior average; taking the minimax infimum gives $R_{\ast }(ϵ{)}^2\ge J_{\ast }\mathrm{/}\sqrt{2e}$.

For any $\theta \in {\mathrm{\Theta }}_a(Q)$, the sequence $r_j={\theta }_j^2$ belongs to $\mathcal{C}$. Step 3.1 bounds its omitted squared norm. The cutoff risk is that tail plus the retained variance by [F4], so at the cutoff $D$ from step 2.1 its worst-case risk is at most ${\sum }_{j>D}{r}_j^{\ast }+2{\sum }_{j\le D}{\tau }_j^2\le 2J_{\ast }$. Therefore $R_{\mathrm{c}\mathrm{u}\mathrm{t}}(ϵ{)}^2\le 2J_{\ast }$.

Combining steps 4.1 and 3.2 yields $R_{\mathrm{c}\mathrm{u}\mathrm{t}}(ϵ{)}^2\le 2\sqrt{2e}{R}_{\ast }(ϵ{)}^2$.

Since both risks are nonnegative, steps 5.1 and 1.4 imply $R_{\mathrm{c}\mathrm{u}\mathrm{t}}(ϵ)\le \frac{11}{5}{R}_{\ast }(ϵ)$.

For $q>0$ and integer $D\ge 2$, ${\sum }_{j=1}^D{j}^q\le D^{q+1}$. For a lower bound, the at least $D\mathrm{/}2$ indices with $D\mathrm{/}2\le j\le D$ each satisfy $j^q\ge (D\mathrm{/}2{)}^q$, so ${\sum }_{j=1}^D{j}^q\ge (D\mathrm{/}2{)}^{q+1}$. Thus ${\sum }_{j\le D}{j}^q\asymp D^{q+1}$. Also the finite geometric-sum identity gives ${\sum }_{j\le D}{e}^{2pj}\asymp e^{2pD}$ for each fixed $p>0$.

Suppose $s_j\asymp j^{-p}$ and $a_j\asymp j^{\kappa }$. With $H={ϵ}^{-2\mathrm{/}(2\kappa +2p+1)}$ and $D_0=\lfloor H\rfloor$, step 1.1 and [F2] show that both terms of $B_{D_0}$ are $O(H^{-2\kappa })$. For any $D\le D_0$, monotonicity of $a_j$ gives $Q^2\mathrm{/}{a}_{D+1}^2\ge Q^2\mathrm{/}{a}_{D_0+1}^2\asymp H^{-2\kappa }$; for $D>D_0$, the variance is at least ${ϵ}^2{\sum }_{j\le D_0}{j}^{2p}\asymp {ϵ}^2{H}^{2p+1}=H^{-2\kappa }$. Therefore ${\inf }_D{B}_D\asymp H^{-2\kappa }$ and the cutoff RMS error is $\asymp {ϵ}^{\kappa \mathrm{/}(\kappa +p+1\mathrm{/}2)}$.

Suppose $s_j\asymp e^{-pj}$ and $a_j\asymp j^{\kappa }$. Put $L=\log (1\mathrm{/}ϵ)$ and $D_0=\lfloor (L-\kappa \log L)\mathrm{/}p\rfloor$. By [F7], $\log L\mathrm{/}L\to 0$, so $D_0\asymp L$. At $D_0$, the squared bias and variance in [F2] are both $O(L^{-2\kappa })$. For $D\le D_0$, the bias is at least $Q^2\mathrm{/}{a}_{D_0+1}^2\asymp L^{-2\kappa }$; for $D>D_0$, the geometric variance is at least ${ϵ}^2{e}^{2p{D}_0}\asymp e^{-2L}{e}^{2L}{L}^{-2\kappa }=L^{-2\kappa }$. Hence ${\inf }_D{B}_D\asymp L^{-2\kappa }$ and the cutoff RMS error is $\asymp L^{-\kappa }$.