Decision-Theoretic Upper Bounds — Examples

Both coefficient vectors are distinct, but $X_n{\theta }_1={\mathbf{1}}_n=X_n{\theta }_2$. Thus the asserted injectivity fails, and the observation distributions under ${\theta }_1$ and ${\theta }_2$ are identical because the additive noise law is common.

Let $T(Y)\in {\mathbb{R}}^2$ be any measurable coefficient estimator. For every $a\in {\mathbb{R}}^2$, completing squares gives $\mathrm{\parallel }a-{\theta }_1{\mathrm{\parallel }}_2^2+\mathrm{\parallel }a-{\theta }_2{\mathrm{\parallel }}_2^2=2{\parallel a-(\frac{1}{2},\frac{1}{2})\parallel }_2^2+1\ge 1.$ If either coefficient risk is infinite, their maximum is already at least $1\mathrm{/}2$. Otherwise both squared losses are integrable under the common observation law. Their sum is integrable by [F3], so linearity and monotonicity in [F3] show that the sum of the two risks is at least $1$. Thus their maximum is at least $1\mathrm{/}2$ for every $n$.

The common fitted mean $\mu ={\mathbf{1}}_n$ lies in $\mathrm{col}(X_n)$, which has rank $1$. By [F2], its projected least-squares fit has expected squared fitted-value risk ${\sigma }^2$, or ${\sigma }^2\mathrm{/}n$ per observation by [F4]. This vanishes as $n$ grows while the coefficient-risk lower bound in step 1.2 stays at least $1\mathrm{/}2$.

The three candidate supports are $\mathrm{\varnothing },\{1\},\{2\}$. Their fitted vectors are $0$, $({\epsilon }_1,0)$, and $(0,{\epsilon }_2)$, with residual squares ${\epsilon }_1^2+{\epsilon }_2^2$, ${\epsilon }_2^2$, and ${\epsilon }_1^2$, respectively. Hence the chosen fitted vector has squared norm $\max ({\epsilon }_1^2,{\epsilon }_2^2)$, including a tie.

Pointwise, ${\epsilon }_1^2\le \max ({\epsilon }_1^2,{\epsilon }_2^2)\le {\epsilon }_1^2+{\epsilon }_2^2$. The coordinate squares are integrable by the covariance assumption [F4], so their sum is integrable by [F6]. The nonnegative maximum is dominated by this sum, hence is integrable by nonnegative-integral monotonicity in [F6]. Taking expectations with [F6] and using [F4] gives ${\sigma }^2\le \mathbb{E}\mathrm{\parallel }{\widehat{\mu }}_s{\mathrm{\parallel }}_2^2\le 2{\sigma }^2$. Division by $n=2$ proves the stated interval for fitted-value error.

Here $s=1$, $d=2$, and $n=2$, and the noise satisfies [F3], so [F2] gives expected error at most $C{\sigma }^2\log (2e)\mathrm{/}2$. This is consistent with the direct calculation; the example makes no separate minimax lower-bound claim. By [F5], this expectation is the squared fitted-value risk.

Since $\mathrm{col}(X)=\mathbb{R}$, [F1] gives $\widehat{\mu }=Y=\epsilon$. Therefore the training residual $\mathrm{\mid }Y-\widehat{\mu }{\mathrm{\mid }}^2$ is zero for every outcome.

By [F2], the expected squared error for estimating the true mean is $\mathbb{E}\mathrm{\mid }\widehat{\mu }-\mu {\mathrm{\mid }}^2={\sigma }^2$. This is the scalar MSE and squared-loss risk by [F5].

The test-response error is $Y^{\mathrm{\prime }}-\widehat{\mu }={\epsilon }^{\mathrm{\prime }}-\epsilon$. Expanding its square and using [F3] and [F4], $\mathbb{E}\mathrm{\mid }{\epsilon }^{\mathrm{\prime }}-\epsilon {\mathrm{\mid }}^2=\mathbb{E}[({\epsilon }^{\mathrm{\prime }}{)}^2]+\mathbb{E}[{\epsilon }^2]-2\mathbb{E}[{\epsilon }^{\mathrm{\prime }}]\mathbb{E}[\epsilon ]=2{\sigma }^2$.

The training residual, mean-estimation error, and fresh-response test error are respectively $0$, ${\sigma }^2$, and $2{\sigma }^2$ in this model, so these quantities are distinct.