Isotropic minimum-norm risk limits under a positive spectral gap

Put $P_{\ker X_n}=I_{p_n}-X_n^{+}{X}_n$. For any fixed $v\in {\mathbb{R}}^{p_n}$, ${\mathbb{E}}_{z_{0,n}}(z_{0,n}^{\mathsf{T}}v{)}^2=v^{\mathsf{T}}\mathbb{E}[z_{0,n}{z}_{0,n}^{\mathsf{T}}]v=\mathrm{\parallel }v{\mathrm{\parallel }}_2^2$, so the stated prediction risk equals ${\mathbb{E}}_{{\epsilon }_n}[\mathrm{\parallel }{\widehat{\beta }}_n-{\beta }_n{\mathrm{\parallel }}_2^2\mid X_n]$. By [F2], $P_n:=X_n^{+}{X}_n$ is an orthogonal projection. The Penrose equation $X_n{X}_n^{+}{X}_n=X_n$ shows $\ker (P_n)=\ker (X_n)$: if $X_{n}v=0$, then $P_{n}v=0$, and if $P_{n}v=0$, then $X_{n}v=X_n{X}_n^{+}{X}_{n}v=X_n{P}_{n}v=0$. Hence $\mathrm{Ran}(P_n)=\ker (X_n{)}^{\perp }=\mathrm{Ran}(X_n^{\mathsf{T}})$ by the right-singular-vector basis in [F3], so ${\widehat{\beta }}_n-{\beta }_n=-P_{\ker X_n}{\beta }_n+X_n^{+}{\epsilon }_n$. Conditional on $X_n$, the cross term has expectation zero because $\mathbb{E}[{\epsilon }_n\mid X_n]=0$. Its noise contribution is ${\mathbb{E}}_{{\epsilon }_n}\mathrm{\parallel }{X}_n^{+}{\epsilon }_n{\mathrm{\parallel }}_2^2={\sigma }^2\mathrm{tr}(X_n^{+}(X_n^{+}{)}^{\mathsf{T}})={\sigma }^2\mathrm{tr}((X_n^{\mathsf{T}}{X}_n{)}^{+})=\frac{{\sigma }^2}{n}\mathrm{tr}(S_n^{+}),$ where the middle equality follows by writing an SVD and the last uses $(aB{)}^{+}=a^{-1}{B}^{+}$ for $a>0$. With $x_n={\beta }_n\mathrm{/}r$, this proves the exact identity ${\mathcal{R}}_n=r^2{x}_n^{\mathsf{T}}{P}_{\ker X_n}{x}_n+\frac{{\sigma }^2}{n}\mathrm{tr}(S_n^{+})\mathrm{.}$

First identify the isotropic limit. By [F7], ${\nu }_{\gamma }=(1-\gamma ){\delta }_0+\gamma {\mu }_{\gamma }$ is a nonnegative probability measure: when $\gamma >1$, the negative coefficient cancels exactly the zero atom of $\gamma {\mu }_{\gamma }$. Thus $u=-(1-\gamma )\mathrm{/}z+\gamma m_{\gamma }(z)$ is its Stieltjes transform and has positive imaginary part, as does $m_{\gamma }(z)$. The scalar equation in [F7] rearranges to $m_{\gamma }=(1-\gamma -z-\gamma z{m}_{\gamma }{)}^{-1}$; [F9] therefore identifies $F_{\gamma ,{\delta }_1}={\mu }_{\gamma }$. For any deterministic unit directions $x_n$ and fixed $z\in {\mathbb{C}}^{+}$, we claim $\mathbb{E}[x_n^{\mathsf{T}}(S_n-zI{)}^{-1}{x}_n]\to m_{\gamma }(z)$. Indeed, every subsequence has a further subsequence $(n_j)$ with strictly increasing $p_{n_j}$, since $p_n\to \mathrm{\infty }$. In the transposed infinite array, prescribe $N_d=n_j$ and direction $x_{n_j}$ at $d=p_{n_j}$, and at all other dimensions prescribe $N_d=\max (1,\lfloor d\mathrm{/}\gamma \rfloor )$ and the first coordinate vector. Then $d\mathrm{/}{N}_d\to \gamma$, so [F6] applies to this completed sequence. The bound $\mathrm{\mid }(t-z{)}^{-1}\mathrm{\mid }\le 1\mathrm{/}\mathrm{Im}z$ and [F10] give the claimed expectation limit along $(n_j)$. The subsequence criterion gives it along the original sequence. This argument also applies to each fixed standardized bounded truncation of the entry law.

Suppose $\gamma <1$ and set $a=(1-\sqrt{\gamma }{)}^2>0$. By [F4], $X_n$ has full column rank eventually and every eigenvalue of $S_n$ is at least $a\mathrm{/}2$ eventually. Thus the bias term in step 1.1 is zero and $p_n^{-1}\mathrm{tr}(S_n^{+})=\int f(t)d{\mu }_n(t)$ for the bounded continuous function $f(t)=1\mathrm{/}\max (t,a\mathrm{/}2)$. By [F5] and [F8], this converges to $J_{\gamma }=\int t^{-1}d{\mu }_{\gamma }(t)$, since the Marchenko–Pastur support is bounded below by $a$. For $s>0$, define $m_{\gamma }(-s)=\int (t+s{)}^{-1}d{\mu }_{\gamma }(t)$. The integrands are bounded by $1\mathrm{/}a$ and converge pointwise to $1\mathrm{/}t$, so [F10] gives $m_{\gamma }(-s)\to J_{\gamma }$ as $s\downarrow 0$. The fixed-point equation in [F7], initially stated for $z\in {\mathbb{C}}^{+}$, extends to $z=-s<0$ by dominated convergence in its defining integral; its denominator stays nonzero because $1+\gamma m_{\gamma }(-s)>0$. Taking $s\downarrow 0$ yields $1+(\gamma -1)J_{\gamma }=0$, hence $J_{\gamma }=1\mathrm{/}(1-\gamma )$. Since $p_n\mathrm{/}n\to \gamma$, step 1.1 gives ${\mathcal{R}}_n\to {\sigma }^2\gamma \mathrm{/}(1-\gamma )$.

For such a bounded unit-variance truncation, let $Z_n$ be its $n\times p_n$ block and $S_n^{(K)}=Z_n^{\mathsf{T}}{Z}_n\mathrm{/}n$. Apply the finite-column argument [F12] to $Z_n^{\mathsf{T}}$, which has $n$ independent columns and identity population. More explicitly, each leave-one-column resolvent quadratic-form increment $U_j$ has $\mathbb{E}\mathrm{\mid }{U}_j{\mathrm{\mid }}^4\le C_K{n}^{-4}$ by the bounded-entry eighth-moment linear-form estimate; its column-filtration difference $D_j$ satisfies $\mathbb{E}\mathrm{\mid }{D}_j{\mathrm{\mid }}^4\le 16\mathbb{E}\mathrm{\mid }{U}_j{\mathrm{\mid }}^4$. The fourth-moment recursion in [F12], $a_j\le a_{j-1}+C_K\sqrt{a_{j-1}}\mathrm{/}{n}^2+C_K\mathrm{/}{n}^4$, gives $a_j\le C_K^{\mathrm{\prime }}{j}^2\mathrm{/}{n}^4$, and hence $\mathbb{E}\mathrm{\mid }{x}_n^{\mathsf{T}}((S_n^{(K)}-zI{)}^{-1}-\mathbb{E}(S_n^{(K)}-zI{)}^{-1})x_n{\mathrm{\mid }}^4\le C_K^{\mathrm{\prime }}\mathrm{/}{n}^2$. These are finite-matrix estimates, with no indexing or distinctness assumption on $p_n$. By [F14], their tails are summable in the original index $n$. Together with step 1.2 this proves $x_n^{\mathsf{T}}(S_n^{(K)}-zI{)}^{-1}{x}_n\to m_{\gamma }(z)$ almost surely for each fixed $K,z$.

To remove boundedness, put ${\xi }^{(K)}=\xi {\mathbf{1}}_{\{\mathrm{\mid }\xi \mathrm{\mid }\le K\}}-\mathbb{E}[\xi {\mathbf{1}}_{\{\mathrm{\mid }\xi \mathrm{\mid }\le K\}}]$, ${\sigma }_K^2=\mathbb{E}\mathrm{\mid }{\xi }^{(K)}{\mathrm{\mid }}^2\to 1$, and ${\tau }_K^2=\mathbb{E}\mathrm{\mid }\xi -{\xi }^{(K)}{\mathrm{\mid }}^2\to 0$. For sufficiently large integers $K$, form $Z_n$ from ${\xi }^{(K)}\mathrm{/}{\sigma }_K$ and write $X_n={\sigma }_K{Z}_n+D_n$. The residual entries are centered with finite fourth moment and variance ${\tau }_K^2$. Applying [F13] to the original, standardized and variance-normalized residual arrays yields, with $L=1+\sqrt{\gamma }$, almost surely $\mathrm{lim\hspace{0.17em}sup}\mathrm{\parallel }{X}_n\mathrm{\parallel }\mathrm{/}\sqrt{n}\le L$, $\mathrm{lim\hspace{0.17em}sup}\mathrm{\parallel }{Z}_n\mathrm{\parallel }\mathrm{/}\sqrt{n}\le L$, and $\mathrm{lim\hspace{0.17em}sup}\mathrm{\parallel }{D}_n\mathrm{\parallel }\mathrm{/}\sqrt{n}\le L{\tau }_K$. If ${\tau }_K=0$, countability of the array makes every residual entry zero almost surely. The Gram-matrix identity then gives $\mathrm{lim\hspace{0.17em}sup}\mathrm{\parallel }{S}_n-S_n^{(K)}\mathrm{\parallel }\le L^2((1+{\sigma }_K){\tau }_K+\mathrm{\mid }{\sigma }_K^2-1\mathrm{\mid })$. For $\eta =\mathrm{Im}z>0$, the resolvent identity bounds the difference of their directional transforms by ${\eta }^{-2}\mathrm{\parallel }{S}_n-S_n^{(K)}\mathrm{\parallel }$. Intersect the probability-one events over integers $K$ and let $K\to \mathrm{\infty }$ in step 2.2 to obtain $x_n^{\mathsf{T}}(S_n-zI{)}^{-1}{x}_n\to m_{\gamma }(z)$ almost surely.

Suppose $\gamma >1$ and set $a=(\sqrt{\gamma }-1{)}^2>0$. By [F4], $X_n$ has full row rank eventually, $S_n$ has exactly $p_n-n$ zero eigenvalues, and all positive eigenvalues are at least $a\mathrm{/}2$ eventually. For each rational $s>0$, the spectral identity bounds the difference between the directional transforms at $-s$ and $-s+i\eta$ by $\eta \mathrm{/}{s}^2$, both for $S_n$ and for ${\mu }_{\gamma }$. Use step 3.1 at $-s+i\mathrm{/}k$ for all positive integers $k$, then let $k\to \mathrm{\infty }$, to obtain $x_n^{\mathsf{T}}(S_n+sI{)}^{-1}{x}_n\to m_{\gamma }(-s)$ on one probability-one event for all rational $s>0$. The gap gives $0\le s{x}_n^{\mathsf{T}}(S_n+sI{)}^{-1}{x}_n-x_n^{\mathsf{T}}{P}_{\ker X_n}{x}_n\le 2s\mathrm{/}a$ eventually. Since $s{m}_{\gamma }(-s)\to {\mu }_{\gamma }(\{0\})=1-1\mathrm{/}\gamma$ by [F7] and [F10], letting rational $s\downarrow 0$ proves the required bias limit. Define $h(0)=0$ and, for $t>0$, $h(t)=t^{-1}\min (1,\max (0,4t\mathrm{/}a-1))$. This is bounded and continuous, equals zero at zero, and equals $1\mathrm{/}t$ for $t\ge a\mathrm{/}2$. Therefore $p_n^{-1}\mathrm{tr}(S_n^{+})=\int h(t)d{\mu }_n(t)\to J_{\gamma }^{+}:={\int }_{(0,\mathrm{\infty })}{t}^{-1}d{\mu }_{\gamma }(t)$ by [F5] and [F8]. [F7] gives $m_{\gamma }(-s)=(1-1\mathrm{/}\gamma )\mathrm{/}s+J_{\gamma }^{+}+o(1)$ as $s\downarrow 0$: subtract the zero atom and use [F10] by dominated convergence on the positive support, whose lower endpoint is $a>0$. To compute the constant term, write $\alpha =1-1\mathrm{/}\gamma$ and substitute $m=\alpha \mathrm{/}s+J_{\gamma }^{+}+o(1)$ into $1+(\gamma -1-s)m-\gamma s{m}^2=0$, the fixed-point equation at $z=-s$ (again obtained by dominated convergence from [F7]). The coefficient of $s^{-1}$ vanishes because $\gamma \alpha =\gamma -1$; the constant term is $1-\alpha -(\gamma -1)J_{\gamma }^{+}+o(1)$. It must vanish, so $J_{\gamma }^{+}=1\mathrm{/}[\gamma (\gamma -1)]$. Using $p_n\mathrm{/}n\to \gamma$ in step 1.1 proves ${\mathcal{R}}_n\to r^2(1-1\mathrm{/}\gamma )+{\sigma }^2\mathrm{/}(\gamma -1)$.