PCA and eigengap stability

For every unit vector $x$, [F3] and [F4] give $\mathrm{\mid }{x}^{\mathsf{T}}Ex\mathrm{\mid }\le \mathrm{\parallel }x{\mathrm{\parallel }}_2\mathrm{\parallel }Ex{\mathrm{\parallel }}_2\le ϵ$. The top-eigenvalue Rayleigh characterization in [F2] therefore gives ${\lambda }_1(A+E)\le {\lambda }_1(A)+ϵ$, while evaluating the perturbed Rayleigh quotient at $v$ gives ${\lambda }_1(A+E)\ge v^{\mathsf{T}}(A+E)v\ge {\lambda }_1(A)-ϵ$. This proves the eigenvalue bound.

The minimum--maximum formula in [F2], with the $(p-1)$-dimensional subspace $v^{\perp }$, gives ${\lambda }_2(A+E)\le {\max }_{\mathrm{\parallel }x{\mathrm{\parallel }}_2=1,x\perp v}{x}^{\mathsf{T}}(A+E)x\le {\lambda }_2(A)+ϵ$, since every such $x$ has $x^{\mathsf{T}}Ax\le {\lambda }_2(A)$. Step 1.1 gives ${\lambda }_1(A+E)\ge {\lambda }_1(A)-ϵ$, hence ${\lambda }_1(A+E)-{\lambda }_2(A+E)\ge g-2ϵ>0$; the perturbed leading eigenvalue is simple. Also ${\lambda }_1(A+E)-{\lambda }_2(A)\ge g-ϵ>0$.

Let $P=I-v{v}^{\mathsf{T}}$, the orthogonal projection onto $v^{\perp }$, and write ${\tilde{\lambda }}_1={\lambda }_1(A+E)$. Projecting $(A+E)\tilde{v}={\tilde{\lambda }}_1\tilde{v}$ onto $v^{\perp }$ gives $({\tilde{\lambda }}_{1}I-A{\mathrm{\mid }}_{v^{\perp }})P\tilde{v}=PE\tilde{v}$. In an orthonormal eigenbasis of $A{\mathrm{\mid }}_{v^{\perp }}$, every eigenvalue of ${\tilde{\lambda }}_{1}I-A{\mathrm{\mid }}_{v^{\perp }}$ is at least ${\tilde{\lambda }}_1-{\lambda }_2(A)\ge g-ϵ$ by step 2.1. Therefore $(g-ϵ)\mathrm{\parallel }P\tilde{v}{\mathrm{\parallel }}_2\le \mathrm{\parallel }PE\tilde{v}{\mathrm{\parallel }}_2\le ϵ$. Indeed, $z=Pz+(I-P)z$ is an orthogonal decomposition, so $\mathrm{\parallel }Pz{\mathrm{\parallel }}_2\le \mathrm{\parallel }z{\mathrm{\parallel }}_2$; then [F3] and $\mathrm{\parallel }\tilde{v}{\mathrm{\parallel }}_2=1$ give the final inequality. Since $\mathrm{\parallel }P\tilde{v}{\mathrm{\parallel }}_2^2=1-(v^{\mathsf{T}}\tilde{v}{)}^2$, division by $g-ϵ>0$ proves the angle bound.

Since $t^2\le 1+t^4$ for $t\ge 0$, the fourth-moment assumption gives $\mathbb{E}\mathrm{\parallel }X{\mathrm{\parallel }}_2^2<\mathrm{\infty }$. Then $\mathrm{\parallel }X{\mathrm{\parallel }}_2\le 1+\mathrm{\parallel }X{\mathrm{\parallel }}_2^2$ gives $\mathbb{E}\mathrm{\parallel }X{\mathrm{\parallel }}_2<\mathrm{\infty }$, so the coordinatewise mean $\mu$ is finite. Put $Y_{n,i}=X_{n,i}-\mu$ and $M_n=n^{-1}{\sum }_i{Y}_{n,i}{Y}_{n,i}^{\mathsf{T}}$. By [F8], $\mathrm{\parallel }{Y}_{n,1}{\mathrm{\parallel }}_2\le \mathrm{\parallel }{X}_{n,1}{\mathrm{\parallel }}_2+\mathrm{\parallel }\mu {\mathrm{\parallel }}_2$. For $a,b\ge 0$, $2ab\le a^2+b^2$ gives $(a+b{)}^2\le 2(a^2+b^2)$ and then $(a+b{)}^4\le 4(a^2+b^2{)}^2\le 8(a^4+b^4)$. Hence $\mathbb{E}\mathrm{\parallel }{Y}_{n,1}{\mathrm{\parallel }}_2^4\le 8(\mathbb{E}\mathrm{\parallel }X{\mathrm{\parallel }}_2^4+\mathrm{\parallel }\mu {\mathrm{\parallel }}_2^4)<\mathrm{\infty }$. For each $j,k$, the variables $W_{n,i}^{jk}:=Y_{n,i,j}{Y}_{n,i,k}-{\mathrm{\Sigma }}_{jk}$ are centered and square-integrable because $(Y_{n,i,j}{Y}_{n,i,k}{)}^2\le \mathrm{\parallel }{Y}_{n,i}{\mathrm{\parallel }}_2^4$. The map $y\mapsto y_j{y}_k-{\mathrm{\Sigma }}_{jk}$ is Borel, so [F1] and [F2] make $(W_{n,i}^{jk}{)}_{i=1}^n$ independent.

Let ${\bar{Y}}_n=n^{-1}{\sum }_i{Y}_{n,i}$. By [F1] and [F2], the coordinate variables $Y_{n,i,j}$ are independent across $i$. Thus [F4] makes their cross-covariances zero, and [F4] and [F5] give $\mathbb{E}\mathrm{\parallel }{\bar{Y}}_n{\mathrm{\parallel }}_2^2=\mathrm{tr}(\mathrm{\Sigma })\mathrm{/}n$. Also $\mathrm{\parallel }{\bar{Y}}_n{\bar{Y}}_n^{\mathsf{T}}{\mathrm{\parallel }}_F=\mathrm{\parallel }{\bar{Y}}_n{\mathrm{\parallel }}_2^2$ by expanding both squares. For every $C>0$, [F6] therefore gives $\mathbb{P}(\sqrt{n}\mathrm{\parallel }{\bar{Y}}_n{\bar{Y}}_n^{\mathsf{T}}{\mathrm{\parallel }}_F>C)\le \mathrm{tr}(\mathrm{\Sigma })\mathrm{/}(C\sqrt{n})\to 0$. Thus this centered-mean term converges to zero in probability.

For fixed $j,k$, all $W_{n,i}^{jk}$ are the same Borel function of the iid vectors $Y_{n,i}$, so they are iid: their independence follows from [F1] and [F2], and equal laws follow because a measurable map sends equal input laws to equal output laws. Thus [F4] gives $\mathbb{E}[(M_{n,jk}-{\mathrm{\Sigma }}_{jk}{)}^2]=n^{-2}{\sum }_i\mathrm{Var}(W_{n,i}^{jk})=n^{-1}\mathrm{Var}(W_{n,1}^{jk})$. Since $\mathbb{E}[Y_{n,1,j}{Y}_{n,1,k}]={\mathrm{\Sigma }}_{jk}$, $\mathrm{Var}(W_{n,1}^{jk})=\mathbb{E}[Y_{n,1,j}^2{Y}_{n,1,k}^2]-{\mathrm{\Sigma }}_{jk}^2\le \mathbb{E}[Y_{n,1,j}^2{Y}_{n,1,k}^2]$. Summing over $j,k$ and using ${\sum }_{j,k}{Y}_{n,1,j}^2{Y}_{n,1,k}^2=\mathrm{\parallel }{Y}_{n,1}{\mathrm{\parallel }}_2^4$ yields $\mathbb{E}\mathrm{\parallel }{M}_n-\mathrm{\Sigma }{\mathrm{\parallel }}_F^2\le n^{-1}\mathbb{E}\mathrm{\parallel }{Y}_{n,1}{\mathrm{\parallel }}_2^4$. Consequently [F6] gives, for every $C>0$, $\mathbb{P}(\sqrt{n}\mathrm{\parallel }{M}_n-\mathrm{\Sigma }{\mathrm{\parallel }}_F>C)\le \mathbb{E}\mathrm{\parallel }{Y}_{n,1}{\mathrm{\parallel }}_2^4\mathrm{/}{C}^2$, so $\sqrt{n}\mathrm{\parallel }{M}_n-\mathrm{\Sigma }{\mathrm{\parallel }}_F=O_P(1)$ by [F7].

Expanding the centered sum in [F3] gives ${\widehat{\mathrm{\Sigma }}}_n^{\ast }=M_n-{\bar{Y}}_n{\bar{Y}}_n^{\mathsf{T}}$. For a matrix $B$, let $\mathrm{vec}(B)\in {\mathbb{R}}^{p^2}$ list its entries; by the definition of the Frobenius norm, $\mathrm{\parallel }\mathrm{vec}(B){\mathrm{\parallel }}_2=\mathrm{\parallel }B{\mathrm{\parallel }}_F$. Applying [F8] to these vectors gives $\mathrm{\parallel }B+C{\mathrm{\parallel }}_F\le \mathrm{\parallel }B{\mathrm{\parallel }}_F+\mathrm{\parallel }C{\mathrm{\parallel }}_F$, including the triangle inequality used below. For any $\eta >0$, choose $C_1$ so the limsup tail of $\sqrt{n}\mathrm{\parallel }{M}_n-\mathrm{\Sigma }{\mathrm{\parallel }}_F$ above $C_1$ is at most $\eta \mathrm{/}2$ by step 2.2. For all sufficiently large $n$, the probability that $\sqrt{n}\mathrm{\parallel }{\bar{Y}}_n{\bar{Y}}_n^{\mathsf{T}}{\mathrm{\parallel }}_F>1$ is at most $\eta \mathrm{/}2$ by step 2.1. The displayed Frobenius triangle inequality then bounds the limsup probability that $\sqrt{n}\mathrm{\parallel }{\widehat{\mathrm{\Sigma }}}_n^{\ast }-\mathrm{\Sigma }{\mathrm{\parallel }}_F>C_1+1$ by $\eta$. This is $\sqrt{n}\mathrm{\parallel }{\widehat{\mathrm{\Sigma }}}_n^{\ast }-\mathrm{\Sigma }{\mathrm{\parallel }}_F=O_P(1)$ by [F7], which is the claimed root-n rate.

Regard ${\mathbb{S}}^p$ as the subspace of ${\mathbb{R}}^{p^2}$ with its Borel sigma-algebra. Each matrix coordinate of $T_n$ is a polynomial in the $np$ real sample coordinates, so $T_n:({\mathbb{R}}^p{)}^n\to {\mathbb{R}}^{p^2}$ is continuous. Its values are symmetric because every outer product is symmetric, hence it is also continuous as a map into the subspace ${\mathbb{S}}^p$. By [F7] it is Borel measurable.

By [F1] and [F2], the sample experiment is the stated parameter-indexed iid experiment; [F3] places ${\mathrm{\Sigma }}_{\theta }$ in ${\mathbb{S}}^p$, and [F4] identifies it as the estimand target. The formula in [F5] is exactly $T_n$. Thus [F6] and step 1.1 make $T_n$ a statistic and, since its target space is the same ${\mathbb{S}}^p$, an estimator of $\psi$.

Put $Y=X-{\mu }_{\theta }$. Finite second moments make every coordinate in the finite sums below integrable. Expanding the centered scalar square and using [F4] gives ${\mathrm{Var}}_{\theta }(a^{\mathsf{T}}X)={\mathbb{E}}_{\theta }[(a^{\mathsf{T}}Y{)}^2]={\sum }_{j,k}{a}_j{a}_k{\mathbb{E}}_{\theta }[Y_j{Y}_k]=a^{\mathsf{T}}{\mathrm{\Sigma }}_{\theta }a$ by [F2] and [F3].

By [F5], write $a={\sum }_{j=1}^p{c}_j{e}_j$ in an orthonormal eigenbasis of ${\mathrm{\Sigma }}_{\theta }$ with eigenvalues ${\lambda }_1\ge \cdots \ge {\lambda }_p$ and ${\sum }_j{c}_j^2=1$. Then $a^{\mathsf{T}}{\mathrm{\Sigma }}_{\theta }a={\sum }_j{\lambda }_j{c}_j^2\le {\lambda }_1{\sum }_j{c}_j^2={\lambda }_1$. The difference is ${\lambda }_1-a^{\mathsf{T}}{\mathrm{\Sigma }}_{\theta }a={\sum }_j({\lambda }_1-{\lambda }_j)c_j^2$, a sum of nonnegative terms, so equality holds exactly when $c_j=0$ for every ${\lambda }_j<{\lambda }_1$. This is exactly the unit sphere in the leading eigenspace. Step 1.1 identifies this Rayleigh quotient with projected variance, proving the covariance-PCA assertion.

If all coordinate variances are positive, [F1] defines $Z_{\theta }=D_{\theta }^{-1}(X-{\mu }_{\theta })$ with covariance $R_{\theta }=D_{\theta }^{-1}{\mathrm{\Sigma }}_{\theta }{D}_{\theta }^{-1}$. The hypotheses of steps 1.1--2.1 hold with $Z_{\theta }$ and $R_{\theta }$ in place of $X$ and ${\mathrm{\Sigma }}_{\theta }$, so the leading correlation-PCA eigenspace maximizes the variance of unit linear scores of $Z_{\theta }$.

In the orthonormal eigenbasis supplied by [F2], the equality ${\lambda }_1={\lambda }_2$ gives at least two linearly independent eigenvectors with eigenvalue ${\lambda }_1$. The full eigenspace $E_1=\ker (\mathrm{\Sigma }-{\lambda }_{1}I)$ contains these vectors, is specified by $\mathrm{\Sigma }$ itself, and therefore has dimension at least two.

Every unit vector in $E_1$ is a leading principal direction by [F3]. Because a space of dimension at least two contains noncollinear unit vectors, there is no single unit vector selected by $\mathrm{\Sigma }$ alone. The canonical object is the whole leading eigenspace $E_1$.

For every $p\times p$ matrix $B$ and unit vector $u$, applying Cauchy--Schwarz to each row gives $\mathrm{\parallel }Bu{\mathrm{\parallel }}_2^2={\sum }_j({\sum }_k{B}_{jk}{u}_k{)}^2\le {\sum }_{j,k}{B}_{jk}^2=\mathrm{\parallel }B{\mathrm{\parallel }}_F^2$. Hence [F4] gives $\mathrm{\parallel }B{\mathrm{\parallel }}_{\mathrm{o}\mathrm{p}}\le \mathrm{\parallel }B{\mathrm{\parallel }}_F$. With ${ϵ}_n=\mathrm{\parallel }{\widehat{\mathrm{\Sigma }}}_n^{\ast }-\mathrm{\Sigma }{\mathrm{\parallel }}_{\mathrm{o}\mathrm{p}}$, [F1] implies ${ϵ}_n=O_P(n^{-1\mathrm{/}2})$ by [F6]. To see $\mathbb{P}(G_n)\to 1$, fix $\eta >0$ and choose $C$ so the limsup tail of $\sqrt{n}\mathrm{\parallel }{\widehat{\mathrm{\Sigma }}}_n^{\ast }-\mathrm{\Sigma }{\mathrm{\parallel }}_F$ above $C$ is at most $\eta$. For all sufficiently large $n$, $C\mathrm{/}\sqrt{n}<g\mathrm{/}2$, and then $G_n^c\subseteq \{\sqrt{n}\mathrm{\parallel }{\widehat{\mathrm{\Sigma }}}_n^{\ast }-\mathrm{\Sigma }{\mathrm{\parallel }}_F>C\}$. Thus ${\mathrm{lim\hspace{0.17em}sup}}_n\mathbb{P}(G_n^c)\le \eta$, so the claimed probability tends to one.

On $G_n$, [F2] says that the sample leading eigenvalue is simple and $Z_n\le {ϵ}_n\mathrm{/}(g-{ϵ}_n)\le 2{ϵ}_n\mathrm{/}g$. The sample matrix is measurable by [F3]; its operator-norm distance from $\mathrm{\Sigma }$ is therefore measurable. The leading line varies continuously on the set of matrices with a simple top eigenvalue: applying [F2] with any such matrix as the base and a perturbation tending to zero makes the sign-free line angle tend to zero. Hence $Z_n$, extended by zero off the Borel event $G_n$, is measurable. Pointwise, $\sqrt{n}{Z}_n\le (2\mathrm{/}g)\sqrt{n}{ϵ}_n$, so [F1] and [F6] imply $\sqrt{n}{Z}_n=O_P(1)$ and therefore $Z_n=O_P(n^{-1\mathrm{/}2})$.

Fix $\delta >0$ and $\eta >0$. Since [F6] defines $\sqrt{n}{Z}_n=O_P(1)$, choose $C$ so its limsup tail above $C$ is at most $\eta$. For all sufficiently large $n$, $C\mathrm{/}\sqrt{n}<\delta$, and then $\{Z_n>\delta \}\subseteq \{\sqrt{n}{Z}_n>C\}$. Hence ${\mathrm{lim\hspace{0.17em}sup}}_n\mathbb{P}(Z_n>\delta )\le \eta$. Since $\eta$ was arbitrary, $Z_n\to 0$ in probability.