PCA and eigengap stability — Examples

The eigenvalues of $A_{\delta }$ are $1+\delta$ and $1$, so its leading line is $\mathrm{span}(e_1)$ and its gap is $g_{\delta }=\delta$. Also $E_{\delta }^{\mathsf{T}}{E}_{\delta }=(\delta \mathrm{/}4{)}^2{I}_2$, whence [F1] gives $\mathrm{\parallel }{E}_{\delta }{\mathrm{\parallel }}_{\mathrm{o}\mathrm{p}}=\delta \mathrm{/}4$.

The characteristic polynomial of $A_{\delta }+E_{\delta }=\left(\begin{array}{cc}1+\delta & \delta \mathrm{/}4\\ \delta \mathrm{/}4& 1\end{array}\right)$ has roots $1+\delta \mathrm{/}2\pm \sqrt{(\delta \mathrm{/}2{)}^2+(\delta \mathrm{/}4{)}^2}=1+\delta \mathrm{/}2\pm \sqrt{5}\delta \mathrm{/}4$, so the larger root is the leading eigenvalue. If $(x,y{)}^{\mathsf{T}}$ is its leading eigenvector, the first row of the eigenvalue equation gives $y\mathrm{/}x=4({\lambda }_1(A_{\delta }+E_{\delta })-1-\delta )\mathrm{/}\delta =\sqrt{5}-2$. Thus its line makes angle $\arctan (\sqrt{5}-2)>0$ with $\mathrm{span}(e_1)$, independently of $\delta$.

Letting $\delta \downarrow 0$ makes $\mathrm{\parallel }{E}_{\delta }{\mathrm{\parallel }}_{\mathrm{o}\mathrm{p}}\to 0$ by step 1.1 while the angle in step 1.2 stays fixed and positive, contradicting the proposed uniform claim. Here ${ϵ}_{\delta }=g_{\delta }\mathrm{/}4<g_{\delta }\mathrm{/}2$, so [F2] still applies; its gap-dependent bound is consistent with the example.

On this four-point space, $\mathbb{E}U=\mathbb{E}V=0$, $\mathbb{E}{U}^2=\mathbb{E}{V}^2=1$, and $\mathbb{E}[UV]=0$ because two points have $UV=1$ and two have $UV=-1$. Thus $\mathbb{E}X=0$, $\mathrm{Var}(3U)=9$, $\mathrm{Var}(U+\sqrt{3}V)=1+3=4$, and $\mathrm{Cov}(3U,U+\sqrt{3}V)=3$. This gives the stated covariance matrix.

The positive standard-deviation matrix is $D=\mathrm{diag}(3,2)$, so [F2] gives $D^{-1}\mathrm{Cov}(X)D^{-1}=\left(\begin{array}{cc}1& 1\mathrm{/}2\\ 1\mathrm{/}2& 1\end{array}\right)$. This matrix sends $(1,1{)}^{\mathsf{T}}$ to $\frac{3}{2}(1,1{)}^{\mathsf{T}}$ and $(1,-1{)}^{\mathsf{T}}$ to $\frac{1}{2}(1,-1{)}^{\mathsf{T}}$; hence its leading line is $\mathrm{span}(1,1)$.

The covariance matrix has characteristic roots $(13\pm \sqrt{61})\mathrm{/}2$, so its leading eigenspace is a line. It sends $(1,1{)}^{\mathsf{T}}$ to $(12,7{)}^{\mathsf{T}}$, which is not a scalar multiple of $(1,1{)}^{\mathsf{T}}$. Thus the correlation-PCA leading line is not the covariance-PCA leading line, and [F3] identifies each as the maximizing line for its own projected variance.

The eigenvalues of ${\mathrm{\Sigma }}_1=I_2$ are $1,1$, and $I_2-I_2$ is the zero matrix, so the defined leading eigenspace is $E_1=\ker (I_2-I_2)={\mathbb{R}}^2$. For any finite-second-moment random vector $X$ with covariance ${\mathrm{\Sigma }}_1$ and every unit vector $a\in {\mathbb{R}}^2$, [F4] gives $\mathrm{Var}(a^{\mathsf{T}}X)=a^{\mathsf{T}}{I}_{2}a=\mathrm{\parallel }a{\mathrm{\parallel }}_2^2=1$. Thus every unit vector in $E_1$ attains the same projected variance $1$ and maximizes it; each of its lines is a variance-maximizing direction, while the defined leading PCA subspace remains all of ${\mathbb{R}}^2$.

The eigenvalues of ${\mathrm{\Sigma }}_2=\mathrm{diag}(2,1)$ are $2>1$, with leading eigenspace $\mathrm{span}(e_1)$. Both $e_1$ and $-e_1$ have unit norm and satisfy ${\mathrm{\Sigma }}_2{e}_1=2e_1$ and ${\mathrm{\Sigma }}_2(-e_1)=2(-e_1)$. They determine the same line, so even a simple leading eigenvalue does not orient its eigenvector.