Covariance and correlation principal component analysis

Let $\mathsf{E}=((\mathcal{X},\mathcal{A}),\mathrm{\Theta },(P_{\theta }{)}_{\theta \in \mathrm{\Theta }})$ be a parametric statistical model, let $p\ge 1$, and let $X:\mathcal{X}\to {\mathbb{R}}^p$ be a measurable random vector with ${\mathbb{E}}_{\theta }\mathrm{\parallel }X{\mathrm{\parallel }}_2^2<\mathrm{\infty }$ for every $\theta \in \mathrm{\Theta }$. Write ${\mu }_{\theta }={\mathbb{E}}_{\theta }X$ and ${\mathrm{\Sigma }}_{\theta }={\mathrm{Cov}}_{\theta }(X)$ as in Covariance matrix of a finite-dimensional random vector. The map $\psi (\theta )={\mathrm{\Sigma }}_{\theta }$ is the covariance-matrix estimand on this model (Estimands on a statistical model).

For each $\theta$, list the real eigenvalues of ${\mathrm{\Sigma }}_{\theta }$ in nonincreasing order and counting multiplicity. The covariance-PCA decomposition is the list of eigenspaces of ${\mathrm{\Sigma }}_{\theta }$, ordered by their distinct eigenvalues from largest to smallest. Equivalently, for each distinct eigenvalue $\lambda$, its component subspace is $\ker ({\mathrm{\Sigma }}_{\theta }-\lambda I)$. The top eigenspace is the leading principal subspace; when its eigenvalue is simple it is a line, and either unit representative has the same sign-free direction. The well-definedness result A finite second moment defines a positive semidefinite covariance matrix ensures these covariance matrices are finite, symmetric, and positive semidefinite. The real spectral theorem Real spectral theorem: a self-adjoint endomorphism of a finite-dimensional real inner product space has an orthonormal eigenbasis ensures their eigenspaces span ${\mathbb{R}}^p$.

If every coordinate variance $({\mathrm{\Sigma }}_{\theta }{)}_{jj}$ is positive, put $D_{\theta }=\mathrm{diag}(\sqrt{({\mathrm{\Sigma }}_{\theta }{)}_{11}},\dots ,\sqrt{({\mathrm{\Sigma }}_{\theta }{)}_{pp}})$ and standardize the centered vector as $Z_{\theta }=D_{\theta }^{-1}(X-{\mu }_{\theta })$. Its covariance is the correlation matrix $R_{\theta }=D_{\theta }^{-1}{\mathrm{\Sigma }}_{\theta }{D}_{\theta }^{-1}$. Correlation-PCA is the ordered eigenspace decomposition of $R_{\theta }$. Thus both population conventions center first; correlation-PCA additionally divides each coordinate by its positive population standard deviation.

For an iid sample $X_1,\dots ,X_n$, sample covariance-PCA uses the eigenspaces of the sample-centered covariance ${\widehat{\mathrm{\Sigma }}}_n^{\ast }=n^{-1}{\sum }_i(X_i-\bar{X})(X_i-\bar{X}{)}^{\mathsf{T}}$ from Sample covariance estimators and squared operator loss, where $\bar{X}=n^{-1}{\sum }_i{X}_i$. Sample correlation-PCA standardizes by the positive sample standard deviations when all sample coordinate variances are positive. The divisor convention here is $n$, not $n-1$. The sample covariance matrix is a statistic, and it is an estimator of the covariance target in its sample experiment, as proved in Sample-centered covariance is a measurable statistic and estimator.

This definition specifies mathematical matrices under an assumed law. The scientific model contract distinguishes that assumption from evidence that an empirical law is adequate; PCA itself makes no such adequacy claim (Scientific statistical model contracts).