The leading PCA eigenspace maximizes projected variance

Put $Y=X-{\mu }_{\theta }$. Finite second moments make every coordinate in the finite sums below integrable. Expanding the centered scalar square and using [F4] gives ${\mathrm{Var}}_{\theta }(a^{\mathsf{T}}X)={\mathbb{E}}_{\theta }[(a^{\mathsf{T}}Y{)}^2]={\sum }_{j,k}{a}_j{a}_k{\mathbb{E}}_{\theta }[Y_j{Y}_k]=a^{\mathsf{T}}{\mathrm{\Sigma }}_{\theta }a$ by [F2] and [F3].

By [F5], write $a={\sum }_{j=1}^p{c}_j{e}_j$ in an orthonormal eigenbasis of ${\mathrm{\Sigma }}_{\theta }$ with eigenvalues ${\lambda }_1\ge \cdots \ge {\lambda }_p$ and ${\sum }_j{c}_j^2=1$. Then $a^{\mathsf{T}}{\mathrm{\Sigma }}_{\theta }a={\sum }_j{\lambda }_j{c}_j^2\le {\lambda }_1{\sum }_j{c}_j^2={\lambda }_1$. The difference is ${\lambda }_1-a^{\mathsf{T}}{\mathrm{\Sigma }}_{\theta }a={\sum }_j({\lambda }_1-{\lambda }_j)c_j^2$, a sum of nonnegative terms, so equality holds exactly when $c_j=0$ for every ${\lambda }_j<{\lambda }_1$. This is exactly the unit sphere in the leading eigenspace. Step 1.1 identifies this Rayleigh quotient with projected variance, proving the covariance-PCA assertion.

If all coordinate variances are positive, [F1] defines $Z_{\theta }=D_{\theta }^{-1}(X-{\mu }_{\theta })$ with covariance $R_{\theta }=D_{\theta }^{-1}{\mathrm{\Sigma }}_{\theta }{D}_{\theta }^{-1}$. The hypotheses of steps 1.1--2.1 hold with $Z_{\theta }$ and $R_{\theta }$ in place of $X$ and ${\mathrm{\Sigma }}_{\theta }$, so the leading correlation-PCA eigenspace maximizes the variance of unit linear scores of $Z_{\theta }$.