Tangent spaces and Frobenius projections on Stiefel and Grassmann

The derivative of $F$ at $X$ is $D{F}_X[Z]=X^{\mathsf{T}}Z+Z^{\mathsf{T}}X$. It is onto the symmetric matrices because for any symmetric $H$, the choice $Z=XH\mathrm{/}2$ gives $D{F}_X[Z]=H$. Thus $0$ is a regular value; [F2] identifies the tangent space with the displayed kernel.

Write $S=\mathrm{sym}(X^{\mathsf{T}}G)$. Then $X^{\mathsf{T}}(G-XS)+(G-XS{)}^{\mathsf{T}}X=0$, so $G-XS$ is tangent. For every symmetric $H$ and tangent $Z$, $⟨XH,Z{⟩}_F=\mathrm{tr}(H{X}^{\mathsf{T}}Z)=0$ because $X^{\mathsf{T}}Z$ is skew-symmetric. Hence $XS$ is normal and $G-XS$ is the orthogonal projection.

A curve $Q(t)$ in $\mathrm{O}(r)$ through $I_r$ has $\mathrm{\Omega }=Q^{\mathrm{\prime }}(0)$ skew-symmetric by differentiating $Q(t{)}^{\mathsf{T}}Q(t)=I_r$. Conversely, for any skew-symmetric $\mathrm{\Omega }$, the Cayley curve $Q(t)=(I_r-\frac{t}{2}\mathrm{\Omega }{)}^{-1}(I_r+\frac{t}{2}\mathrm{\Omega })$ is defined near zero, satisfies $Q(t{)}^{\mathsf{T}}Q(t)=I_r$, and has $Q(t)=I_r+t\mathrm{\Omega }+O(t^2)$; hence the vertical tangent directions are exactly $X\mathrm{\Omega }$. Every Stiefel tangent $Z$ decomposes as $Z=X\mathrm{\Omega }+(I-X{X}^{\mathsf{T}})Z$, where $\mathrm{\Omega }=X^{\mathsf{T}}Z$ is skew and the second term is horizontal. These summands are orthogonal, so the quotient tangent is represented uniquely by the horizontal summand.

The matrix $(I-X{X}^{\mathsf{T}})G$ is horizontal. For every horizontal $Z$, $⟨G-(I-X{X}^{\mathsf{T}})G,Z{⟩}_F=⟨X{X}^{\mathsf{T}}G,Z{⟩}_F=0$, since $X^{\mathsf{T}}Z=0$. Therefore it is the quotient-metric projection. Under $X\mapsto XQ$ and $G\mapsto GQ$ it becomes $(I-X{X}^{\mathsf{T}})GQ$, the same tangent vector in the changed representative.