The orthogonal group $O(3)$ and the rotation group $SO(3)$

Let $M_3(\mathbb{R})$ be the set of $3\times 3$ matrices over $\mathbb{R}$ (The vector space $M_{m\times n}(F):=F^{m\times n}$ of $m$ by $n$ matrices over a field, with entrywise operations), with identity $I_3$ and product as in Rectangular matrix multiplication and the identity matrix $I_n$, including zero-sized shapes, transpose $R^{\mathsf{T}}$ as in The transpose $A^{\mathsf{T}}$ of a matrix, and determinant $\det R$ as in For $n\ge 1$, the determinant over a commutative ring by the Leibniz formula, and $\mathrm{\mid }\det A\mathrm{\mid }$ for a real matrix.

The orthogonal group and the special orthogonal group are the subsets

$O(3):=\{R\in M_3(\mathbb{R}):R^{\mathsf{T}}R=I_3\},SO(3):=\{R\in M_3(\mathbb{R}):R^{\mathsf{T}}R=I_3\text{ and }\det R=1\}\mathrm{.}$

Equivalently $SO(3)=O(3)\cap \{R:\det R=1\}$. Elements of $SO(3)$ are called rotations. The identification of $M_3(\mathbb{R})$ with the function space ${\mathbb{R}}^{3\times 3}$ and hence with ${\mathbb{R}}^9$ is the one of The vector space $M_{m\times n}(F):=F^{m\times n}$ of $m$ by $n$ matrices over a field, with entrywise operations; when $M_3(\mathbb{R})$ is regarded as a metric or topological space, this is always the Euclidean metric of that identification.

• The set $O(3)$ is a group under matrix multiplication. It contains $I_3$, and if $A,B\in O(3)$ then $(AB{)}^{\mathsf{T}}(AB)=B^{\mathsf{T}}{A}^{\mathsf{T}}AB=I_3$. For $A\in O(3)$, determinant multiplicativity and $\det (A^{\mathsf{T}})=\det A$ give $(\det A{)}^2=\det (A^{\mathsf{T}}A)=1$, so $A$ is invertible; $A^{\mathsf{T}}A=I_3$ then gives $A^{-1}=A^{\mathsf{T}}$ and $A{A}^{\mathsf{T}}=I_3$, hence $A^{\mathsf{T}}\in O(3)$. The stronger fact that $SO(3)$ is a subgroup of $G{L}_3(\mathbb{R})$ is proved in $SO(3)$ is a subgroup of the general linear group, with $R^{-1}=R^{\mathsf{T}}$.

• An element $R\in O(3)$ preserves the Euclidean inner product of ${\mathbb{R}}^3$: $⟨Ru,Rv{⟩}_2=u^{\mathsf{T}}{R}^{\mathsf{T}}Rv=⟨u,v{⟩}_2$ for all $u,v$. Its three columns therefore form an orthonormal triple. Relative to the standard orientation determined by the standard basis, $\det R=1$ means orientation preserving and $\det R=-1$ means orientation reversing (Orientation of a finite-dimensional real vector space).

• The set $O(3)$ appears here as a subset of $M_3(\mathbb{R})$ with a specific defining equation; no claim is made in this item about its manifold structure, dimension or compactness. Those are supplied by $SO(3)$ is a compact three-dimensional embedded submanifold of $M_3(\mathbb{R})$, with tangent space the matrices $R\mathrm{\Omega }$ for skew $\mathrm{\Omega }$.