Reflections: involutivity, form invariance, fixed hyperplane, and exact rank-two order

The functions $e_s$ are linearly independent: if ${\sum }_{s\in S}{\lambda }_s{e}_s=0_V$ then evaluating at $s$ gives ${\lambda }_s=0$. They span $V$: every $u\in V$ satisfies $u={\sum }_{s\in S}u(s)e_s$, since the right side is a finite sum ($S$ is finite) whose value at any $t\in S$ is $u(t)$. So $(e_s{)}_{s\in S}$ is a basis of $V$. Consequently, for all $u,w\in V$ one has $B(u,w)={\sum }_{s,t\in S}u(s)w(t)B(e_s,e_t)$: expanding $u$ and $w$ in the basis and distributing with bilinearity in each variable gives that finite sum. In particular a bilinear form on $V$ is determined by the numbers $B(e_s,e_t)$.

The displayed $r_a$ for $B(a,a)\mathrm{\ne }0$ is linear in $v$, because $v\mapsto B(v,a)$ is linear and $r_a(v)=v-\frac{2B(v,a)}{B(a,a)}a$ is a sum of the identity and the composite of that functional with scalar multiplication by $a$; also $r_a(a)=a-\frac{2B(a,a)}{B(a,a)}a=a-2a=-a$, and $r_a(v)=v$ for every $v\in \ker B(-,a)$. Moreover $B(r_{a}v,a)=B(v,a)-\frac{2B(v,a)}{B(a,a)}B(a,a)=-B(v,a)$, so $r_a(r_a(v))=r_a(v)-\frac{2B(r_{a}v,a)}{B(a,a)}a=r_a(v)+\frac{2B(v,a)}{B(a,a)}a=v$; hence $r_a^2={\mathrm{i}\mathrm{d}}_V$. Finally, expanding with bilinearity and cancelling the equal cross terms via symmetry $B(u,a)=B(a,u)$ and $B(w,a)=B(a,w)$, $B(r_{a}u,r_{a}w)=B(u,w)-\frac{2B(u,a)B(a,w)}{B(a,a)}-\frac{2B(w,a)B(a,u)}{B(a,a)}+\frac{4B(u,a)B(w,a)}{B(a,a)}=B(u,w),$ so $r_a$ is $B$-preserving.

Put $\phi (v):=B(v,a)$. By $B(a,a)\mathrm{\ne }0$ the value $\phi (a)$ is nonzero, so the image of $\phi$ is all of $\mathbb{R}$ and its rank is $1$; the kernel $\ker B(-,a)=\ker \phi$ is a linear subspace and rank-nullity gives $\dim \ker B(-,a)=\dim V-1$.

For distinct $s,t\in S$, evaluating the definition of $r$ at $e_s$ and $e_t$ with $B(e_s,e_s)=B(e_t,e_t)=1$ and $B(e_s,e_t)=-c$ gives $r_s(e_s)=-e_s$, $r_s(e_t)=e_t+2c{e}_s$, $r_t(e_t)=-e_t$ and $r_t(e_s)=e_s+2c{e}_t$. Hence in the ordered basis $(e_s,e_t)$ the matrices are $[r_s]=\left(\begin{array}{cc}-1& 2c\\ 0& 1\end{array}\right)$ and $[r_t]=\left(\begin{array}{cc}1& 0\\ 2c& -1\end{array}\right)$, and the composite $A:=r_s{r}_t$ has matrix $[A]=[r_s][r_t]=\left(\begin{array}{cc}4c^2-1& -2c\\ 2c& -1\end{array}\right),\det [A]=(4c^2-1)(-1)-(2c)(-2c)=1.$

On $P=\mathbb{R}{e}_s+\mathbb{R}{e}_t$ write $x=x_s{e}_s+x_t{e}_t$. Bilinearity and the values $B(e_s,e_s)=B(e_t,e_t)=1$, $B(e_s,e_t)=-c$ give $B(x,x)=x_s^2-2c{x}_s{x}_t+x_t^2=(x_s-c{x}_t{)}^2+(1-c^2)x_t^2$. For finite $m$ one has $1-c^2={\sin }^2(\pi \mathrm{/}m)$ and $\sin (\pi \mathrm{/}m)\mathrm{\ne }0$ because $0<\pi \mathrm{/}m\le \pi \mathrm{/}2$ and the sine zero set is $\pi \mathbb{Z}$; hence $B(x,x)>0$ whenever $x\mathrm{\ne }0$, so $B{\mathrm{\mid }}_P$ is positive definite. For $m=\mathrm{\infty }$ one has $c=1$ and $B(x,x)=(x_s-x_t{)}^2\ge 0$, which vanishes exactly when $x_s=x_t$, i.e. on $\mathbb{R}(e_s+e_t)$; and $\mathbb{R}(e_s+e_t)$ is precisely the radical of $B{\mathrm{\mid }}_P$, since $B(x,e_s)=x_s-c{x}_t$ and $B(x,e_t)=x_t-c{x}_s$ vanish for all such $x$, while $x_s\mathrm{\ne }{x}_t$ makes $B(x,e_s)\mathrm{\ne }0$.

Assume $m<\mathrm{\infty }$ and put $\theta :=\pi \mathrm{/}m$, so $c=\cos \theta$. The trace of the matrix in 1.4 is $\mathrm{tr}[A]=4c^2-2=2(2c^2-1)=2\cos 2\theta$ by the double-angle instance of the addition formulas, and the $2\times 2$ Cayley-Hamilton identity, verified directly from the displayed entries, reads $A^2=2\cos 2\theta A-I_2$. For $m\ge 3$ one has $2\theta \in (0,\pi )$ and $\sin 2\theta \mathrm{\ne }0$, and for $m=2$ one has $c=\cos (\pi \mathrm{/}2)=0$, so $[A]=-I_2$.

Assume $m\ge 3$; then $\sin 2\theta \mathrm{\ne }0$ and induction on $k\ge 1$ using $A^2=2\cos 2\theta A-I_2$ and the product-to-sum identity $2\sin x\cos y=\sin (x+y)+\sin (x-y)$ gives $A^k=\frac{\sin (2k\theta )A-\sin (2(k-1)\theta )I_2}{\sin 2\theta }$ for every $k\ge 1$: the case $k=1$ is $\sin 0=0$, and the induction step replaces $A^{k+1}=A\cdot A^k$ using the displayed identity. Hence $A^m=I_2$ because $\sin (2m\theta )=\sin (2\pi )=0$ and $\sin (2(m-1)\theta )=-\sin 2\theta$. If $1\le k<m$ and $A^k=I_2$, then $\sin (2k\theta )A=(\sin 2\theta +\sin (2(k-1)\theta ))I_2$; were $\sin (2k\theta )\mathrm{\ne }0$, the matrix $A$ would be scalar, contradicting its nonzero off-diagonal entry $-2c$ (for $m\ge 3$ the value $c$ is nonzero, since $c=0$ would force $\pi \mathrm{/}m=(j+\frac{1}{2})\pi$, that is $2\mathrm{/}m=2j+1>0$ for an integer $j$, so $j\ge 0$ and $2\mathrm{/}m\ge 1$, impossible because $m\ge 3$). So $\sin (2k\theta )=0$, and then $\sin (2(k-1)\theta )=\sin (2k\theta -2\theta )=-\cos (2k\theta )\sin 2\theta$ forces $\cos (2k\theta )=1$; with $\sin (2k\theta )=0$ it gives $2k\theta \in 2\pi \mathbb{Z}$ (indeed $\sin x=0$ writes $x=\mathrm{\ell }\pi$, and $\cos (\mathrm{\ell }\pi )=(-1{)}^{\mathrm{\ell }}$ follows from $\cos 0=1$ and $\cos (x+\pi )=-\cos x$, so $\cos (2k\theta )=1$ forces $\mathrm{\ell }$ even), hence $m\mid k$, contradicting $1\le k<m$. Hence $A^k\mathrm{\ne }{I}_2$ for $0<k<m$.

Assume $m=\mathrm{\infty }$, so $c=1$ and $[A]=\left(\begin{array}{cc}3& -2\\ 2& -1\end{array}\right)=I_2+N$ with $N=\left(\begin{array}{cc}2& -2\\ 2& -2\end{array}\right)$. Then $N\mathrm{\ne }0$ and $N^2=0$ by direct multiplication, so the binomial theorem gives $A^k=I_2+kN$ for every $k\ge 0$, while $A^{-1}=I_2-N$ because $(I_2+N)(I_2-N)=I_2-N^2=I_2$, so $(I_2-N{)}^j=I_2+(-j)N$ for $j\ge 0$ and $A^k=I_2+kN$ for every $k\in \mathbb{Z}$. For $k\mathrm{\ne }0$ the matrix $kN$ has entry $2k\mathrm{\ne }0$ in its first column, so $A^k\mathrm{\ne }{I}_2$; the product $A=r_s{r}_t$ therefore has infinite order on $P$.

The Coxeter form $B$ of the Statement is well defined: the prescription $B(u,w):={\sum }_{s,t\in S}u(s)w(t)(-c(s,t))$ is a finite sum of scalar multiples of products of coordinate values, hence a function $V\times V\to \mathbb{R}$ linear in each variable; it is symmetric because $c(s,t)=c(t,s)$, which follows from $m(s,t)=m(t,s)$ and the symmetry of cosine; and it has $B(e_s,e_t)=-c(s,t)$ for all $s,t$. By 1.1 it is the unique symmetric bilinear form with these values, and $c(s,s)=\cos (\pi \mathrm{/}1)=\cos \pi =-1$ gives $B(e_s,e_s)=1$.

The maps $r_s$ and $r_t$ fix $P^{\perp }=\{v\in V:B(v,e_s)=B(v,e_t)=0\}$ pointwise, since $r_s(v)=v-\frac{2B(v,e_s)}{B(e_s,e_s)}{e}_s=v$ and likewise for $t$. If $m<\mathrm{\infty }$, then $V=P\oplus P^{\perp }$: the Gram matrix $\left(\begin{array}{cc}1& -c\\ -c& 1\end{array}\right)$ of $B{\mathrm{\mid }}_P$ has determinant $1-c^2={\sin }^2(\pi \mathrm{/}m)\mathrm{\ne }0$, so for given $v$ the $2\times 2$ system $B(p,e_s)=B(v,e_s)$, $B(p,e_t)=B(v,e_t)$ has a unique solution $p\in P$; then $z:=v-p$ satisfies $B(z,e_s)=B(z,e_t)=0$, hence $z\in P^{\perp }$ and $v=p+z$, so $P+P^{\perp }=V$. If $p\in P\cap P^{\perp }$ then $B(p,p)=0$ with $p\in P$, and positive definiteness of $B{\mathrm{\mid }}_P$ forces $p=0$; thus $P\cap P^{\perp }=\{0\}$ and $V=P\oplus P^{\perp }$.

Let $A=r_s{r}_t$. If $m<\mathrm{\infty }$, then $A^m{\mathrm{\mid }}_P=I_2$ and $A^k{\mathrm{\mid }}_P\mathrm{\ne }{I}_2$ for $0<k<m$ by steps 1.6 and 2.1 and the case $m=2$ of step 1.6, while $A{\mathrm{\mid }}_P$ has the matrix displayed in step 1.4; on $P^{\perp }$ the map $A$ fixes every vector by step 2.4, so $A^m={\mathrm{i}\mathrm{d}}_V$ and $A^k\mathrm{\ne }{\mathrm{i}\mathrm{d}}_V$ for $0<k<m$ because its restriction to the direct summand $P$ differs from $I_2$. If $m=\mathrm{\infty }$, then $[A]=I_2+N$ with $N\mathrm{\ne }0$, $N^2=0$ and $A^k{\mathrm{\mid }}_P=I_2+kN\mathrm{\ne }{I}_2$ for every $k\mathrm{\ne }0$ by step 2.2, so $A^k\mathrm{\ne }{\mathrm{i}\mathrm{d}}_V$ for every $k\mathrm{\ne }0$ and $r_s{r}_t$ has infinite order on $V$.