Finiteness criterion: W is finite exactly when the Coxeter form is positive definite

(The dual form.) The case $S=\mathrm{\varnothing }$ is trivial: then $V=\{0\}$, $W=\{1\}$ is finite by [F12] and $B$ is positive definite vacuously, so assume $S\mathrm{\ne }\mathrm{\varnothing }$ and put $n:=\mathrm{\mid }S\mathrm{\mid }\ge 1$. Assume $B$ positive definite. If $B(v,x)=0$ for all $x$ then $B(v,v)=0$, so $v=0$ by [F8]; hence the linear map $b:V\to V^{\ast }$, $b(v):=B(v,\cdot )$ [F9], has kernel $\{0\}$ and, since $\dim V^{\ast }=\dim V=n$ is finite, it is a linear isomorphism [F9]. The form $B^{\ast }(f,g):=B(b^{-1}f,b^{-1}g)$ is symmetric and bilinear, and positive definite because $f\mathrm{\ne }0$ gives $b^{-1}f\mathrm{\ne }0$ and $B^{\ast }(f,f)=B(b^{-1}f,b^{-1}f)>0$ [F8]. For $w\in W$ and $v\in V$ the two functionals $b(\rho (w)v)$ and $w\cdot b(v)$ agree: at $x$ both take the value $B(\rho (w)v,x)=B(v,\rho (w{)}^{-1}x)$, by [F3] applied to the pair $\rho (w{)}^{-1}x$ and by the dual-action formula [F4]; hence $b^{-1}(w\cdot f)=\rho (w)b^{-1}f$ and $B^{\ast }(w\cdot f,w\cdot g)=B(\rho (w)b^{-1}f,\rho (w)b^{-1}g)=B(b^{-1}f,b^{-1}g)=B^{\ast }(f,g)$ for all $f,g$, so the dual action preserves $B^{\ast }$ [F3]. This is clause (2).

(Isolation of the identity.) Let $f\in C^{\circ }$, nonempty by [F5], and put ${\mathrm{\Omega }}_f:=\{h\in \mathrm{G}\mathrm{L}(V^{\ast }):h(f)\in C^{\circ }\}$ and ${\mathrm{\Omega }}_f^{\mathrm{\prime }}:=\{g\in \mathrm{G}\mathrm{L}(V):f\circ g\in C^{\circ }\}$. $C^{\circ }$ is open in $V^{\ast }$: it is the finite intersection of the sets $\{f^{\mathrm{\prime }}:f^{\mathrm{\prime }}(e_s)>0\}$ [F4], each the preimage of the open interval $(0,\mathrm{\infty })\subseteq \mathbb{R}$ under the coordinate functional $f^{\mathrm{\prime }}\mapsto f^{\mathrm{\prime }}(e_s)$, which is continuous since its difference is bounded by the maximum coordinate difference [F10]; the maps $h\mapsto h(f)$ and $g\mapsto f\circ g$ are linear on the finite-dimensional spaces $\mathrm{E}\mathrm{n}\mathrm{d}(V^{\ast })$ and $\mathrm{E}\mathrm{n}\mathrm{d}(V)$ and hence continuous (each coordinate is a finite sum of matrix entries multiplied by fixed coordinates) [F9, F10], so ${\mathrm{\Omega }}_f$ and ${\mathrm{\Omega }}_f^{\mathrm{\prime }}$ are open neighbourhoods of the identities, because $f\in C^{\circ }$. If ${\rho }^{\ast }(w)\in {\mathrm{\Omega }}_f$, that is $w\cdot f={\rho }^{\ast }(w)f\in C^{\circ }$, then $f\in C$ and $w\cdot f\in C$, and the collision theorem [F7] gives $w\in W_{S(f)}$ with $S(f)=\mathrm{\varnothing }$ because $f\in C^{\circ }$; as $W_{\mathrm{\varnothing }}=\{1\}$ [F12], $w=1$ and ${\mathrm{\Omega }}_f\cap {\rho }^{\ast }(W)=\{\mathrm{i}\mathrm{d}\}$. If instead $\rho (w)\in {\mathrm{\Omega }}_f^{\mathrm{\prime }}$, then $w^{-1}\cdot f=f\circ \rho (w)\in C^{\circ }$ by [F4], so the same argument with $w^{-1}$ gives $w^{-1}\in W_{S(f)}=\{1\}$ and $\rho (w)={\mathrm{i}\mathrm{d}}_V$; hence ${\mathrm{\Omega }}_f^{\mathrm{\prime }}\cap \rho (W)=\{{\mathrm{i}\mathrm{d}}_V\}$. No positive definiteness is used in this step.

(Finite groups have positive definite form.) If $W$ is finite then $B$ is positive definite by [F1]; this proves the finite-$W$ to positive-definite-$B$ direction of (1), and no other argument is needed for it.

(Closedness and boundedness of $O(B^{\ast })$.) Assume $B$ positive definite and let $B^{\ast }$ be the form of step 1.1; put $O(B^{\ast }):=\{h\in \mathrm{G}\mathrm{L}(V^{\ast }):B^{\ast }(h\phi ,h\psi )=B^{\ast }(\phi ,\psi )\mathrm{\forall }\phi ,\psi \in V^{\ast }\}$. Identify $\mathrm{E}\mathrm{n}\mathrm{d}(V^{\ast })$ with ${\mathbb{R}}^{n^2}$ by the dual basis $(f_s)$ of [F9] [F10]. For each pair $i,j$ the function $h\mapsto B^{\ast }(h{f}_i,h{f}_j)$ is a finite sum of products ${\sum }_{p,q}{h}_{pi}{h}_{qj}{B}^{\ast }(f_p,f_q)$ of entries of $h$ with constants, hence continuous [F10]; Any endomorphism preserving $B^{\ast }$ is injective: $h\phi =0$ implies $B^{\ast }(\phi ,\phi )=0$ and hence $\phi =0$; it is invertible by rank-nullity in finite dimension [F9]. Thus $O(B^{\ast })$ is exactly the preimage of the single point $(B^{\ast }(f_i,f_j){)}_{ij}$ under a continuous map $\mathrm{E}\mathrm{n}\mathrm{d}(V^{\ast })\to {\mathbb{R}}^{n^2}$, hence closed [F10]. For boundedness fix an orthonormal basis $({\phi }_1,\dots ,{\phi }_n)$ of $V^{\ast }$ for $B^{\ast }$ [F11]; if $h\in O(B^{\ast })$ and $h{\phi }_i={\sum }_j{a}_{ji}{\phi }_j$, then $a_{ji}=B^{\ast }(h{\phi }_i,{\phi }_j)$ by the orthonormal expansion [F11], so $\mathrm{\mid }{a}_{ji}\mathrm{\mid }\le \mathrm{\parallel }h{\phi }_i\mathrm{\parallel }\mathrm{\parallel }{\phi }_j\mathrm{\parallel }=1$ by Cauchy-Schwarz because $B^{\ast }(h{\phi }_i,h{\phi }_i)=B^{\ast }({\phi }_i,{\phi }_i)=1$ [F11]; hence all matrix entries in this orthonormal basis are bounded by $1$. A change to the fixed dual basis expresses each new entry as a finite linear combination of these entries with fixed coefficients; its absolute value is bounded by the sum of the absolute values of those coefficients. Therefore $O(B^{\ast })$ is bounded in the original ${\mathbb{R}}^{n^2}$ coordinates [F10].

(Discreteness of both images.) Let $\gamma \in {\rho }^{\ast }(W)$. The set $\gamma {\mathrm{\Omega }}_f=\{h\in \mathrm{G}\mathrm{L}(V^{\ast }):{\gamma }^{-1}h\in {\mathrm{\Omega }}_f\}$ is open in $\mathrm{G}\mathrm{L}(V^{\ast })$ as the preimage of the open set ${\mathrm{\Omega }}_f$ under the continuous map $h\mapsto {\gamma }^{-1}h$ [F10], and it contains $\gamma$; if $w=\gamma h\in {\rho }^{\ast }(W)$ with $h\in {\mathrm{\Omega }}_f$, then $h={\gamma }^{-1}w\in {\rho }^{\ast }(W)$ (a subgroup) and step 1.2 forces $h=\mathrm{i}\mathrm{d}$, so $w=\gamma$. Hence $\gamma {\mathrm{\Omega }}_f\cap {\rho }^{\ast }(W)=\{\gamma \}$ for every $\gamma$, so each point of ${\rho }^{\ast }(W)$ is open in the subspace topology and ${\rho }^{\ast }(W)$ is discrete [F10]. The same translation argument with ${\mathrm{\Omega }}_f^{\mathrm{\prime }}$ shows that each point of $\rho (W)$ is isolated, so $\rho (W)$ is discrete as well. This proves clause (3), and no positive definiteness was used.

(Continuity of the operations, and a symmetric neighbourhood squaring into ${\mathrm{\Omega }}_f$.) With $\mathrm{E}\mathrm{n}\mathrm{d}(V^{\ast })\cong {\mathbb{R}}^{n^2}$ and $\mathrm{E}\mathrm{n}\mathrm{d}(V)\cong {\mathbb{R}}^{n^2}$ as in [F10], matrix multiplication has entries that are finite sums of products of entries of the factors, hence is continuous [F10], and the entries of the inverse are given by the adjugate formula $h^{-1}=\det (h{)}^{-1}\mathrm{adj}(h)$ [F11], a polynomial in the entries divided by the continuous function $\det$, which is nonzero on $\mathrm{G}\mathrm{L}$ [F10]; hence multiplication and inversion are continuous on the general linear groups [F10], and $\mathrm{G}\mathrm{L}(V^{\ast })$ is open in $\mathrm{E}\mathrm{n}\mathrm{d}(V^{\ast })$ as the preimage of $\mathbb{R}\setminus \{0\}$ under $\det$ [F10, F11]. Since multiplication sends $(\mathrm{i}\mathrm{d},\mathrm{i}\mathrm{d})$ to $\mathrm{i}\mathrm{d}\in {\mathrm{\Omega }}_f$ with ${\mathrm{\Omega }}_f$ open [step 1.2], there are open neighbourhoods $V_1,V_2\subseteq \mathrm{G}\mathrm{L}(V^{\ast })$ of $\mathrm{i}\mathrm{d}$ with $m(V_1\times V_2)\subseteq {\mathrm{\Omega }}_f$: the open preimage $m^{-1}({\mathrm{\Omega }}_f)$ contains a maximum-coordinate ball around $(\mathrm{i}\mathrm{d},\mathrm{i}\mathrm{d})$ in the paired matrix coordinates, and such a ball is a product of two balls about $\mathrm{i}\mathrm{d}$ [F10]; then $U:=V_1\cap V_2\cap V_1^{-1}\cap V_2^{-1}$ (where $W^{-1}:=\{h^{-1}:h\in W\}$) is an open symmetric neighbourhood of $\mathrm{i}\mathrm{d}$ in $\mathrm{G}\mathrm{L}(V^{\ast })$, since $U=U^{-1}$, with $U\cdot U\subseteq V_1\cdot V_2\subseteq {\mathrm{\Omega }}_f$ [F10].

(Compactness of $O(B^{\ast })$.) By step 2.1, $O(B^{\ast })$ is a closed and bounded subset of $\mathrm{E}\mathrm{n}\mathrm{d}(V^{\ast })\cong {\mathbb{R}}^{n^2}$; by the Heine-Borel theorem a subset of ${\mathbb{R}}^{n^2}$ is compact if and only if it is closed and bounded [F10], so $O(B^{\ast })$ is compact.

(Positive definite $B$ gives finite $W$; conclusion.) Assume $B$ positive definite and $S\mathrm{\ne }\mathrm{\varnothing }$ as in step 1.1; put $\mathrm{\Gamma }:={\rho }^{\ast }(W)$, a subgroup of $\mathrm{G}\mathrm{L}(V^{\ast })$ that is contained in $O(B^{\ast })$ by the invariance proved in step 1.1 [step 1.1]. Let $f\in C^{\circ }$ and let $U$ be the open symmetric neighbourhood of $\mathrm{i}\mathrm{d}$ with $U\cdot U\subseteq {\mathrm{\Omega }}_f$ from step 2.3 [step 2.3]. Each $hU$ ($h\in O(B^{\ast })$) is open in $\mathrm{E}\mathrm{n}\mathrm{d}(V^{\ast })$, being the image of the open set $U$ under the linear isomorphism $u\mapsto hu$ with inverse $k\mapsto h^{-1}k$ [F10, F11], so the family $\{hU\cap O(B^{\ast }):h\in O(B^{\ast })\}$ is an open cover of the compact space $O(B^{\ast })$ from step 3.1 [step 3.1]; choose a finite subcover, say $O(B^{\ast })={\bigcup }_{i=1}^N(h_{i}U\cap O(B^{\ast }))$ [F10]. Each $h_{i}U$ contains at most one element of $\mathrm{\Gamma }$: if $w_1=h_i{u}_1$ and $w_2=h_i{u}_2$ with $u_1,u_2\in U$ and $w_1,w_2\in \mathrm{\Gamma }$, then $w_2^{-1}{w}_1=u_2^{-1}{u}_1\in U\cdot U\subseteq {\mathrm{\Omega }}_f$ because $U=U^{-1}$, and $w_2^{-1}{w}_1\in \mathrm{\Gamma }$, so step 1.2 gives $w_2^{-1}{w}_1=\mathrm{i}\mathrm{d}$ and $w_1=w_2$ [step 1.2]. Hence $\mathrm{\Gamma }$ has at most $N$ elements, and faithfulness of the dual action [F6] gives $\mathrm{\mid }W\mathrm{\mid }=\mathrm{\mid }\mathrm{\Gamma }\mathrm{\mid }\le N<\mathrm{\infty }$. Therefore $W$ finite if and only if $B$ is positive definite, which is (1); clause (2) is step 1.1, clause (3) is steps 1.2 and 2.2, and the finite-$W$ direction of (1) is step 1.3.