Mackey-Shoda non-equivalence criterion for monomial representations

Assume, toward the contrapositive, that ${\pi }_1$ and ${\pi }_2$ are unitarily equivalent, and let $S:{\mathrm{\ell }}^2(T_1)\to {\mathrm{\ell }}^2(T_2)$ be a unitary intertwiner, so that $S\mathrm{\ne }0$ and $S^{\ast }=S^{-1}$ satisfies $S^{\ast }{\pi }_2(g)={\pi }_1(g)S^{\ast }$ for all $g\in G$. Put $f:=S{\delta }_e\in {\mathrm{\ell }}^2(T_2)$; by the first clause of [F3], $f\mathrm{\ne }0$.

The second clause of [F3] shows that $f$ vanishes on every $t\in T_2$ with infinite $H_1$-orbit, so we may choose $t\in T_2$ with $f(t)\mathrm{\ne }0$ and finite $H_1$-orbit.

For $h\in H_1$ the identity $t{\cdot }_{2}h=t$ holds exactly when $th{t}^{-1}={\alpha }_2(t,h)\in H_2$, by the unique factorization $th={\alpha }_2(t,h)(t{\cdot }_{2}h)$ of [F2]; hence the stabiliser of $t$ in $H_1$ equals $H_1\cap t^{-1}{H}_{2}t$, and finiteness of the $H_1$-orbit gives $[H_1:t^{-1}{H}_{2}t\cap H_1]<\mathrm{\infty }$.

Put $t^{\ast }:=e{\cdot }_1{t}^{-1}\in T_1$ and $f^{\mathrm{\prime }}:=S^{\ast }{\delta }_e\in {\mathrm{\ell }}^2(T_1)$. The model formula of [F2] for ${\pi }_1$ gives ${\pi }_1(t){\delta }_e={\chi }_1({\alpha }_1(t^{\ast },t)){\delta }_{t^{\ast }}$, and therefore ${\delta }_{t^{\ast }}={\chi }_1({\alpha }_1(t^{\ast },t){)}^{-1}{\pi }_1(t){\delta }_e$ and $S{\delta }_{t^{\ast }}={\chi }_1({\alpha }_1(t^{\ast },t){)}^{-1}{\pi }_2(t)f$, since $S{\pi }_1(t)={\pi }_2(t)S$. It follows that $\mathrm{\mid }{f}^{\mathrm{\prime }}(t^{\ast })\mathrm{\mid }=\mathrm{\mid }⟨S^{\ast }{\delta }_e,{\delta }_{t^{\ast }}⟩\mathrm{\mid }=\mathrm{\mid }⟨{\delta }_e,S{\delta }_{t^{\ast }}⟩\mathrm{\mid }=\mathrm{\mid }⟨{\delta }_e,{\pi }_2(t)f⟩\mathrm{\mid }=\mathrm{\mid }⟨{\pi }_2(t{)}^{\ast }{\delta }_e,f⟩\mathrm{\mid }=\mathrm{\mid }⟨{\delta }_t,f⟩\mathrm{\mid }=\mathrm{\mid }f(t)\mathrm{\mid }\mathrm{\ne }0$, because $\mathrm{\mid }{\chi }_1({\alpha }_1(t^{\ast },t))\mathrm{\mid }=1$ and ${\pi }_2(t{)}^{\ast }={\pi }_2(t{)}^{-1}={\pi }_2(t^{-1})$ has ${\pi }_2(t^{-1}){\delta }_e={\delta }_t$. Applying the second clause of [F3] to the intertwiner $S^{\ast }$ of ${\pi }_2$ with ${\pi }_1$ shows that $t^{\ast }$ has finite $H_2$-orbit.

The stabiliser of $t^{\ast }$ in $H_2$ is $H_2\cap (t^{\ast }{)}^{-1}{H}_1{t}^{\ast }$ by the same computation as step 3.1, applied to ${\pi }_1$ and the subgroup $H_2$ acting on $T_1$. Since $t^{\ast }=e{\cdot }_1{t}^{-1}$ lies in the coset $H_1{t}^{-1}$, there is $h_1\in H_1$ with $t^{\ast }=h_1{t}^{-1}$, hence $(t^{\ast }{)}^{-1}{H}_1{t}^{\ast }=t{H}_1{t}^{-1}$ and $H_2\cap (t^{\ast }{)}^{-1}{H}_1{t}^{\ast }=H_2\cap t{H}_1{t}^{-1}$. Step 4.1 therefore gives $[H_2:t{H}_1{t}^{-1}\cap H_2]<\mathrm{\infty }$, and conjugating by $t$ gives $[t^{-1}{H}_{2}t:t^{-1}{H}_{2}t\cap H_1]<\mathrm{\infty }$; with step 3.1, $t^{-1}{H}_{2}t\cap H_1$ has finite index in both $t^{-1}{H}_{2}t$ and $H_1$.

Let $h\in t^{-1}{H}_{2}t\cap H_1$; then $th{t}^{-1}\in H_2$, so $t{\cdot }_{2}h=t$ by step 3.1, and $f(t)\mathrm{\ne }0$. The third clause of [F3] therefore gives ${\chi }_1(h)={\chi }_2(th{t}^{-1})={\chi }_2^t(h)$: the restrictions of ${\chi }_2^t$ and ${\chi }_1$ to $t^{-1}{H}_{2}t\cap H_1$ coincide, although this intersection has finite index in both $t^{-1}{H}_{2}t$ and $H_1$ by step 5.1. This contradicts the hypothesis of the Statement at $g=t$; the contrapositive is proved, so the two representations are not equivalent.

Finally, if $g^{-1}{H}_{2}g\cap H_1$ has infinite index in $H_1$ for every $g\in G$, then no $g$ satisfies the finite-index hypothesis of the Statement, so the criterion applies vacuously and the two representations are inequivalent; this covers in particular the case in which the intersection is trivial and $H_1$ is infinite, since then $[H_1:\{e\}]=\mathrm{\mid }{H}_1\mathrm{\mid }=\mathrm{\infty }$.