Mackey-Shoda irreducibility criterion for monomial representations

Assume, toward the contrapositive, that $\pi$ is not irreducible. Then there is a closed $\pi (G)$-invariant subspace $K$ with $\{0\}\mathrm{\ne }K\mathrm{\ne }{\mathrm{\ell }}^2(T)$; let $P$ be the orthogonal projection onto $K$ supplied by the decomposition ${\mathrm{\ell }}^2(T)=K\oplus K^{\perp }$ of [F5], so that $\mathrm{\parallel }x{\mathrm{\parallel }}^2=\mathrm{\parallel }Px{\mathrm{\parallel }}^2+\mathrm{\parallel }x-Px{\mathrm{\parallel }}^2$, $\mathrm{\parallel }P\mathrm{\parallel }\le 1$, and $⟨Px,y⟩=⟨Px,Py⟩=⟨x,Py⟩$ for all $x,y$, so $P$ is self-adjoint. Since $\pi (g)$ is unitary and $\pi (g)K\subseteq K$, also $\pi (g)(K^{\perp })\subseteq K^{\perp }$, because $⟨\pi (g)n,\pi (g)k⟩=⟨n,k⟩=0$ for $n\in K^{\perp }$ and $k\in K$; hence $P\pi (g)=\pi (g)P$ for every $g\in G$. Since $K\mathrm{\ne }\{0\}$ and $K\mathrm{\ne }{\mathrm{\ell }}^2(T)$, the operator $P$ is neither $0$ nor the identity and is therefore not a scalar operator.

Put $f:=P{\delta }_e\in {\mathrm{\ell }}^2(T)$. By the scalar criterion of [F3] applied to the self-intertwiner $P$, the vector $f$ is not a scalar multiple of ${\delta }_e$; hence there is $t\in T\setminus \{e\}$ with $f(t)\mathrm{\ne }0$. By the orbit criterion of [F3], every element of $T$ with nonzero $f$-value has finite $H$-orbit, so the $H$-orbit of $t$ is finite.

For $h\in H$ we have $t\cdot h=t$ exactly when $th{t}^{-1}=\alpha (t,h)\in H$, by the unique factorization $th=\alpha (t,h)(t\cdot h)$ of [F2]; hence the stabiliser of $t$ in $H$ is exactly $H\cap t^{-1}Ht$, and finiteness of the orbit gives $[H:H\cap t^{-1}Ht]<\mathrm{\infty }$. Moreover $t\mathrm{\notin }H$, because $t\in T\setminus \{e\}$ and $T$ meets the coset $H=He$ exactly in $e$.

Put $t^{\ast }:=e\cdot t^{-1}\in T$. The model formula of [F2] gives $\pi (t){\delta }_e=\chi (\alpha (t^{\ast },t)){\delta }_{t^{\ast }}$: indeed $\pi (t){\delta }_e={\sum }_{s\in T}\chi (\alpha (s,t)){\delta }_e(s\cdot t)$, and $s\cdot t=e$ holds for the unique $s\in T$ with $st\in H$, namely $s=t^{\ast }$. Since $\mathrm{\mid }\chi (\alpha (t^{\ast },t))\mathrm{\mid }=1$, the vectors $P{\delta }_{t^{\ast }}=\chi (\alpha (t^{\ast },t){)}^{-1}\pi (t)f$ and $\pi (t)f$ have equal norms; using that $P$ is self-adjoint, we get $\mathrm{\mid }f(t^{\ast })\mathrm{\mid }=\mathrm{\mid }⟨{\delta }_e,P{\delta }_{t^{\ast }}⟩\mathrm{\mid }=\mathrm{\mid }⟨{\delta }_e,\pi (t)f⟩\mathrm{\mid }=\mathrm{\mid }⟨\pi (t{)}^{\ast }{\delta }_e,f⟩\mathrm{\mid }=\mathrm{\mid }⟨{\delta }_t,f⟩\mathrm{\mid }=\mathrm{\mid }f(t)\mathrm{\mid }\mathrm{\ne }0$. The orbit criterion of [F3] applied to the point $t^{\ast }$ therefore shows that $t^{\ast }$ has finite $H$-orbit.

The stabiliser of $t^{\ast }$ in $H$ is $H\cap (t^{\ast }{)}^{-1}H{t}^{\ast }$ by the same computation as step 3.1. Since $t^{\ast }=e\cdot t^{-1}$ lies in the coset $H{t}^{-1}$, there is $h_1\in H$ with $t^{\ast }=h_1{t}^{-1}$, so $(t^{\ast }{)}^{-1}H{t}^{\ast }=tH{t}^{-1}$ and $H\cap (t^{\ast }{)}^{-1}H{t}^{\ast }=H\cap tH{t}^{-1}$. Step 4.1 therefore gives $[H:H\cap tH{t}^{-1}]<\mathrm{\infty }$; conjugating by $t$ gives $[t^{-1}Ht:t^{-1}Ht\cap H]<\mathrm{\infty }$, so with step 3.1 we obtain $t\in {\mathrm{Comm}}_G(H)\setminus H$.

Let $h\in H\cap t^{-1}Ht$; then $h\in H$ and $th{t}^{-1}\in H$, so $t\cdot h=t$ by step 3.1, while $f(t)\mathrm{\ne }0$. The stabiliser criterion of [F3] therefore gives $\chi (h)=\chi (th{t}^{-1})={\chi }^t(h)$: the characters $\chi$ and ${\chi }^t$ coincide on $H\cap t^{-1}Ht$, although $t\in {\mathrm{Comm}}_G(H)\setminus H$ by step 5.1. This contradicts the hypothesis of the Statement; the contrapositive is proved, so ${\mathrm{Ind}}_H^G\chi$ is irreducible.

If ${\mathrm{Comm}}_G(H)=H$, the assumed condition is vacuous and the irreducibility just proved applies to every unitary character of $H$. If $N⊴G$ is open, then $N\cap g^{-1}Ng=N$ has finite index in $N$ for every $g$, so ${\mathrm{Comm}}_G(N)=G$; the criterion therefore gives that ${\mathrm{Ind}}_N^G\chi$ is irreducible whenever ${\chi }^g\mathrm{\ne }\chi$ for every $g\in G\setminus N$, since here $N\cap g^{-1}Ng=N$ and the restriction of ${\chi }^g$ to $N$ is ${\chi }^g$ itself.

For the converse direction in the normal case, assume ${\chi }^{g_0}=\chi$ for some $g_0\in G\setminus N$; we shall construct a non-scalar bounded self-intertwiner of $\pi$. Every $x\in G$ has a unique factorization $x=nt$ with $n\in N$ and $t\in T$, because $G={⨆}_{t\in T}Nt$. Define $H_d:=\{F:G\to \mathbb{C}:F(nt)=\chi (n)F(t)\text{ for all }n\in N,t\in T,\text{ and }{\sum }_{t\in T}\mathrm{\mid }F(t){\mathrm{\mid }}^2<\mathrm{\infty }\}$ with norm $\mathrm{\parallel }F{\mathrm{\parallel }}_d^2:={\sum }_{t\in T}\mathrm{\mid }F(t){\mathrm{\mid }}^2$, and define $\mathrm{\Phi }:{\mathrm{\ell }}^2(T)\to H_d$ by $(\mathrm{\Phi }f)(nt):=\chi (n)f(t)$. Then $\mathrm{\Phi }$ is a linear bijection with $\mathrm{\parallel }\mathrm{\Phi }f{\mathrm{\parallel }}_d=\mathrm{\parallel }f\mathrm{\parallel }$ for all $f$, because every $F\in H_d$ is determined by its restriction to $T$.

Define $(DF)(x):=F(g_{0}x)$ for $F\in H_d$. For $x=nt$ write $g_{0}t=m_t\tau (t)$ with $\tau (t)\in T$ and $m_t=g_{0}t\tau (t{)}^{-1}\in N$; then $g_{0}x=(g_{0}n{g}_0^{-1})m_t\tau (t)$ with $g_{0}n{g}_0^{-1}\in N$ by normality, so $DF$ satisfies the covariance identity of $H_d$: for $n^{\mathrm{\prime }}\in N$, $(DF)(n^{\mathrm{\prime }}x)=F(g_0{n}^{\mathrm{\prime }}x)=\chi (g_0{n}^{\mathrm{\prime }}{g}_0^{-1})F(g_{0}x)=\chi (n^{\mathrm{\prime }})(DF)(x)$, using ${\chi }^{g_0}=\chi$. Since $t\mapsto g_{0}t$ induces the bijection $Nt\mapsto N{g}_{0}t$ of the coset space, with inverse induced by $g_0^{-1}$, the map $\tau$ is a bijection of $T$; hence $\mathrm{\parallel }DF{\mathrm{\parallel }}_d^2={\sum }_{t\in T}\mathrm{\mid }F(g_{0}t){\mathrm{\mid }}^2={\sum }_{t\in T}\mathrm{\mid }F(\tau (t)){\mathrm{\mid }}^2=\mathrm{\parallel }F{\mathrm{\parallel }}_d^2$, and $D$ is a surjective isometry of $H_d$.

The operator $D$ commutes with every right translation: with $(g\cdot F)(x):=F(xg)$ we get $(g\cdot (DF))(x)=(DF)(xg)=F(g_{0}xg)=(g\cdot F)(g_{0}x)=D(g\cdot F)(x)$ for all $x\in G$. Moreover $\mathrm{\Phi }$ intertwines the right-translation action with $\pi$, since $\mathrm{\Phi }(\pi (g)f)(nt)=\chi (n)\chi (\alpha (t,g))f(t\cdot g)$ and $(g\cdot \mathrm{\Phi }f)(nt)=(\mathrm{\Phi }f)(n\alpha (t,g)(t\cdot g))=\chi (n\alpha (t,g))f(t\cdot g)$ agree by the cocycle identity of [F2]. Hence $S:={\mathrm{\Phi }}^{-1}D\mathrm{\Phi }$ is a surjective isometry of ${\mathrm{\ell }}^2(T)$ satisfying $S\pi (g)=\pi (g)S$ for every $g\in G$.

With $F_e:=\mathrm{\Phi }{\delta }_e$, so that $F_e(nt)=\chi (n){\delta }_e(t)$, we get $(S{\delta }_e)(e)=(D{F}_e)(e)=F_e(g_0)$. Writing $g_0=n_0{t}_0$ with $n_0\in N$, $t_0\in T$, we have $t_0\mathrm{\ne }e$ because $g_0\mathrm{\notin }N$, so $F_e(g_0)=\chi (n_0){\delta }_e(t_0)=0\mathrm{\ne }1={\delta }_e(e)$. If $S=\lambda I$ then $\lambda =(S{\delta }_e)(e)=0$, contradicting that $S$ is a surjective isometry; thus $S$ is a non-scalar bounded self-intertwiner of $\pi$. By the contrapositive of Schur's lemma [F4], $\pi$ is not irreducible. This proves the converse direction, and with step 7.1 the stated equivalence for open normal subgroups follows.