Schur orthogonality

For $A\in \mathrm{Hom}(V_{\sigma },V_{\pi })$ define $T(A):={\int }_G\pi (g)A\sigma (g{)}^{-1}d\mu (g)$, the entrywise integral of the continuous matrix-valued integrand; by [L4] this is a well-defined element of $\mathrm{Hom}(V_{\sigma },V_{\pi })$ and $A\mapsto T(A)$ is complex-linear.

For every $h\in G$ one has $T(A)\sigma (h)=\pi (h)T(A)$: indeed $T(A)\sigma (h)={\int }_G\pi (g)A\sigma (g{)}^{-1}\sigma (h)d\mu (g)={\int }_G\pi (g)A\sigma (g^{-1}h)d\mu (g)$, and substituting $g=h{g}^{\mathrm{\prime }}$ in the integral, which the left invariance [L3] permits, gives ${\int }_G\pi (h{g}^{\mathrm{\prime }})A\sigma (g^{\mathrm{\prime }-1})d\mu (g^{\mathrm{\prime }})=\pi (h)T(A)$.

In the special case $V_{\sigma }=V_{\pi }=V$ and $\sigma =\pi$, every $A\in \mathrm{End}(V)$ satisfies $\mathrm{tr}T(A)=\mathrm{tr}A$: by [L5] and [L4], $\mathrm{tr}T(A)={\int }_G\mathrm{tr}(\pi (g)A\pi (g{)}^{-1})d\mu (g)={\int }_G\mathrm{tr}Ad\mu (g)=\mathrm{tr}A,$ because $\mu$ is a probability measure.

Let $A=⟨\cdot ,e_l⟩f_j\in \mathrm{Hom}(V_{\sigma },V_{\pi })$. Then $A\sigma (g{)}^{-1}{e}_k=\overline{{\sigma }_{kl}(g)}{f}_j$ by [L6], and for all $i,k$, $⟨T(A)e_k,f_i⟩={\int }_G{\pi }_{ij}(g)\overline{{\sigma }_{kl}(g)}d\mu (g)\mathrm{.}$

If $\pi$ and $\sigma$ are inequivalent, then $T(A)=0$ for every $A$: by step 2.1 the map $T(A)$ intertwines the irreducible $\sigma$ with the irreducible $\pi$, so if it were nonzero it would be an isomorphism by Schur's lemma [L1], contradicting inequivalence.

If $V_{\sigma }=V_{\pi }=V$ and $\sigma =\pi$, then $T(A)=\lambda (A){\mathrm{id}}_V$ for every $A$: by step 2.1 the endomorphism $T(A)$ intertwines the irreducible representation $\pi$ with itself, so the intertwining endomorphisms form a division ring [L1]; if $T(A)\mathrm{\ne }0$, then $T(A)$ has an eigenvalue $\lambda$ by [L1] applied to the nonzero finite-dimensional complex space $V$, and $T(A)-\lambda {\mathrm{id}}_V$ is a non-injective intertwining endomorphism, hence $0$ in the division ring, so $T(A)=\lambda {\mathrm{id}}_V$ (the case $T(A)=0$ is the same statement with $\lambda =0$).

In the equal-representation case of the statement, use the fixed identification for which $V_{\sigma }=V_{\pi }=V$, $\sigma =\pi$, and $e_k=f_k$. Then the rank-one operator $A$ of step 2.3 is an endomorphism with $\mathrm{tr}A={\delta }_{jl}$. Combining steps 3.2 and 2.2 gives $T(A)=\lambda (A){\mathrm{id}}_V$ with $\lambda (A)=\mathrm{tr}A\mathrm{/}{d}_{\pi }={\delta }_{jl}\mathrm{/}{d}_{\pi }$, and hence $⟨T(A)e_k,f_i⟩={\delta }_{ik}{\delta }_{jl}\mathrm{/}{d}_{\pi }$.

In the general equivalent case fix an intertwining isomorphism $S:V_{\sigma }\to V_{\pi }$. The identity $\sigma (g{)}^{-1}{S}^{-1}=S^{-1}\pi (g{)}^{-1}$ shows that $T(A)S^{-1}={\int }_G\pi (g)(A{S}^{-1})\pi (g{)}^{-1}d\mu (g)$. Apply the equal-representation scalar and trace calculations of steps 3.2 and 2.2 to the endomorphism $A{S}^{-1}$ of $V_{\pi }$. They give $T(A)=S\mathrm{tr}(A{S}^{-1})\mathrm{/}{d}_{\pi }$. For the rank-one $A$ of step 2.3, the trace is $(S^{-1}{)}_{lj}$, so $⟨T(A)e_k,f_i⟩=S_{ik}(S^{-1}{)}_{lj}\mathrm{/}{d}_{\pi }$. No unitary normalization of $S$ is needed, and the formula holds for every such intertwiner.

Comparing step 2.3 with step 4.1 yields ${\int }_G{\pi }_{ij}(g)\overline{{\sigma }_{kl}(g)}d\mu (g)={\delta }_{ik}{\delta }_{jl}\mathrm{/}{d}_{\pi }$ in the equal-representation, aligned-basis case, and comparing step 2.3 with step 3.1 yields zero in the inequivalent case. Step 4.2 supplies the general equivalent case, so the alternatives exhaust all pairs. Irreducibility excludes the zero representation, hence $d_{\pi }>0$ and every division is defined. AC covers [L3] and the countable-choice terminology in [L6].