The unitary dual of a locally compact group

Assume the Axiom of Choice. Let $G$ be a topological group. Two strongly continuous unitary representations $\pi$ on $H$ and $\rho$ on $K$ are unitarily equivalent when there is a unitary intertwiner $U:H\to K$ with $U\pi (g)=\rho (g)U$ for every $g\in G$ (Strongly continuous unitary representations, invariant linear subspaces and intertwiners). Irreducibility has the invariant-subspace meaning recalled there, so an irreducible representation acts on a nonzero Hilbert space. The unitary dual $\widehat{G}$ is the set of unitary equivalence classes of irreducible strongly continuous unitary representations of $G$. The zero representation is not an element of $\widehat{G}$, since it is not irreducible.

• Equivalence is an equivalence relation. Identity intertwiners give reflexivity, inverses of unitary intertwiners give symmetry, and compositions of unitary intertwiners give transitivity; irreducibility is a class property, so the phrase "classes of irreducible representations" is unambiguous.
• Why the dual is a set. Hilbert spaces form no set, so the classes are not taken over all carriers. Instead, let $P_1(G)\subseteq {\mathbb{C}}^G$ be the set of normalized continuous functions of positive type. If $\pi$ is irreducible and $\xi \mathrm{\ne }0$, the closed linear span of $\{\pi (g)\xi :g\in G\}$ is a nonzero closed invariant subspace (Cyclic vector and cyclic unitary representation), hence all of $H$; so $\xi$ is cyclic, and its normalized diagonal coefficient lies in $P_1(G)$. By Normalized positive type and pointed cyclic unitary representations the map from equivalence classes of pointed cyclic triples to $P_1(G)$ is a bijection, with inverse given by the GNS construction (GNS construction for a continuous positive-type function). The subset $I\subseteq P_1(G)$ of those $\phi$ whose GNS representation is irreducible is then a set, and $\widehat{G}$ is, equivalently, the image of $I$ under the assignment $\phi \mapsto {\pi }_{\phi }$ followed by passage to unitary equivalence: the quotient identifies two functions when their GNS representations are unitarily equivalent after forgetting the distinguished vectors. An intertwiner gives matching unit vectors by transporting one chosen vector to the other carrier, and every irreducible class contains a normalized cyclic pointed representative. This realizes $\widehat{G}$ as a quotient of the set $I$, with no dimension bound assumed.
• Compact groups. When $G$ is compact the same construction applies verbatim and gives the usual dual of a compact group; no separability is assumed.
• Choice. The Axiom of Choice is inherited from the GNS construction and from Schur's lemma, which are the only steps of the construction that use it (The Axiom of Choice).