Parabolic Mackey formula for finite GL_n

The groups attached to $\rho$. Fix $\rho \in S_n$, put $w:=w_{\rho }$ and $\kappa (k):={\mathrm{blk}}_{\beta }({\rho }^{-1}(k))$ for $1\le k\le n$, so that $g\in M_{\rho }=wM{w}^{-1}$ holds exactly when $(w^{-1}gw{)}_{uv}=0$ for all $u,v$ with ${\mathrm{blk}}_{\beta }(u)\mathrm{\ne }{\mathrm{blk}}_{\beta }(v)$; since $(w^{-1}gw{)}_{uv}=g_{\rho (u),\rho (v)}$ and ${\mathrm{blk}}_{\beta }(u)$ for $u={\rho }^{-1}(k)$ is $\kappa (k)$ by [L2], the entry criteria of [L1] for the composition $\beta$ give $M_{\rho }=\{g:g_{kl}=0\text{ whenever }\kappa (k)\mathrm{\ne }\kappa (l)\},V_{\rho }=\{g:g_{kk}=1\text{ for all }k,g_{kl}=0\text{ whenever }k\mathrm{\ne }l\text{ and }\kappa (k)\ge \kappa (l)\},$ and intersecting with the criteria of [L1] for the composition $\alpha$ yields $C_{\rho }=\{g\in G:g_{kl}=0\text{ whenever }{\mathrm{blk}}_{\alpha }(k)\mathrm{\ne }{\mathrm{blk}}_{\alpha }(l)\text{ or }\kappa (k)\mathrm{\ne }\kappa (l)\},$ $D_{\rho }=\{g:g_{kk}=1,g_{kl}=0\text{ whenever }k\mathrm{\ne }l\text{ and }({\mathrm{blk}}_{\alpha }(k)\ge {\mathrm{blk}}_{\alpha }(l)\text{ or }\kappa (k)\mathrm{\ne }\kappa (l))\},$ $A_{\rho }=\{g:g_{kk}=1,g_{kl}=0\text{ whenever }k\mathrm{\ne }l\text{ and }({\mathrm{blk}}_{\alpha }(k)\mathrm{\ne }{\mathrm{blk}}_{\alpha }(l)\text{ or }\kappa (k)\ge \kappa (l))\}\mathrm{.}$

Transport data for coordinate parabolics. Let $\gamma ,\delta ,\sigma ,\tau ,u,v$ be as in claim 3; then $P_{\gamma }=u{P}_{\alpha }{u}^{-1}$, $L_{\gamma }=uL{u}^{-1}$, $U_{\gamma }=uU{u}^{-1}$, $P_{\delta }=vQ{v}^{-1}$, $L_{\delta }=vM{v}^{-1}$, $U_{\delta }=vV{v}^{-1}$, $\gamma =\sigma (\alpha )$ and $\delta =\tau (\beta )$ by [L7], the permutation matrices satisfy $w_{{\rho }^{\mathrm{\prime }}}=P_{\sigma \rho {\tau }^{-1}}=P_{\sigma }{P}_{\rho }{P}_{{\tau }^{-1}}=u{w}_{\rho }{v}^{-1}$ and $w_{{\rho }^{\mathrm{\prime }}}^{-1}=v{w}_{\rho }^{-1}{u}^{-1}$ by [L2], and $\rho \mapsto {\rho }^{\mathrm{\prime }}=\sigma \rho {\tau }^{-1}$ is a bijection of $S_n$ with inverse ${\rho }^{\mathrm{\prime }}\mapsto {\sigma }^{-1}{\rho }^{\mathrm{\prime }}\tau$ by [L8]; since $W_{\gamma }=\sigma W_{\alpha }{\sigma }^{-1}$ and $W_{\delta }=\tau W_{\beta }{\tau }^{-1}$ by [L7], this bijection carries the $(W_{\alpha },W_{\beta })$-double coset of $\rho$ onto the $(W_{\gamma },W_{\delta })$-double coset of ${\rho }^{\mathrm{\prime }}$, so the images of $\mathcal{R}$ are one per $(W_{\gamma },W_{\delta })$-double coset of $S_n$.

The $U$-invariants as covariant functions. Let $Y:=R_M^{G}X={\mathrm{Ind}}_Q^G({\mathrm{Inf}}_M^{Q}X)$ and let $\pi :Q\to M$ be the projection with kernel $V$, so that $Y$ is the set of functions $f:G\to X$ with $f(gq)=\pi (q{)}^{-1}f(g)$ for all $g\in G$, $q\in Q$, the $U$-invariants are $Y^U=\{f\in Y:f(ug)=f(g)\text{ for all }u\in U\}$, and $Y^U$ carries the $L$-action $(l\cdot f)(g)=f(l^{-1}g)$ by [L5]; define $\mathcal{F}:=\{\phi :U\mathrm{\backslash }G\to X:\phi (Ug\cdot q)=\pi (q{)}^{-1}\phi (Ug)\text{ for all }g\in G,q\in Q\},$ the right action being $(Ug)\cdot q:=Ugq$ of [L9], with the left action $(l\cdot \phi )(Ug):=\phi (U{l}^{-1}g)$ of [L6]. Then $\mathrm{\Psi }(f)(Ug):=f(g)$ is a bijection $Y^U\to \mathcal{F}$: it is well defined and injective because $Ug=U{g}^{\mathrm{\prime }}$ exactly when $g^{\mathrm{\prime }}=ug$ with $u\in U$ and $f(ug)=f(g)$; it is surjective because for $\phi \in \mathcal{F}$ the formula $f_{\phi }(g):=\phi (Ug)$ defines an element of $Y^U$ with $\mathrm{\Psi }(f_{\phi })=\phi$, since $f_{\phi }(gq)=\phi (Ug\cdot q)=\pi (q{)}^{-1}{f}_{\phi }(g)$ and $f_{\phi }(ug)=\phi (Uug)=\phi (Ug)$; and it is $L$-equivariant because $\mathrm{\Psi }(l\cdot f)(Ug)=(l\cdot f)(g)=f(l^{-1}g)=\mathrm{\Psi }(f)(U{l}^{-1}g)=(l\cdot \mathrm{\Psi }(f))(Ug)$ for the well-defined left action of [L6].

Decomposition along the double cosets. By [L3] the double cosets $P_{\alpha }{w}_{\rho }Q$ with $\rho \in \mathcal{R}$ are pairwise disjoint and cover $G$, hence the sets $Z_{\rho }:=\{Ux:x\in P_{\alpha }{w}_{\rho }Q\}=U\mathrm{\backslash }(P_{\alpha }{w}_{\rho }Q)$ of right cosets are pairwise disjoint, nonempty ($U{w}_{\rho }\in Z_{\rho }$) and cover $U\mathrm{\backslash }G$; each $Z_{\rho }$ is stable under the left action of $L$, because $l\cdot (Ux)=Ulx$ with $lx\in P_{\alpha }{w}_{\rho }Q$ for $l\in L\le P_{\alpha }$ by [L6], and applying the same to $l^{-1}$ gives $l\cdot Z_{\rho }=Z_{\rho }$. Writing ${\mathcal{F}}_{\rho }:=\{\phi \in \mathcal{F}:\phi (z)=0\text{ for all }z\mathrm{\notin }{Z}_{\rho }\}$, the support $\{z:\phi (z)\mathrm{\ne }0\}$ of $\phi \in \mathcal{F}$ is a union of right $Q$-orbits, since $\phi (z)\mathrm{\ne }0$ implies $\phi (zq)=\pi (q{)}^{-1}\phi (z)\mathrm{\ne }0$; because $Z_{\rho }\cdot Q=Z_{\rho }$ for every $\rho$, every $\phi$ is the unique sum of its restrictions ${\phi }_{\rho }\in {\mathcal{F}}_{\rho }$, and each ${\mathcal{F}}_{\rho }$ is $L$-stable because $Z_{\rho }$ is; hence $\mathcal{F}={⨁}_{\rho \in \mathcal{R}}{\mathcal{F}}_{\rho }$ as $L$-modules.

The first intersected parabolic. Let $\rho \in S_n$ and $g\in P_{\alpha }\cap M_{\rho }$. Write $g=lu$ with $l\in L=L_{\alpha }$ and $u\in U=U_{\alpha }$, the unique factorisation of [L1]; then $l$ is the $\alpha$-block diagonal part of $g$, because for ${\mathrm{blk}}_{\alpha }(k)={\mathrm{blk}}_{\alpha }(l)$ the entry $(lu{)}_{kl}$ equals $l_{kl}{u}_{ll}=l_{kl}$ by the entry criteria of [L1] for $U_{\alpha }$, while $l_{kl}=0$ for ${\mathrm{blk}}_{\alpha }(k)\mathrm{\ne }{\mathrm{blk}}_{\alpha }(l)$ by the entry criteria of [L1] for $L_{\alpha }$; by step 1.1 the matrix $g$ satisfies $g_{kl}=0$ whenever $\kappa (k)\mathrm{\ne }\kappa (l)$, and $l$ inherits this because its entries are entries of $g$ inside $\alpha$-blocks, so $l\in L\cap M_{\rho }=C_{\rho }$; also $d:=l^{-1}g=u$ lies in $U$ and in $M_{\rho }$, because $l\in M_{\rho }$ and $g\in M_{\rho }$, so $d\in U\cap M_{\rho }=D_{\rho }$ and $g=ld\in C_{\rho }{D}_{\rho }$. Conversely $C_{\rho }\le L\le P_{\alpha }$ and $D_{\rho }\le U\le P_{\alpha }$ by [L1], and both lie in $M_{\rho }$, so $P_{\alpha }\cap M_{\rho }=C_{\rho }{D}_{\rho }$. Inside each coordinate block $\rho (J_j)$ of $M_{\rho }$, its matrices are upper block triangular on the nonempty intersections $I_i\cap \rho (J_j)$ ordered by $i$. Thus it is a coordinate parabolic of $M_{\rho }$ with Levi $C_{\rho }$ and radical $D_{\rho }$; this is the internal semidirect product $C_{\rho }⋉D_{\rho }$, because $C_{\rho }$ normalises $D_{\rho }$ by [L4] and $C_{\rho }\cap D_{\rho }\subseteq L\cap U=\{I_n\}$ by [L1].

The second intersected parabolic. Let $\rho \in S_n$ and $g\in L\cap w_{\rho }Q{w}_{\rho }^{-1}$. By [L1] the element $g$ is $\alpha$-block diagonal, and since $w_{\rho }Q{w}_{\rho }^{-1}=w_{\rho }(MV)w_{\rho }^{-1}=M_{\rho }{V}_{\rho }$ there are $m\in M_{\rho }$ and $a\in V_{\rho }$ with $g=ma$; by step 1.1 the matrix $m$ is $\kappa$-block diagonal, so for $\kappa (k)=\kappa (l)$ the entry $(ma{)}_{kl}$ equals $m_{kl}{a}_{ll}=m_{kl}$, every other term $m_{ki}{a}_{il}$ of the sum vanishing because $m_{ki}=0$ for $\kappa (k)\mathrm{\ne }\kappa (i)$ or because $a_{il}=0$ for $i\mathrm{\ne }l$ with $\kappa (i)=\kappa (l)$; hence $m$ is the $\kappa$-block diagonal part of $g$ and, its entries being entries of the $\alpha$-block diagonal matrix $g$, it lies in $L\cap M_{\rho }=C_{\rho }$, while $a=m^{-1}g$ lies in $L$, as a product of elements of $L$, and in $V_{\rho }$, that is $a\in L\cap V_{\rho }=A_{\rho }$; so $g=ma\in C_{\rho }{A}_{\rho }$. Conversely $C_{\rho }\subseteq M_{\rho }\subseteq w_{\rho }Q{w}_{\rho }^{-1}$ and $A_{\rho }\subseteq V_{\rho }\subseteq w_{\rho }Q{w}_{\rho }^{-1}$, and both lie in $L$, so $L\cap w_{\rho }Q{w}_{\rho }^{-1}=C_{\rho }{A}_{\rho }$, which is the internal semidirect product $C_{\rho }⋉A_{\rho }$ by [L4]. Inside each block $I_i$ of $L$, the nonempty intersections $I_i\cap \rho (J_j)$ ordered by $j$ give precisely this upper block triangular subgroup, so it is a coordinate parabolic of $L$ with Levi $C_{\rho }$ and radical $A_{\rho }$.

The piece is a space of $M$-covariant functions on the biset. Fix $\rho \in \mathcal{R}$; by step 2.1 the sets $Z_{\rho }$ and the spaces ${\mathcal{F}}_{\rho }$ are as defined there, and by [L1] and [L5] the projection $\pi :Q\to M$ has kernel $V$ and satisfies $\pi (m)=m$ for $m\in M$, so for $\phi \in {\mathcal{F}}_{\rho }$ and $v\in V$ one has $\phi (zv)=\pi (v{)}^{-1}\phi (z)=\phi (z)$: the function $\phi$ is constant on the right $V$-orbits of $Z_{\rho }$, which are the sets $\{Uxv:v\in V\}=UxV$ with $x\in P_{\alpha }{w}_{\rho }Q$, so $Z_{\rho }\mathrm{/}V$ is the set $S_{\rho }=\{UxV:x\in P_{\alpha }{w}_{\rho }Q\}$ of [L4] and $\mathrm{\Theta }(UxV):=\phi (Ux)$ is a well-defined function $\mathrm{\Theta }:S_{\rho }\to X$. The right action $S_{\rho }\times M\to S_{\rho }$, $(UxV)\cdot m:=UxmV$, is well defined because $mV{m}^{-1}=V$ by [L4], and $\mathrm{\Theta }((UxV)\cdot m)=\phi (Uxm)=\pi (m{)}^{-1}\phi (Ux)=m^{-1}\mathrm{\Theta }(UxV)$; conversely, if $\mathrm{\Theta }:S_{\rho }\to X$ satisfies $\mathrm{\Theta }(s\cdot m)=m^{-1}\mathrm{\Theta }(s)$ for all $s\in S_{\rho }$, $m\in M$, then $\phi (Ux):=\mathrm{\Theta }(UxV)$ is well defined on $Z_{\rho }$ (if $Ux=U{x}^{\mathrm{\prime }}$ then $UxV=U{x}^{\mathrm{\prime }}V$) and satisfies $\phi (Uxq)=\mathrm{\Theta }(UxmV)=\mathrm{\Theta }((UxV)\cdot m)=m^{-1}\mathrm{\Theta }(UxV)=\pi (q{)}^{-1}\phi (Ux)$ for $q=mv\in Q$ written with $m\in M$, $v\in V$ as the unique factorisation of [L1], so $\phi \in {\mathcal{F}}_{\rho }$. The two assignments are inverse to each other, and they are $L$-equivariant for the left actions $(l\cdot \mathrm{\Theta })(s):=\mathrm{\Theta }(l^{-1}\cdot s)$ on $S_{\rho }$ and $(l\cdot \phi )(z)=\phi (l^{-1}\cdot z)$ on $Z_{\rho }$, because $\phi (U{l}^{-1}x)=\mathrm{\Theta }(U{l}^{-1}xV)$ for all $l\in L$ and $x$.

Transport along the balanced product. Fix $\rho \in \mathcal{R}$ and write $A:=A_{\rho }$, $C:=C_{\rho }$, $D:=D_{\rho }$. By [L4] the map $\mathrm{\Phi }([lA,Dm]):=Ulm{w}_{\rho }V$ is a bijection from the balanced product $W_{\rho }:=(L\mathrm{/}A){\times }_C(D\mathrm{\backslash }{M}_{\rho })$ onto $S_{\rho }$, the left action $l^{\mathrm{\prime }}\cdot [lA,Dm]=[l^{\mathrm{\prime }}lA,Dm]$ corresponds to $l^{\mathrm{\prime }}\cdot (UxV)=U{l}^{\mathrm{\prime }}xV$, and the right action of $b\in L_{\beta }=M$ given by $[lA,Dm]\cdot b=[lA,Dm{w}_{\rho }b{w}_{\rho }^{-1}]$ corresponds to $(UxV)\cdot b=UxbV$; since $b\mapsto w_{\rho }b{w}_{\rho }^{-1}$ is a bijection from $M$ onto $M_{\rho }$ with inverse $\mu \mapsto w_{\rho }^{-1}\mu w_{\rho }$, this right action is the right translation $[lA,Dm]\cdot \mu =[lA,Dm\mu ]$ by the elements of $M_{\rho }$, so that $[lA,Dm]=[lA,D{I}_n]\cdot m$ for every $m\in M_{\rho }$. Transporting a function $\mathrm{\Theta }:S_{\rho }\to X$ of step 3.1 along $\mathrm{\Phi }$ gives the function $\chi :=\mathrm{\Theta }\circ \mathrm{\Phi }$ on $W_{\rho }$, and $\mathrm{\Theta }\mapsto \chi$ is a bijection between the functions with the covariance of step 3.1 and the functions $\chi :W_{\rho }\to X$ with $\chi (y\cdot b)=b^{-1}\chi (y)$ for all $y\in W_{\rho }$, $b\in M$, under which the left $L$-actions become $(l^{\mathrm{\prime }}\cdot \chi )(y)=\chi (l^{\mathrm{\prime }-1}\cdot y)$.

Extraction of the coefficient function. Fix $\rho \in \mathcal{R}$, keep $A,C,D$ of step 4.1, and let $\chi :W_{\rho }\to X$ satisfy $\chi (y\cdot b)=b^{-1}\chi (y)$ for all $y\in W_{\rho }$, $b\in M$; define $F:L\mathrm{/}A\to X$ by $F(lA):=\chi ([lA,D{I}_n])$, which is well defined because the class $[lA,D{I}_n]$ depends only on the coset $lA$. Then $F$ determines $\chi$, because $[lA,Dm]=[lA,D{I}_n]\cdot m$ by step 4.1 and hence $\chi ([lA,Dm])=m^{-1}F(lA)$ for every $m\in M_{\rho }$, and $\chi$ determines $F$ by definition, so the correspondence is a bijection onto its image; furthermore $F(lA)\in ({}^{\rho }X{)}^{D_{\rho }}$ for every $l$, because for $d\in D_{\rho }$ the element $m_0:=w_{\rho }^{-1}d{w}_{\rho }$ lies in $M$ and $[lA,D{I}_n]\cdot m_0=[lA,D{w}_{\rho }{m}_0{w}_{\rho }^{-1}]=[lA,Dd]=[lA,D{I}_n]$, so the covariance gives $F(lA)=\chi ([lA,D{I}_n])=\chi ([lA,D{I}_n]\cdot m_0)=m_0^{-1}F(lA)=(w_{\rho }^{-1}d{w}_{\rho }{)}^{-1}F(lA)$, that is, $d$ fixes $F(lA)$ under the $M_{\rho }$-action of ${}^{\rho }X$; and $F(lcA)=c^{-1}F(lA)$ for every $c\in C_{\rho }$, because the generating relation of [L4] gives $[lcA,D{I}_n]=[lA,Dc]$ and $[lA,Dc]=[lA,D{I}_n]\cdot (w_{\rho }^{-1}c{w}_{\rho })$, so $F(lcA)=\chi ([lA,Dc])=(w_{\rho }^{-1}c{w}_{\rho }{)}^{-1}F(lA)=c^{-1}F(lA)$ for the action of ${}^{\rho }X$. Conversely, if $F:L\mathrm{/}{A}_{\rho }\to ({}^{\rho }X{)}^{D_{\rho }}$ satisfies $F(lc{A}_{\rho })=c^{-1}F(l{A}_{\rho })$ for all $c\in C_{\rho }$, then ${\chi }_F([l{A}_{\rho },D_{\rho }m]):=m^{-1}F(l{A}_{\rho })$ is well defined: replacing $m$ by $dm$ with $d\in D_{\rho }$ replaces $m^{-1}$ by $m^{-1}(w_{\rho }^{-1}d{w}_{\rho }{)}^{-1}$, which acts trivially on the values of $F$ because $F$ takes values in $({}^{\rho }X{)}^{D_{\rho }}$, and replacing the pair $(l{A}_{\rho },D_{\rho }m)$ by $(lc{A}_{\rho },D_{\rho }{c}^{-1}m)$ in the same class replaces $m^{-1}F(l{A}_{\rho })$ by $(c^{-1}m{)}^{-1}F(lc{A}_{\rho })=m^{-1}c{c}^{-1}F(l{A}_{\rho })=m^{-1}F(l{A}_{\rho })$; the so-defined ${\chi }_F$ satisfies the covariance of step 4.1 and is $L$-equivariant, because $(l^{\mathrm{\prime }}\cdot {\chi }_F)(y)={\chi }_F(l^{\mathrm{\prime }-1}y)$ and $(l^{\mathrm{\prime }}\cdot F)(lA)=F(l^{\mathrm{\prime }-1}lA)$; hence $\chi \mapsto F$ is a bijection from the covariant functions of step 4.1 onto the functions $F:L\mathrm{/}{A}_{\rho }\to ({}^{\rho }X{)}^{D_{\rho }}$ with $F(lc{A}_{\rho })=c^{-1}F(l{A}_{\rho })$.