Position-free unmasked self-attention is permutation equivariant

For one head, write $Q=X{W}^Q$, $K=X{W}^K$, and $V=X{W}^V$. On input $PX$, the projections are $PQ$, $PK$, and $PV$. The unmasked score matrix therefore becomes $(PQ)(PK{)}^{\mathsf{T}}=P(Q{K}^{\mathsf{T}})P^{\mathsf{T}}$.

Row softmax commutes with simultaneous row and column permutation: every row's logits are merely reordered, so its normalized entries are reordered by the same permutation. Thus ${\mathrm{softmax}}_{\mathrm{r}\mathrm{o}\mathrm{w}}(PS{P}^{\mathsf{T}})=P{\mathrm{softmax}}_{\mathrm{r}\mathrm{o}\mathrm{w}}(S)P^{\mathsf{T}}$; multiplying by $PV$ gives $P{\mathrm{softmax}}_{\mathrm{r}\mathrm{o}\mathrm{w}}(S)V$. Concatenation and the shared output projection preserve this identity, and so does the residual addition.

A shared positionwise map applies the same function to each row, hence maps $PX$ to the permutation of its outputs. Rowwise layer normalization has the same property because each row's mean and variance are unchanged by reordering rows. Combining these identities with the attention and residual identity in step 2.1 shows that one encoder layer is equivariant; induction over the finite stack gives $T(PX)=PT(X)$.

For a masked head, transforming the mask to $PM{P}^{\mathsf{T}}$ gives the same score-and-weight calculation as steps 1.1–2.1, proving the stated mask identity. A permutation preserving the causal mask must preserve the relation $j\le i$ on the finite chain $1<\cdots <n$; its unique least element must map to itself, and then induction on the remaining ordered positions makes every position fixed. Therefore only the identity preserves the standard causal mask.