Positional encodings and causal masks

For a sequence of length $n$ with $d$-dimensional token representations, a positional encoding is a fixed or trainable matrix $P\in {\mathbb{R}}^{n\times d}$ added rowwise to the token matrix $X\in {\mathbb{R}}^{n\times d}$, giving $X+P$. One fixed choice, with positions $s\in {\mathbb{N}}_0$ and coordinates indexed from $0$, is $P_{s,2r}=\sin (s\mathrm{/}{10000}^{2r\mathrm{/}d})$ and $P_{s,2r+1}=\cos (s\mathrm{/}{10000}^{2r\mathrm{/}d})$ whenever the indicated coordinate is less than $d$.

For decoder self-attention on $n$ positions indexed by $0,\dots ,n-1$, the causal mask is $M\in (\mathbb{R}\cup \{-\mathrm{\infty }\}{)}^{n\times n}$ with $M_{ij}=0$ when $j\le i$ and $M_{ij}=-\mathrm{\infty }$ when $j>i$. Thus query position $i$ can use keys at positions $0,\dots ,i$ and cannot use a future position; every row has an allowed key because $M_{ii}=0$.

Vaswani et al., §3.5: positional encodings added to embeddings and sinusoidal formula

Vaswani et al., §3.2.3: masking illegal decoder connections before softmax

Dive into Deep Learning, §11.6.3: absolute and relative positional information

Dive into Deep Learning, §11.7.5: autoregressive decoder mask