DefinitionDefinition: Literature-sourcedProof: Not applicablePipeline-generatedjudge pass (gpt-6-luna)

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

Multi-head attention and self-attention

Definition

Let $h, d_{q}, d_{k}, d_{v}, d_{h}, d_{v h}, d_{o}, n_{q}, n_{k}$ be positive integers. For query, key, and value matrices $Q \in R^{n_{q} \times d_{q}}$ , $K \in R^{n_{k} \times d_{k}}$ , and $V \in R^{n_{k} \times d_{v}}$ , choose matrices $W_{i}^{Q} \in R^{d_{q} \times d_{h}}$ , $W_{i}^{K} \in R^{d_{k} \times d_{h}}$ , and $W_{i}^{V} \in R^{d_{v} \times d_{v h}}$ for each head $i = 1, \dots, h$ , and choose $W^{O} \in R^{h d_{v h} \times d_{o}}$ . The $i$ -th head is

H_{i} = Attn (Q W_{i}^{Q}, K W_{i}^{K}, V W_{i}^{V}) \in R^{n_{q} \times d_{v h}},

where $Attn$ is scaled dot-product attention, with any chosen mask applied to its score matrix. Multi-head attention is

MHA (Q, K, V) = {Concat}_{f e a t u r e} (H_{1}, \dots, H_{h}) W^{O} \in R^{n_{q} \times d_{o}} .

Self-attention is the specialization $Q = K = V = X$ for a single token matrix $X \in R^{n \times d}$ ; each row is one token representation. Unmasked self-attention permits every query row to use every key row. Masked self-attention applies one specified mask to every head.

Definition references

Vaswani et al., §3.2.2: learned head projections, parallel attention heads, concatenation, and output projection

Dive into Deep Learning, §11.5.1: multi-head query/key/value projections and output shape

Scaled dot-product attention: row-softmax weighted-value formula and nonempty-mask-row condition.

Depends on

Scaled dot-product attention

Used by

Dependency tree · one level

1 result within one dependency step of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Vaswani et al., Attention Is All You Need, §3.2.2 (standard reference, not scraped)
Dive into Deep Learning, §11.5.1 Multi-Head Attention (standard reference, not scraped)