Low-rank adaptation of a linear layer

Let $d,k$ be positive integers, let $1\le r\le \min (d,k)$, let $c\in \mathbb{R}\setminus \{0\}$, and fix a matrix $W_0\in {\mathbb{R}}^{d\times k}$. A rank-$r$ adapter for the linear layer $x\mapsto W_{0}x$ consists of trainable factor matrices $B\in {\mathbb{R}}^{d\times r}$ and $A\in {\mathbb{R}}^{r\times k}$ and the updated weight matrix

$W=W_0+cBA\mathrm{.}$

Its output on $x\in {\mathbb{R}}^k$ is $Wx=W_{0}x+cB(Ax)\in {\mathbb{R}}^d$. The base matrix $W_0$ is frozen, meaning that an adaptation procedure does not update its entries; only entries of $A$ and $B$ are trainable in this definition. The update matrix is $W-W_0=cBA$, whose rank is the dimension of its image as a linear map from ${\mathbb{R}}^k$ to ${\mathbb{R}}^d$. No bias, dropout, random initialization, optimization rule, or convergence guarantee is part of this definition.

Finite feedforward computation graphs and layered MLPs: its finite layered perceptron specializes to a linear matrix map when the layer has identity output activation and zero bias.

Hu et al., §4.1: frozen base matrix and trainable low-rank factors

MIT OCW, §§8.3–8.4: finite differentiable computation graphs and reverse-mode gradients