Low-rank adaptation and the geometry of parameter updates — Examples

Direct multiplication gives $BA=\left(\begin{array}{cc}3& 4\\ 6& 8\end{array}\right)$.

Since $M=3$ is invertible, $B^{\mathrm{\prime }}=(3,6{)}^{\mathsf{T}}$ and $A^{\mathrm{\prime }}=(1,4\mathrm{/}3)$, and direct multiplication gives $B^{\mathrm{\prime }}{A}^{\mathrm{\prime }}=\left(\begin{array}{cc}3& 4\\ 6& 8\end{array}\right)=BA$, as also guaranteed by [F1].

The identity matrix $I_2$ maps ${\mathbb{R}}^2$ onto ${\mathbb{R}}^2$, so $\mathrm{rank}(I_2)=2$.

For every $B\in {\mathbb{R}}^{2\times 1}$ and $A\in {\mathbb{R}}^{1\times 2}$, $\mathrm{im}(BA)\subseteq \mathrm{im}(B)$ and $\dim \mathrm{im}(B)\le 1$; hence $\mathrm{rank}(BA)\le 1$ by [F1]. This contradicts step 1.1 if $BA=I_2$, so no such factors represent the target.

At either initialization, $W_0+c{B}_0{A}_0=0$, so the scalar loss derivative with respect to the weight is $G={\mathrm{\ell }}_1^{\mathrm{\prime }}(0)=0-1=-1$. The loss is a polynomial defined and differentiable on all of $\mathbb{R}$, so the one-step reverse-mode call is admissible.

If $A_0=B_0=0$, [F1] gives ${\mathrm{\nabla }}_B\mathcal{L}=(-1)0=0$ and ${\mathrm{\nabla }}_A\mathcal{L}=0(-1)=0$. By [F2], subtracting these zero gradients leaves $A_1=B_1=0$.

If $A_0=1$ and $B_0=0$, [F1] gives ${\mathrm{\nabla }}_A\mathcal{L}=0$ and ${\mathrm{\nabla }}_B\mathcal{L}=(-1)(1)=-1$. The update in [F2] therefore gives $A_1=1-{\eta }_0\cdot 0=1$ and $B_1=0-{\eta }_0(-1)={\eta }_0$.

Multiplication gives $BA=\left(\begin{array}{cc}3& 6\\ 4& 8\end{array}\right)$, so $W_0+2BA=\left(\begin{array}{cc}7& 12\\ 8& 17\end{array}\right)$ and $(W_0+2BA)x=(19,25{)}^{\mathsf{T}}$.

In the unmerged calculation, $W_{0}x=(1,1{)}^{\mathsf{T}}$ and $Ax=3$, so $W_{0}x+2B(Ax)=(1,1{)}^{\mathsf{T}}+2(3,4{)}^{\mathsf{T}}3=(19,25{)}^{\mathsf{T}}$, in agreement with [F1].