Low-rank adaptation and the geometry of parameter updates

Let $d,k$ be positive integers, let $1\le r\le \min (d,k)$, let $c\in \mathbb{R}\setminus \{0\}$, and fix a matrix $W_0\in {\mathbb{R}}^{d\times k}$. A rank-$r$ adapter for the linear layer $x\mapsto W_{0}x$ consists of trainable factor matrices $B\in {\mathbb{R}}^{d\times r}$ and $A\in {\mathbb{R}}^{r\times k}$ and the updated weight matrix

$W=W_0+cBA\mathrm{.}$

Its output on $x\in {\mathbb{R}}^k$ is $Wx=W_{0}x+cB(Ax)\in {\mathbb{R}}^d$. The base matrix $W_0$ is frozen, meaning that an adaptation procedure does not update its entries; only entries of $A$ and $B$ are trainable in this definition. The update matrix is $W-W_0=cBA$, whose rank is the dimension of its image as a linear map from ${\mathbb{R}}^k$ to ${\mathbb{R}}^d$. No bias, dropout, random initialization, optimization rule, or convergence guarantee is part of this definition.

Finite feedforward computation graphs and layered MLPs: its finite layered perceptron specializes to a linear matrix map when the layer has identity output activation and zero bias.

Hu et al., §4.1: frozen base matrix and trainable low-rank factors

MIT OCW, §§8.3–8.4: finite differentiable computation graphs and reverse-mode gradients

Fix positive integers $d,k,n$, an integer $1\le r\le \min (d,k)$, a frozen matrix $W_0\in {\mathbb{R}}^{d\times k}$, a nonzero scale $c\in \mathbb{R}$, a finite ordered data list $(x_i,y_i{)}_{i=1}^n$ with $x_i\in {\mathbb{R}}^k$, and, for each data entry, a loss program ${\mathrm{\ell }}_i$ that returns a real scalar wherever its primitive calls are defined. Let $T\ge 0$ be a finite integer, let $I_0,\dots ,I_{T-1}$ be fixed finite nonempty ordered lists of indices in $\{1,\dots ,n\}$ (repeated indices are allowed), let ${\eta }_0,\dots ,{\eta }_{T-1}\ge 0$ be finite step sizes, and supply an initial matrix $A_0\in {\mathbb{R}}^{r\times k}$. The procedure has no dropout or random initialization; any random initialization must be sampled externally and supplied as $A_0$. Set $B_0=0\in {\mathbb{R}}^{d\times r}$.

Represent each loss evaluation by a specified finite real-arithmetic program with a scalar output. Each primitive is continuously differentiable on a declared open domain or is piecewise continuously differentiable, with finitely many open branch regions, a continuously differentiable formula on each region, and a separately specified primal boundary value when one is defined. Here $C^1$ means totally differentiable with continuous derivative. Check admissibility at the current factors on each step; a strict branch used at one step need not be the branch used at another.

$L_t(A,B)=\frac{1}{\mathrm{\mid }{I}_t\mathrm{\mid }}{\sum }_{i\in I_t}{\mathrm{\ell }}_i\text{\hspace{0.17em}⁣}((W_0+cBA)x_i)$

Starting with $t=0$, perform the following finite loop while $t<T$:

1. Evaluate the displayed program at $(A_t,B_t)$, visiting its primitive calls in evaluation order. At the first call with no primal value, return DOMAIN_ERROR with the current factors and step index $t$. At the first defined piecewise primitive whose input lies on a branch boundary or has ambiguous branch selection, return NONDIFFERENTIABLE with the current factors and step index $t$. If every primitive call has a value and every piecewise primitive is on a unique strict branch, run reverse-mode automatic differentiation on this same scalar-output program at $(A_t,B_t)$, with the selected branch choices, with respect to the ordered list of all entries of $A_t$ followed by all entries of $B_t$, each matrix flattened row by row. If reverse mode returns either failure status, return it with the current factors and step index $t$. No factor update is made on a failure status.
2. If reverse mode returns gradients, reshape the first and second blocks into $G_t^A\in {\mathbb{R}}^{r\times k}$ and $G_t^B\in {\mathbb{R}}^{d\times r}$, respectively, and set

$A_{t+1}=A_t-{\eta }_t{G}_t^A,B_{t+1}=B_t-{\eta }_t{G}_t^B\mathrm{.}$

Increment $t$ by one.

After $T$ successful updates, return SUCCESS, the matrices $A_T,B_T$, and the merged weight $W_0+c{B}_T{A}_T$. At $T=0$ return the supplied $A_0$, zero $B_0$, and $W_0$ without evaluating a loss. This is a finite real-arithmetic procedure; it asserts no convergence, minimization, generalization, floating-point accuracy, speed, or memory guarantee.

Low-rank adaptation of a linear layer: the factor shapes, frozen base matrix, and updated matrix used here.

Finite straight-line computational programs and real-arithmetic tapes: finite ordered real-arithmetic graph, C1 primitive, admissible domain, and strict-branch conventions.

Reverse-mode automatic differentiation for a scalar graph output: scalar-output reverse sweep and its domain-error and branch-boundary statuses.

Hu et al., §4.1: frozen base weights, trainable factors, and merged weights

MIT OCW, §§8.3–8.4: reverse-mode accumulation for finite scalar objectives

Fix positive integers $d,k$, an integer $1\le r\le \min (d,k)$, a nonzero real number $c$, and a base matrix $W_0\in {\mathbb{R}}^{d\times k}$. For every pair $B\in {\mathbb{R}}^{d\times r}$ and $A\in {\mathbb{R}}^{r\times k}$, the adapter update $\mathrm{\Delta }=cBA$ has matrix rank at most $r$, where rank is the dimension of the image of the associated linear map. Conversely, every target update $\mathrm{\Delta }\in {\mathbb{R}}^{d\times k}$ of rank at most $r$ has at least one representation $\mathrm{\Delta }=cBA$ with these factor shapes. For any such factors and any invertible matrix $M\in {\mathbb{R}}^{r\times r}$, the transformed factors $BM$ and $M^{-1}A$ represent the same update. Thus the factorization is not identifiable: every target update of rank at most $r$ has more than one factor pair. A target update of rank greater than $r$ cannot be represented.

Axler, Linear Algebra Done Right, §7E, Theorem 7.70: singular value decomposition for finite-dimensional inner-product spaces

Axler, Linear Algebra Done Right, §7E, Theorem 7.80: matrix factorization from the singular value decomposition

Fix dimensions and matrices as in the definition of a low-rank adapter. Let $m\ge 1$, let $x_1,\dots ,x_m\in {\mathbb{R}}^k$, and let each ${\mathrm{\ell }}_j$ be a finite real-arithmetic loss program with scalar real output wherever its primitive calls are defined. Each primitive map is $C^1$ on a declared open domain or is piecewise $C^1$ with finitely many open branch regions and a $C^1$ formula on each region; when defined, a boundary primal value is specified separately. Here $C^1$ means totally differentiable with continuous derivative. Define the finite empirical loss on a weight matrix $U\in {\mathbb{R}}^{d\times k}$ by the following formula wherever all the programs return scalar values:

$\mathcal{L}(U)=\frac{1}{m}{\sum }_{j=1}^m{\mathrm{\ell }}_j(U{x}_j)\mathrm{.}$

At any $W=W_0+cBA$ where all primitive calls have values and every piecewise primitive call is on a unique strict branch, write $G={\mathrm{\nabla }}_W\mathcal{L}(W)$ for the Frobenius gradient, defined by $D\mathcal{L}(W)[H]=⟨G,H{⟩}_F$ for every $H\in {\mathbb{R}}^{d\times k}$, where $⟨X,Y{⟩}_F=\mathrm{tr}(X^{\mathsf{T}}Y)$. Then the factor gradients are

${\mathrm{\nabla }}_B\mathcal{L}(W_0+cBA)=cG{A}^{\mathsf{T}},{\mathrm{\nabla }}_A\mathcal{L}(W_0+cBA)=c{B}^{\mathsf{T}}G\mathrm{.}$

At every successful step $t$ of the finite-budget adaptation algorithm, write its ordered minibatch as $I_t=(i_{t,1},\dots ,i_{t,m_t})$, retaining repeated indices, and define

${\mathcal{L}}_t(U)=\frac{1}{m_t}{\sum }_{s=1}^{m_t}{\mathrm{\ell }}_{i_{t,s}}(U{x}_{i_{t,s}})\mathrm{.}$

With $W_t=W_0+c{B}_t{A}_t$ and $G_t={\mathrm{\nabla }}_W{\mathcal{L}}_t(W_t)$, all primitive calls in this batch objective have values and each piecewise primitive call is on a unique strict branch; branch choices may differ between successful steps. Reverse-mode automatic differentiation returns the exact real-arithmetic gradients ${\mathrm{\nabla }}_{B_t}{\mathcal{L}}_t(W_t)=c{G}_t{A}_t^{\mathsf{T}}$ and ${\mathrm{\nabla }}_{A_t}{\mathcal{L}}_t(W_t)=c{B}_t^{\mathsf{T}}{G}_t$. If $B=0$, then ${\mathrm{\nabla }}_A\mathcal{L}=0$ for every such empirical loss; if also $A=0$, then ${\mathrm{\nabla }}_B\mathcal{L}=0$. For every input $x\in {\mathbb{R}}^k$, the unmerged and merged layer outputs agree:

$W_{0}x+cB(Ax)=(W_0+cBA)x\mathrm{.}$

The factor matrices contain $rk+dr=r(d+k)$ trainable scalar coordinates, excluding the frozen entries of $W_0$; this coordinate count is not an assertion about identifiable degrees of freedom. All equalities are in real arithmetic and imply no floating-point, convergence, task-transfer, or hardware-performance guarantee.

Let $T\ge 0$ be the finite horizon of the procedure. The finite-budget procedure also has the following termination and status property. Suppose each primal value/status check terminates, and each defined primitive evaluation and each local derivative evaluation on a strict branch returns its specified real-arithmetic value or derivative in finite time. Then at the first step whose ordered primal evaluation encounters a primitive with no value or a defined piecewise primitive on a boundary or ambiguous branch, the procedure returns DOMAIN_ERROR or NONDIFFERENTIABLE, respectively, before updating the factors at that step. If no such evaluation occurs, it returns SUCCESS after exactly $T$ updates. For $T=0$, it returns immediately without evaluating a loss.

Hu et al., §4.1: factorized update, frozen base weights, and exact merge identity

MIT OCW, §§8.3–8.4: chain-rule accumulation and reverse-mode gradients of scalar objectives

Low-rank adaptation constrains the update of one linear layer to rank at most $r$ and uses $r(d+k)$ trainable scalar coordinates for the two factors, excluding the frozen base matrix and any optional biases. These algebraic facts do not imply that a downstream task has an optimal or effective update of rank at most $r$, that a finite-budget training procedure reaches an optimum, that transfer performance improves, or that low rank prevents catastrophic forgetting. Empirical rank, quality, throughput, memory, and latency findings for a particular model, data set, and hardware configuration remain empirical observations under that setup; they are not consequences of the factorization or gradient formulas. Merging the factors has an exact real-arithmetic output identity, but this does not assert bitwise equality, floating-point accuracy, adapter-switching latency, storage savings, or hardware speed.

Rank capacity and factorization ambiguity of a low-rank adapter: rank capacity and factorization symmetry proved for finite real matrices.

Exact merging and gradients for a low-rank adapter: exact real-arithmetic merge and factor-gradient formulas.

Hu et al., §§4–5: LoRA parameterization and source-specific empirical context