Unrolled regularization architecture

Let $n,m\ge 1$ be integers, $X={\mathbb{R}}^n$, and $Y={\mathbb{R}}^m$ with their standard Euclidean inner products and Borel sigma-algebras. Borel sets are members of the sigma-algebra generated by open sets, and a map is Borel if preimages of Borel sets are Borel. An instance is a pair $(A,y)$ in a declared Borel set $\mathcal{D}\subseteq {\mathbb{R}}^{m\times n}\times Y$, where $A$ is the measurement matrix, $y$ is the observed data, and $A^{\mathsf{T}}$ is the transpose matrix. For fixed $A$, the forward map is $F_A:X\to Y$, $F_A(x)=Ax$, and the reconstruction target is the signal $x$ itself. For $x\in X$, define the data-consistency correction by $c(A,y,x):=A^{\mathsf{T}}(y-Ax)$.

For an integer depth $N\ge 1$, an unrolled regularization architecture consists of a Borel initialization map $I_{\theta }:\mathcal{D}\to X$, parameters $\theta =({\theta }_0,\dots ,{\theta }_{N-1})$ and real step parameters ${\tau }_0,\dots ,{\tau }_{N-1}$ held fixed at deployment, and Borel update maps $B_{{\theta }_k}:D_k\to X$ with $D_k\subseteq \mathcal{D}\times X$ Borel. Set $x_0=I_{\theta }(A,y)$ and, for $k=0,\dots ,N-1$, set

$x_{k+1}=B_{{\theta }_k}((A,y),x_k+{\tau }_{k}c(A,y,x_k))\mathrm{.}$

For fixed parameters and depth, let ${\mathcal{D}}_{\theta ,N}$ be the Borel subset of $\mathcal{D}$ for which the initialization and all $N$ recursive evaluations are defined. For $(A,y)\in {\mathcal{D}}_{\theta ,N}$, write $U_{{\theta }_k}^{A,y}(x):=B_{{\theta }_k}((A,y),x+{\tau }_{k}c(A,y,x))$. The deployed map is

${\mathrm{\Phi }}_{\theta ,N}(A,y)=(U_{{\theta }_{N-1}}^{A,y}\circ \cdots \circ U_{{\theta }_0}^{A,y})(I_{\theta }(A,y))\mathrm{.}$

Admissibility requires this finite map to be Borel; it is a point reconstruction rule from the input pair $(A,y)$ to the signal target in $X$, not a probability-distribution-valued rule. The parameters may have been selected by any training procedure, and the architecture supplies no training-law specification or convergence, data-residual, or test-risk guarantee.

An exact proximal block is permitted when, for each fixed $(A,y)$, $B_{{\theta }_k}((A,y),v)={\mathrm{prox}}_{{\alpha }_{k}g}(v)$ for some ${\alpha }_k>0$, where $g=g_{{\theta }_k,(A,y)}:{\mathbb{R}}^n\to (-\mathrm{\infty },+\mathrm{\infty }]$ has nonempty effective domain $\{u:g(u)<+\mathrm{\infty }\}$, satisfies $g((1-\lambda )u+\lambda v)\le (1-\lambda )g(u)+\lambda g(v)$ for $u,v\in {\mathbb{R}}^n$ and $0\le \lambda \le 1$, and has closed epigraph $\{(u,s):g(u)\le s\}$. Here ${\mathrm{prox}}_{\alpha g}(v)$ denotes the unique global minimizer $u\in {\mathbb{R}}^n$ of $g(u)+\mathrm{\parallel }u-v{\mathrm{\parallel }}_2^2\mathrm{/}(2\alpha )$. The proximal-map existence theorem supplies uniqueness under precisely these hypotheses. A general learned block is not assumed to be a proximal map or to minimize any objective.

For differentiation, fix an admissible data pair $(A,y)$ and regard the desired trainable parameter coordinates, including any step parameters to be differentiated, as a finite-dimensional input vector $\eta$ in an open parameter set; hold the other values fixed. At the differentiation point ${\eta }_0$, require an open neighborhood $V$ of ${\eta }_0$ on which the actual initialization and all $N$ deployed updates are defined for this fixed $(A,y)$. Require the graph below to evaluate those same maps for every $\eta \in V$; a smooth extension beyond a declared Borel domain alone does not meet this requirement. Without such a neighborhood, no derivative of the deployed map is asserted. Construct a finite straight-line graph with input $\eta$, initialization node $x_0=I_{\theta }(A,y)$, and, for each $k=0,\dots ,N-1$, distinct nodes

$v_k=x_k+{\tau }_k{A}^{\mathsf{T}}(y-A{x}_k),x_{k+1}=B_{{\theta }_k}((A,y),v_k)\mathrm{.}$

Its output nodes are the coordinates of $x_N$. Thus each stage uses its own map $B_{{\theta }_k}$ and step parameter ${\tau }_k$; no common update map or shared layer parameter is required. Represent each initialization and update by its constituent primitive nodes or by a supplied complete deterministic $C^1$ primitive with its derivative rule; an opaque call without that map and rule is not an admissible graph primitive. Require every graph primitive to be continuously differentiable on an open neighborhood of its visited inputs. For a piecewise-$C^1$ primitive, each visited input must lie strictly inside a declared branch region, with that branch label fixed on an open neighborhood of the input; at a branch boundary or ambiguous branch selection the automatic-differentiation procedure returns no derivative. Apply forward-mode automatic differentiation to each scalar output-coordinate graph in the desired parameter direction. The forward-mode correctness theorem then identifies the returned tangent with the directional derivative of that finite output; coordinate directions give the Jacobian. When all stages instead repeat one $C^1$ map $G(z,\theta )$ with a common parameter, the finite fixed-point-iteration procedure is the corresponding specialization. This differentiates only the deployed finite map and adds no assertion about an infinite iteration.

For $N\ge 3$, the diagram compresses the intervening states and updates into a single arrow; when $N=3$, that arrow is the single update $U_{{\theta }_1}^{A,y}$. For $N=1$ and $N=2$, the recurrence above gives the corresponding one- and two-update chains. At one fixed admissible $(A,y)$, every arrow denotes finite function application, not a limit.

Monga, Li, and Eldar, full algorithm-unrolling treatment

Gregor and LeCun, fixed-depth LISTA construction

Clason, full lecture notes on Landweber regularization

Point estimators and finite unrolled learned-update interface.

Finite graph and primitive conventions.

Borel measurable spaces for the data and signal domains.

Borel maps are characterized by Borel preimages.

Finite product Borel structure on layer inputs.

Automatic differentiation of a finite iteration.

Forward-mode differentiation of a finite computational graph.

Correctness of forward mode on a fixed strict branch.

The proper closed convex proximal-map interface.

The proper closed convex epigraph convention used by the proximal branch.

Existence and uniqueness of the proximal minimizer under those hypotheses.

The finite recursive domain and output are Borel.