Fixed-design least-squares, ridge, lasso, group, and fused objectives

Let $n,p\ge 1$, let $X\in {\mathbb{R}}^{n\times p}$ be a fixed design matrix, let $y\in {\mathbb{R}}^n$, and let $\beta \in {\mathbb{R}}^p$. The empirical squared loss is

${\mathrm{\ell }}_n(\beta )=\frac{1}{2n}\mathrm{\parallel }y-X\beta {\mathrm{\parallel }}_2^2\mathrm{.}$

Ordinary least squares minimizes ${\mathrm{\ell }}_n(\beta )$ over ${\mathbb{R}}^p$. For $\lambda >0$, the ridge (Tikhonov) objective is

$J_{\lambda }^{\mathrm{r}\mathrm{i}\mathrm{d}\mathrm{g}\mathrm{e}}(\beta )=\mathrm{\parallel }y-X\beta {\mathrm{\parallel }}_2^2+\lambda \mathrm{\parallel }\beta {\mathrm{\parallel }}_2^2\mathrm{.}$

For $\lambda >0$, the lasso objective in the empirical-loss convention is

$J_{\lambda }^{\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{o}}(\beta )={\mathrm{\ell }}_n(\beta )+\lambda \mathrm{\parallel }\beta {\mathrm{\parallel }}_1,\mathrm{\parallel }\beta {\mathrm{\parallel }}_1={\sum }_{j=1}^p\mathrm{\mid }{\beta }_j\mathrm{\mid }\mathrm{.}$

This lasso has the same minimizers as $n^{-1}\mathrm{\parallel }y-X\beta {\mathrm{\parallel }}_2^2+2\lambda \mathrm{\parallel }\beta {\mathrm{\parallel }}_1$, the fixed-design convention with penalty parameter $\tau =\lambda$.

For a partition $G_1,\dots ,G_m$ of $\{1,\dots ,p\}$, fixed weights $w_g>0$, and $\lambda \ge 0$, the group-lasso objective is

$J_{\lambda }^{\mathrm{g}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{p}}(\beta )={\mathrm{\ell }}_n(\beta )+\lambda {\sum }_{g=1}^m{w}_g\mathrm{\parallel }{\beta }_{G_g}{\mathrm{\parallel }}_2\mathrm{.}$

For a fixed matrix $D\in {\mathbb{R}}^{q\times p}$ and ${\lambda }_1,{\lambda }_2\ge 0$, a fused-lasso objective is

$J_{{\lambda }_1,{\lambda }_2}^{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}(\beta )={\mathrm{\ell }}_n(\beta )+{\lambda }_1\mathrm{\parallel }\beta {\mathrm{\parallel }}_1+{\lambda }_2\mathrm{\parallel }D\beta {\mathrm{\parallel }}_1\mathrm{.}$

These group and fused formulas define objectives only; no existence, uniqueness, or algorithm guarantee is asserted for them here. The formulas use the displayed feature scales, so rescaling a column changes the penalty unless its coefficient and penalty weight are adjusted. If a separate intercept is included, write $X=[\mathbf{1}{X}_0]$ and $\beta =({\beta }_0,\gamma )$, and define the loss by $\mathrm{\parallel }y-{\beta }_0\mathbf{1}-X_0\gamma {\mathrm{\parallel }}_2^2\mathrm{/}(2n)$. In this convention replace the ridge penalty by $\lambda \mathrm{\parallel }\gamma {\mathrm{\parallel }}_2^2$, the lasso penalty by $\lambda \mathrm{\parallel }\gamma {\mathrm{\parallel }}_1$, the group penalty by $\lambda {\sum }_g{w}_g\mathrm{\parallel }{\gamma }_{G_g}{\mathrm{\parallel }}_2$ with the groups partitioning only the coordinates of $\gamma$, and the fused penalty by ${\lambda }_1\mathrm{\parallel }\gamma {\mathrm{\parallel }}_1+{\lambda }_2\mathrm{\parallel }{D}_0\gamma {\mathrm{\parallel }}_1$ with $D=[0D_0]$. The ridge loss remains unaveraged. Each penalty then acts only on $\gamma$ and the intercept coordinate ${\beta }_0$ is unpenalized.

The published fixed-design Lasso objective convention

The Elements of Statistical Learning, Chapter 3 and §18.4.2

High-Dimensional Statistics lecture notes, Definition 2.12