Binary logistic log-likelihood and L1-penalized objective

Fix a finite design with rows $x_i^T\in {\mathbb{R}}^q$, $1\le i\le n$, where $n\ge 1$ and $q\ge 1$. Use either an intercept design, in which the first coordinate of every row is $1$ and $\beta =({\beta }_0,\theta )\in \mathbb{R}\times {\mathbb{R}}^{q-1}$, or a no-intercept design, in which no coordinate is reserved as an intercept, the intercept coefficient is fixed at zero, and $\beta \in {\mathbb{R}}^q$. Let $y_i\in \{0,1\}$ and assume the responses are conditionally independent given the design, with binary logistic probabilities $p_i=\sigma (x_i^T\beta )$ for finite coefficients. In either design convention, the log-likelihood is

$$\mathrm{\ell }(\beta )=\sum _{i=1}^n[y_i{x}_i^T\beta -\log \text{\hspace{0.17em}⁣}(1+e^{x_i^T\beta })]\mathrm{.}$$

For $\lambda >0$, the L1 objective to minimize in the intercept design is $-\mathrm{\ell }(\beta )+\lambda \mathrm{\parallel }\theta {\mathrm{\parallel }}_1$, so the intercept is unpenalized; in the no-intercept design it is $-\mathrm{\ell }(\beta )+\lambda \mathrm{\parallel }\beta {\mathrm{\parallel }}_1$, so every coefficient is penalized. Here $\mathrm{\parallel }\theta {\mathrm{\parallel }}_1={\sum }_{j=1}^{q-1}\mathrm{\mid }{\theta }_j\mathrm{\mid }$ and $\mathrm{\parallel }\beta {\mathrm{\parallel }}_1={\sum }_{j=1}^q\mathrm{\mid }{\beta }_j\mathrm{\mid }$, with an empty sum equal to zero. These are objective definitions and assert no optimizer existence.

Binary logistic and multiclass softmax conditional models

Hastie, Tibshirani, and Friedman, ESL, §§4.4.1 and 4.4.4, equations (4.20) and (4.31)

Stanford CS229 Spring 2022 lecture notes, §2.1