Indicator-response least-squares classification

Let $n,q\ge 1$ and $K\ge 2$ be finite integers. For a fixed design matrix $X\in {\mathbb{R}}^{n\times q}$ and labels $y_i\in \{1,\dots ,K\}$, form the indicator response matrix $Y\in {\mathbb{R}}^{n\times K}$ by $Y_{ik}=1$ when $y_i=k$ and $Y_{ik}=0$ otherwise. For a coefficient matrix $B\in {\mathbb{R}}^{q\times K}$, set $L(B)=(2n{)}^{-1}\mathrm{\parallel }Y-XB{\mathrm{\parallel }}_F^2$, where $\mathrm{\parallel }A{\mathrm{\parallel }}_F^2$ is the sum of squares of all entries of $A$. Let $B_{\ast }$ be the least-squares minimizer with smallest Frobenius norm. For a new row vector $x^T\in {\mathbb{R}}^{1\times q}$, the indicator-response classifier assigns scores $(x^T{B}_{\ast }{)}_k$ and predicts the smallest class index attaining their maximum. An intercept, if used, is one of the columns of $X$; no rank assumption is imposed on $X$.

Finite linear-score classifiers with a deterministic tie rule

The minimum-norm indicator-response least-squares fit exists uniquely ↗

Hastie, Tibshirani, and Friedman, ESL, §4.2, equations (4.3)–(4.6)

Stanford CS229 Spring 2022 lecture notes, §§1.2 and 2.1