Prediction problems, risks, and empirical minimization — Examples

Let $\mathsf{X}=\{u,v\}$, $\mathsf{Y}=\mathsf{A}=\{0,1\}$ with their discrete sigma-algebras, and let the joint law be $P(u,0)=\frac{1}{8},P(u,1)=\frac{3}{8},P(v,0)=\frac{3}{8},P(v,1)=\frac{1}{8}\mathrm{.}$ Use zero-one loss $\mathrm{\ell }(a,y)={\mathbf{1}}_{\{a\mathrm{\ne }y\}}$. The predictor $f_{10}$ with $f_{10}(u)=1$, $f_{10}(v)=0$ has population risk $1\mathrm{/}4$; the predictor $f_{01}$ with the opposite labels has risk $3\mathrm{/}4$. The two constant predictors each have risk $1\mathrm{/}2$. Thus $f_{10}$ is the unique population-risk minimizer among the four predictors.

Let $\mathsf{X}=\{u,v\}$ with its discrete sigma-algebra, let $\mathsf{Y}=\{0,2,4,6\}$ with its discrete sigma-algebra, let the action space be $\mathsf{A}=\mathbb{R}$ with its Borel sigma-algebra, and let $\mathcal{F}$ be all functions from $\mathsf{X}$ to $\mathbb{R}$. Define $X(x,y)=x$ and $Y(x,y)=y$ on $\mathrm{\Omega }=\mathsf{X}\times \mathsf{Y}$. Give the joint law by $P(u,0)=\frac{1}{8},P(u,2)=\frac{1}{4},P(u,4)=\frac{1}{8},P(v,2)=\frac{1}{4},P(v,6)=\frac{1}{4},$ with all other atoms having probability zero, and set $P_X(x):={\sum }_{y}P(x,y)$. Use squared loss $\mathrm{\ell }(a,y)=(a-y{)}^2$. For $x\in \{u,v\}$ define the conditional mean $m(x):={\sum }_{y}yP(x,y)\mathrm{/}{P}_X(x)$. Then $m(u)=2$ and $m(v)=4$, and the predictor $f^{\ast }(x)=m(x)$ uniquely minimizes population squared risk over $\mathcal{F}$, with risk $3$.

Let $\mathsf{X}=\{u\}$, $\mathsf{Y}=\{0,4\}$ with discrete sigma-algebras, $\mathsf{A}=\mathbb{R}$ with its Borel sigma-algebra, and $\mathcal{F}$ the set of all predictors from $\mathsf{X}$ to $\mathbb{R}$. Give the joint law by $P(u,0)=P(u,4)=1\mathrm{/}2$ and define the label coordinate $Y(u,y)=y$. Use absolute loss $\mathrm{\ell }(a,y)=\mathrm{\mid }a-y\mathrm{\mid }$. A median of a probability law on $\mathbb{R}$ is a value $m$ satisfying both $P(Y\le m)\ge 1\mathrm{/}2$ and $P(Y\ge m)\ge 1\mathrm{/}2$. The conditional law of $Y$ at the only instance assigns probability $1\mathrm{/}2$ to each of $0$ and $4$, so every $m\in [0,4]$ is a median. The predictors $f_m(u)=m$ for $m\in [0,4]$ all minimize population risk, with risk $2$; every predictor with value outside $[0,4]$ has risk greater than $2$.

Let $\mathsf{X}=\{x_0\}$ and $\mathsf{Y}=\{0\}$ with discrete sigma-algebras, let $\mathsf{A}=\mathbb{R}$ with its Borel sigma-algebra, and let $P$ put mass one on $(x_0,0)$. For each positive integer $n$, let $f_n(x_0)=1\mathrm{/}n$ and set $\mathcal{F}=\{f_n:n\ge 1\}$. Use squared loss $\mathrm{\ell }(a,y)=(a-y{)}^2$. For every sample size $m\ge 1$, the only sample has ${\widehat{R}}_S(f_n)=R_P(f_n)=1\mathrm{/}{n}^2$. The empirical and population infima over $\mathcal{F}$ are both $0$ and are not attained. For tolerance $\eta =1\mathrm{/}4$, the constant selector $A_{1\mathrm{/}4}(S,x)=1\mathrm{/}2$ supplies the approximate-ERM procedure with output $f_2$; for tolerance $0$, no exact ERM exists.