Risk estimation versus model identification

Let $(\mathcal{Z},\mathcal{F})$ be an observation space, let $M$ be a nonempty finite set of candidate labels, and for each $m\in M$ let $P_m$ be a probability law on $(\mathcal{Z},\mathcal{F})$. For each $m\in M$ and training size $k\ge 1$, let $A_{m,k}:{\mathcal{Z}}^k\to (\mathcal{H},\mathcal{G})$ be a measurable learning rule into a common measurable predictor space. For a data law $P$ and loss $L$ satisfying the measurability and integrability hypotheses of Cross-validation splits and training-size risk targets, write

$R_{m,k}(P):={\mathbb{E}}_{P^{\otimes (k+1)}}\text{\hspace{0.17em}⁣}\left[L\right(A_{m,k}(Z_1,\dots ,Z_k),Z_{k+1}\left)\right]$

for its risk at training size $k$; when $P$ is fixed, write $R_{m,k}$. A risk-minimizing selector chooses a label minimizing the specified risk criterion, such as $R_{m,k}$ or its cross-validation estimate; a fixed ordering of $M$ may be used to resolve ties. This objective compares expected predictive loss.

For each $n\ge 1$, a data-based selector is a measurable map ${\widehat{m}}_n:{\mathcal{Z}}^n\to M$, where $M$ has its discrete sigma-algebra. A true model label at sample size $n$ is a specified $m_0\in M$ when the observations are IID with law $P_{m_0}$. The selector sequence $({\widehat{m}}_n)$ is model-consistent at $m_0$ when $P_{m_0}^{\otimes n}({\widehat{m}}_n=m_0)\to 1$ as $n\to \mathrm{\infty }$. This identification objective is distinct from minimizing predictive risk: risk minimization alone does not assert that the family is identifiable, that one candidate is true, or that the minimum-risk label is the true label. The distinction between risk estimation/model selection for prediction and model identification follows Cross-validation splits and training-size risk targets's risk target and the source treatments cited below.