Cross-validation splits and training-size risk targets

Let $(\mathcal{Z},\mathcal{F})$ be a measurable sample space with probability law $P$. For each training size $k\ge 1$, a learning rule is a measurable map $A_k:{\mathcal{Z}}^k\to (\mathcal{H},\mathcal{G})$, where $(\mathcal{H},\mathcal{G})$ is a measurable predictor space. This is a statistic in the sense of Statistics and estimators. Let $L:\mathcal{H}\times \mathcal{Z}\to \mathbb{R}$ be jointly measurable and assume $L(A_k(Z_1,\dots ,Z_k),Z_{k+1})$ is integrable under $P^{\otimes (k+1)}$ for every size used. The convention that risk is expected loss agrees with Loss functions and risk for nonnegative losses; the integrable real-valued case is used here as well. The product law is the joint law of independent observations by Independent random elements have product joint law.

The risk at training size $k$ is

$R_k:={\mathbb{E}}_{P^{\otimes (k+1)}}\text{\hspace{0.17em}⁣}\left[L\right(A_k(Z_1,\dots ,Z_k),Z_{k+1}\left)\right],$

where $Z_{k+1}$ is independent of the training sample. A split of $[n]:=\{1,\dots ,n\}$ is a pair $(T,V)$ of disjoint nonempty sets with $T\cup V=[n]$; $T$ is the training index set and $V$ is the validation index set. For a fixed partition $I_1,\dots ,I_K$ of $[n]$ into nonempty folds with $K\ge 2$, the equal-observation-weight cross-validation score is

${\widehat{R}}_{CV}:=\frac{1}{n}{\sum }_{j=1}^K{\sum }_{i\in I_j}L(A_{n-\mathrm{\mid }{I}_j\mathrm{\mid }}(Z_{I_j^c}),Z_i),$

where each training tuple $Z_{I_j^c}$ is ordered by increasing index and $I_j^c=[n]\setminus I_j$. The risk corresponding to this design is the weighted training-size target ${\sum }_{j=1}^K(\mathrm{\mid }{I}_j\mathrm{\mid }\mathrm{/}n)R_{n-\mathrm{\mid }{I}_j\mathrm{\mid }}$. For equal folds of size $m$, this target is $R_{n-m}$; it is not automatically $R_n$, the risk after training on all $n$ observations.