Bayes predictors and excess risk

Let $\mathcal{M}$ be the class of all measurable predictors from $\mathsf{X}$ to $\mathsf{A}$. The class Bayes risk is $R_P^{\ast }(\mathcal{F}):={\inf }_{f\in \mathcal{F}}{R}_P(f)$, and the unrestricted Bayes risk is $R_P^{\ast }:={\inf }_{g\in \mathcal{M}}{R}_P(g)$. A class Bayes predictor is a member of $\mathcal{F}$ attaining the first infimum; an unrestricted Bayes predictor, also called a Bayes predictor, is a member of $\mathcal{M}$ attaining the second. No attainment is asserted. If an infimum is $+\mathrm{\infty }$, every member of its nonempty class has infinite risk and attains that extended infimum, but no finite excess risk results.

For $f\in \mathcal{F}$ with both terms finite, its class excess risk is $R_P(f)-R_P^{\ast }(\mathcal{F})$. For a measurable predictor $g\in \mathcal{M}$ with both terms finite, its unrestricted excess risk is $R_P(g)-R_P^{\ast }$. These differences are left undefined if either risk is infinite.

Write $P_X(C):=P(C\times \mathsf{Y})$ for the instance marginal. If a regular conditional label law $K(x,dy)$ given $x$ is supplied, its conditional risk is $q(x,a):={\int }_{\mathsf{Y}}\mathrm{\ell }(a,y)K(x,dy)$. This function is measurable when the loss is product-measurable. A conditional Bayes rule is a measurable predictor that attains ${\inf }_{a\in \mathsf{A}}q(x,a)$ on one measurable set of full $P_X$-measure. Such a kernel exists under AC whenever the label space is standard Borel, even if the instance space is an arbitrary measurable space; if the label space is an arbitrary measurable space, no conditional law or measurable minimizer is asserted. When both instance and label spaces are standard Borel and AC holds, the conditional Bayes theorem shows that any such rule is an unrestricted Bayes predictor.

Supervised prediction problems: fixes the measurable instance, label and action spaces, joint law, predictor class and loss.

Population and empirical risk: defines $R_P(f)$ as the extended nonnegative population risk.

Standard Borel spaces: a standard Borel space is measurably isomorphic to a Polish space.

Regular conditional distribution: a regular conditional distribution is an everywhere probability kernel satisfying the conditional-event integral identities.

The Axiom of Choice: AC asserts that every family of nonempty sets has a choice function.

A standard-Borel label has a conditional kernel over an arbitrary measurable instance space: under AC, a joint law with standard Borel label space admits an everywhere label-given-instance probability kernel even for an arbitrary measurable instance space.

Measurability of integration against a kernel: integration of a nonnegative product-measurable function against a probability kernel is measurable.

A conditional Bayes rule minimizes population risk: under standard-Borel and AC hypotheses, a measurable conditional minimizer is globally risk-minimizing.

Kallenberg, Foundations of Modern Probability, 2nd ed., Theorem 6.3: a Borel target admits a conditional-distribution kernel from any measurable conditioning space.