Calibration, proper losses, and surrogate classification

For an integer $K\ge 2$, let the label set be $\mathcal{Y}=\{1,\dots ,K\}$ and let ${\mathrm{\Delta }}_K=\{q\in [0,1{]}^K:{\sum }_{j=1}^K{q}_j=1\}$. A finite-label score loss is a function $S:{\mathrm{\Delta }}_K\times \mathcal{Y}\to [0,+\mathrm{\infty }]$ whose section $q\mapsto S(q,j)$ is Borel measurable for every label $j$. If the true label distribution is $p\in {\mathrm{\Delta }}_K$ and the reported forecast is $q\in {\mathrm{\Delta }}_K$, its expected score loss is $\bar{S}(p,q)={\sum }_{j=1}^K{p}_{j}S(q,j)$, where a term with $p_j=0$ contributes zero even when $S(q,j)=+\mathrm{\infty }$. The loss is proper if $\bar{S}(p,p)\le \bar{S}(p,q)$ for every $p,q\in {\mathrm{\Delta }}_K$, and strictly proper if it is proper and equality implies $q=p$. The logarithmic loss is $S_{\log }(q,j)=-\log q_j$, with $-\log 0=+\mathrm{\infty }$, where $\log$ is the natural logarithm; the Brier loss is $S_{\mathrm{B}}(q,j)={\sum }_{i=1}^K(q_i-{\mathbf{1}}_{\{i=j\}}{)}^2$.

Loss functions and risk: a loss is nonnegative and measurable in its action for each fixed parameter; no measurability on the parameter space is required, and its risk is the corresponding extended nonnegative integral.

The natural logarithm as the inverse of the exponential function: $\log$ is the inverse of the exponential function on positive reals.

The Borel sigma-algebra of a topological space: Borel subsets are the sets in the sigma-algebra generated by the open subsets of a topological space.

Wikipedia, Scoring rule: Definitions and Proper scoring rules specify forecasts, expected score, loss orientation, properness, and strictness; Categorical variables gives the finite-label logarithmic and Brier examples. The measurable extended nonnegative-loss conventions are stated explicitly here; the logarithmic reward example is negated.

For an integer $K\ge 2$, let ${\mathrm{\Delta }}_K=\{p\in [0,1{]}^K:{\sum }_{j=1}^K{p}_j=1\}\subset {\mathbb{R}}^K$ with its subspace Borel sigma-algebra, and let $C={\mathrm{\Delta }}_K\times {\mathrm{\Delta }}_K\subset {\mathbb{R}}^{2K}$ with its subspace Borel sigma-algebra. Let $(\mathrm{\Omega },\mathcal{F},\mathbb{P})$ be a probability space, let $Z=(U,V):\mathrm{\Omega }\to C$ be measurable, and let $\psi :C\to \mathbb{R}$ be continuous and jointly convex, meaning $\psi (\lambda z+(1-\lambda )w)\le \lambda \psi (z)+(1-\lambda )\psi (w)$ for all $z,w\in C$ and $\lambda \in [0,1]$. Define $\mathbb{E}Z$ coordinatewise. Then $\mathbb{E}Z=(\mathbb{E}U,\mathbb{E}V)\in C$ and $\psi (\mathbb{E}U,\mathbb{E}V)\le \mathbb{E}[\psi (U,V)]\mathrm{.}$

Durrett, Theorem 1.6.2: scalar Jensen inequality for an integrable random variable.

Vaicenavicius et al., supplementary Theorem 2: applies convexity to the pair of conditional mean forecasts in each fixed bin.

For every real $s\ge 1$, the function ${\phi }_s:[0,\mathrm{\infty })\to \mathbb{R}$ defined by ${\phi }_s(t)=-t^{1\mathrm{/}s}$, with $0^{1\mathrm{/}s}=0$, is convex.

Real powers for positive bases, with the zero-base positive-exponent convention: defines positive-base real powers and the zero-power convention used here.

This elementary lemma is derived locally from the published positive-real-power derivative and second-derivative convexity suppliers cited in its facts; no external source claim is imported.

Let $K\ge 2$, let $Y$ be a random variable into the finite label set $\{1,\dots ,K\}$ with its full power-set sigma-algebra on a probability space $(\mathrm{\Omega },\mathcal{F},\mathbb{P})$, and let $G=(G_1,\dots ,G_K):\mathrm{\Omega }\to {\mathrm{\Delta }}_K$ be a measurable forecast, where ${\mathrm{\Delta }}_K=\{p\in [0,1{]}^K:{\sum }_j{p}_j=1\}$. The forecast is fully reliable if for every label $j$ and every Borel set $B\subseteq {\mathrm{\Delta }}_K$, $\mathbb{P}(Y=j,G\in B)=\mathbb{E}[{\mathbf{1}}_{\{G\in B\}}{G}_j]\mathrm{.}$ It is classwise calibrated if for every $j$ and every Borel set $B\subseteq [0,1]$, $\mathbb{P}(Y=j,G_j\in B)=\mathbb{E}[{\mathbf{1}}_{\{G_j\in B\}}{G}_j]\mathrm{.}$ For the top-confidence calibration condition, define $C={\max }_j{G}_j$ and $\widehat{Y}=\min \{j:G_j=C\}$, so ties go to the smallest label. The forecast is top-confidence calibrated if for every Borel set $B\subseteq [0,1]$, $\mathbb{P}(Y=\widehat{Y},C\in B)=\mathbb{E}[{\mathbf{1}}_{\{C\in B\}}C]\mathrm{.}$ A measurable function $r:{\mathrm{\Delta }}_K\to {\mathrm{\Delta }}_K$ is a calibration function for $G$ if for every $j$ and every Borel set $B\subseteq {\mathrm{\Delta }}_K$, $\mathbb{P}(Y=j,G\in B)=\mathbb{E}[{\mathbf{1}}_{\{G\in B\}}{r}_j(G)]\mathrm{.}$ When such a function is supplied and $d:{\mathrm{\Delta }}_K\times {\mathrm{\Delta }}_K\to [0,+\mathrm{\infty })$ is measurable, the expected calibration error with discrepancy $d$ is ${\eta }_d=\mathbb{E}[d(r(G),G)]$.

Supervised prediction problems: fixes the measurable instance and label spaces, the joint law, and measurable predictors.

Conditional expectation given a sigma algebra: the event-integral identities in this definition are the conditional-expectation characterization when the indicated versions exist.

Probability measures and probability spaces: a probability law is a measure of total mass one.

The Borel sigma-algebra of a topological space: Borel subsets are generated by the open sets of the simplex's Euclidean subspace topology.

Vaicenavicius et al., full paper PDF: reliability, calibration functions, and expected miscalibration are defined in Sections 2, 3.1, and 3.2.

Assume AC. Let a supervised prediction problem have a measurable instance space $(\mathsf{X},{\mathrm{\Sigma }}_X)$, finite label set $\mathcal{Y}=\{y_1,\dots ,y_K\}$ with distinct labels, the full power-set sigma-algebra on $\mathcal{Y}$, and $K\ge 2$, and joint probability law $P$ on $\mathsf{X}\times \mathcal{Y}$; let $X$ and $Y$ be its coordinate random variables and let $\mathcal{G}=\{X^{-1}(A):A\in {\mathrm{\Sigma }}_X\}$, the sigma-algebra generated by $X$. There is a measurable function $\eta =({\eta }_1,\dots ,{\eta }_K):\mathsf{X}\to {\mathrm{\Delta }}_K$, where ${\mathrm{\Delta }}_K=\{p\in [0,1{]}^K:{\sum }_j{p}_j=1\}\subset {\mathbb{R}}^K$ has its subspace Borel sigma-algebra, such that for every label index $j$ and $A\in {\mathrm{\Sigma }}_X$, $P(X\in A,Y=y_j)=\mathbb{E}[{\eta }_j(X){\mathbf{1}}_{\{X\in A\}}]\mathrm{.}$

Supervised prediction problems: supplies measurable instance and label spaces, the joint law, and the coordinate random variables; here the finite label sigma-algebra is explicitly the full power set.

Conditional expectation given a sigma algebra: a conditional-expectation version is measurable with respect to the conditioning sigma-algebra and has the same integrals on its events.

Conditional expectation exists by radon nikodym: under AC, an integrable real random variable has a conditional-expectation version given any sub-sigma-algebra.

Durrett, Section 4.1: defines conditional expectation by event integrals and constructs it from Radon–Nikodym derivatives.

For every $K\ge 2$ and $p,q\in {\mathrm{\Delta }}_K$, the logarithmic and Brier losses in the definition of finite-label score loss satisfy $\bar{S}(p,p)\le \bar{S}(p,q)$, with equality only when $q=p$. The expected logarithmic loss at the truthful forecast is finite, including when some coordinates of $p$ vanish.

Strictly proper scoring rules on a finite label set: defines the simplex, expected loss, logarithmic loss, Brier loss, and propriety conventions.

The natural logarithm as the inverse of the exponential function: fixes the natural logarithm used in the logarithmic score.

The distinct encyclopedia treatment gives the scoring-rule conventions and categorical formulas (with the logarithmic reward negated). Strictness is proved locally from the declared logarithm supplier, rather than imported from a representation theorem. The displayed finite-sum arguments above separately verify the boundary cases $p_j=0$ and $q_j=0$ under this library's nonnegative-loss convention.

Assume AC. Let a finite-label supervised prediction problem have measurable instance space $(\mathsf{X},{\mathrm{\Sigma }}_X)$ and finite labels $\mathcal{Y}=\{y_1,\dots ,y_K\}$ with the full power-set sigma-algebra, where $K\ge 2$. Identify label $y_j$ with its index $j$ when applying numbered-label calibration, so forecast coordinate $j$ always represents the event $Y=y_j$. For every measurable feature $T:\mathsf{X}\to (\mathsf{Z},{\mathrm{\Sigma }}_Z)$, there is a measurable ${\eta }_T:\mathsf{Z}\to {\mathrm{\Delta }}_K$ such that $G={\eta }_T(T(X))$ is a conditional class-probability forecast given $T(X)$ and is fully reliable. This includes the identity feature and a constant feature.

A finite label has a conditional probability function: supplies the measurable simplex-valued conditional probability function and its event identities under AC.

Probability calibration and calibration error: full reliability is the event-integral identity for every Borel subset of the forecast simplex.

Vaicenavicius et al. Proposition 1 states the reliability of a conditional class-probability forecast. The event-integral proof above makes the version and measurable-event hypotheses explicit.

Assume AC. Let $X$ and a finite label $Y\in \{1,\dots ,K\}$ with the full power-set sigma-algebra, $K\ge 2$, have a joint probability law on standard-Borel spaces. For a proper finite-label score loss $S$, a measurable posterior forecast $G(x)=P(Y\in \cdot \mid X=x)$ minimizes the expected score loss among all measurable forecasts. If $S$ is strictly proper and the posterior forecast has finite expected score loss, equality of risk for another forecast $q(X)$ implies $q(X)=G(X)$ almost surely.

Strictly proper scoring rules on a finite label set: defines a proper or strictly proper nonnegative score loss on the finite label simplex.

Standard Borel spaces: standard-Borel spaces are measurably isomorphic to Polish spaces.

A posterior-risk minimizer minimizes prior risk: a measurable posterior-risk minimizer minimizes prior-integrated risk.

Disintegration of a joint law on standard borel spaces: supplies a measurable posterior kernel for the standard-Borel joint law.

The local definition supplies pointwise propriety; this proof integrates that defining inequality against the posterior law and does not import any scoring-rule representation theorem. The standard-Borel and Bayes-risk suppliers provide the measurable posterior and its prior-risk interpretation. Finite posterior risk is stated explicitly because equality of two infinite extended risks alone cannot yield uniqueness.

Let $Y\in \{-1,+1\}$ be the label in a supervised prediction problem with the full power-set sigma-algebra on this binary label set, and choose the finite-label conditional probability function ordered as $(-1,+1)$; write $\eta (x)={\eta }_{+1}(x)$. Let $\mathrm{\Phi }:\mathbb{R}\to [0,\mathrm{\infty })$ be finite, convex and nondecreasing. It is continuous on $\mathbb{R}$; extend it to $\bar{\mathbb{R}}=\mathbb{R}\cup \{-\mathrm{\infty },+\mathrm{\infty }\}$ by its one-sided monotone limits, which may equal $+\mathrm{\infty }$. The resulting extended function is Borel measurable. For $u\in \bar{\mathbb{R}}$ define the conditional margin risk $L_{\mathrm{\Phi }}(x,u)=\eta (x)\mathrm{\Phi }(-u)+(1-\eta (x))\mathrm{\Phi }(u),$ using $0\cdot (+\mathrm{\infty })=0$ at endpoints. A measurable extended-real conditional minimizer is a measurable $h_{\mathrm{\Phi }}^{\ast }:\mathsf{X}\to \bar{\mathbb{R}}$ satisfying $L_{\mathrm{\Phi }}(x,h_{\mathrm{\Phi }}^{\ast }(x))\le L_{\mathrm{\Phi }}(x,u)$ for every $x$ and every $u\in \bar{\mathbb{R}}$. For a measurable real-valued score $h$, its margin loss is ${\mathrm{\ell }}_{\mathrm{\Phi }}(h(x),y)=\mathrm{\Phi }(-yh(x))$, its surrogate risk is $L_{\mathrm{\Phi }}(h)=\mathbb{E}[\mathrm{\Phi }(-Yh(X))]$, and its conditional-minimum risk is $L_{\mathrm{\Phi }}^{\ast }=\mathbb{E}[L_{\mathrm{\Phi }}(X,h_{\mathrm{\Phi }}^{\ast }(X))]$. Under the zero-score-is-error convention, its classification risk is $R_{01}(h)=P(Yh(X)\le 0)$, and the Bayes classification risk is $R_{01}^{\ast }=\mathbb{E}[\min \{\eta (X),1-\eta (X)\}]$. Write $h^{\ast }(x)=\eta (x)-\frac{1}{2}$ for the centered conditional class probability.

Supervised prediction problems: fixes the measurable instance and label spaces and joint law.

A finite label has a conditional probability function: the chosen $\eta$ is a measurable version whose event integrals give the conditional label probabilities.

Population and empirical risk: population risk is an extended nonnegative integral of the loss under the joint law.

Loss functions and risk: fixes the loss and risk conventions used for the margin loss.

A measurable function between measurable spaces: a function is measurable when preimages of measurable sets are measurable.

A convex function on an open convex set is continuous: every finite convex function on the open convex set $\mathbb{R}$ is continuous.

Let $Y$ be a random variable into $\{1,\dots ,K\}$ with its full power-set sigma-algebra, $K\ge 2$, and let $G\in {\mathrm{\Delta }}_K$ be a random forecast with measurable calibration function $r:{\mathrm{\Delta }}_K\to {\mathrm{\Delta }}_K$. Let $d:{\mathrm{\Delta }}_K\times {\mathrm{\Delta }}_K\to [0,\mathrm{\infty })$ be continuous and satisfy $d(p,p)=0$. For a finite data-independent Borel partition $\mathcal{P}$ of ${\mathrm{\Delta }}_K$, draw an iid sequence $(G_i,Y_i)$ from the joint law of $(G,Y)$, let $V_i=e_{Y_i}$ be the corresponding standard basis vector, and form the empirical estimator ${\widehat{E}}_{n,\mathcal{P}}={\sum }_{B\in \mathcal{P}}\frac{N_B}{n}d({\bar{V}}_B,{\bar{G}}_B),$ where $N_B=\mathrm{\#}\{i\le n:G_i\in B\}$ and, when $N_B>0$, ${\bar{V}}_B$ and ${\bar{G}}_B$ are the sample averages in bin $B$; when $N_B=0$, set both to any fixed $q_0\in {\mathrm{\Delta }}_K$. Define $E_{\mathcal{P}}={\sum }_{B\in \mathcal{P}:P(G\in B)>0}P(G\in B)d\text{\hspace{0.17em}⁣}(\mathbb{E}[r(G)\mid G\in B],\mathbb{E}[G\mid G\in B])\mathrm{.}$ For each fixed $\mathcal{P}$, ${\widehat{E}}_{n,\mathcal{P}}\to E_{\mathcal{P}}$ almost surely. If $d$ is additionally jointly convex, then $E_{\mathcal{P}}\le \mathbb{E}[d(r(G),G)]$. Without requiring convexity, if ${\mathcal{P}}_m$ is a sequence of finite data-independent Borel partitions whose mesh ${\max }_{B\in {\mathcal{P}}_m}{\mathrm{diam}}_2(B)$ tends to zero, where ${\mathrm{diam}}_2(B)={\sup }_{u,v\in B}\mathrm{\parallel }u-v{\mathrm{\parallel }}_2$, and $r$ is Lipschitz with respect to the Euclidean metric on ${\mathrm{\Delta }}_K$, then $E_{{\mathcal{P}}_m}\to \mathbb{E}[d(r(G),G)]$. Thus the iterated limit ${\lim }_{m\to \mathrm{\infty }}{\lim }_{n\to \mathrm{\infty }}{\widehat{E}}_{n,{\mathcal{P}}_m}$ equals the expected calibration error almost surely; no diagonal rate is asserted.

Probability calibration and calibration error: defines a calibration function, the event-integral identity, and expected calibration error.

The Borel sigma-algebra of a topological space: Borel sets are generated by the open sets in the forecast simplex topology.

Kolmogorov iid l1 strong law: gives almost-sure convergence of averages of iid integrable real random variables.

Jensen's inequality for pairs of simplex-valued forecasts: gives Jensen's inequality for the pair of mean label and forecast vectors in each bin.

Vaicenavicius et al. Theorems 1–2 and Appendix A.2 provide the fixed-bin Jensen lower bound and the iterated refining-bin consistency route. The proof here explicitly handles empty bins, zero-mass bins, and the order of limits.

For a binary supervised problem with the full power-set sigma-algebra on $Y\in \{-1,+1\}$, conditional positive-label probability $\eta$, and any measurable real-valued score $h$, use $R_{01}(h)=P(Yh(X)\le 0)$ and $R_{01}^{\ast }=\mathbb{E}\min \{\eta (X),1-\eta (X)\}$. Then $R_{01}(h)-R_{01}^{\ast }=2\mathbb{E}\text{\hspace{0.17em}⁣}\left[\mid \eta (X)-\frac{1}{2}\mid {\mathbf{1}}_{\{h(X)(\eta (X)-\frac{1}{2})\le 0\}}\right]+\mathbb{E}\text{\hspace{0.17em}⁣}[\min \{\eta (X),1-\eta (X)\}{\mathbf{1}}_{\{h(X)=0\}}]\mathrm{.}$

Binary surrogate classification risk: fixes $\eta$, $R_{01}$, and $R_{01}^{\ast }$, including the zero-score-is-error convention.

Mohri et al., Lemma 4.5, printed pp. 74–75 (PDF pp. 91–92): the source predicts $+1$ at zero, and its displayed equality overcounts when $h(x)=0$ and $\eta (x)>1\mathrm{/}2$; the corrected zero-score-is-error identity above is proved here.