Top-label calibration

Let $(\mathrm{\Omega },\mathcal{F},\mathbb{P})$ be a probability space, let $K\ge 2$, let $Y:\mathrm{\Omega }\to \{1,\dots ,K\}$ be a random variable, with the label set carrying the full power-set sigma-algebra, and let $G=(G_1,\dots ,G_K):\mathrm{\Omega }\to {\mathrm{\Delta }}_K$ be a measurable random forecast, where the probability simplex is ${\mathrm{\Delta }}_K=\{p\in [0,1{]}^K:{\sum }_{j=1}^K{p}_j=1\}$. Define the predicted label and top confidence by $\widehat{Y}=\min \{j:G_j={\max }_{\mathrm{\ell }\in \{1,\dots ,K\}}{G}_{\mathrm{\ell }}\},C={\max }_{\mathrm{\ell }\in \{1,\dots ,K\}}{G}_{\mathrm{\ell }}\mathrm{.}$ The smallest-index rule resolves ties. The forecast is top-label calibrated if for every Borel set $B\subseteq [0,1]$, $\mathbb{P}(Y=\widehat{Y},C\in B)=\mathbb{E}[{\mathbf{1}}_{\{C\in B\}}C]\mathrm{.}$ Thus, among predictions whose top confidence lies in any specified set of confidence values, the frequency of a correct top label equals the average reported top confidence.

Probability calibration and calibration error: its top-confidence condition is the same event-integral identity, with the same smallest-index tie rule.

Probability measures and probability spaces: fixes the probability law and expectation used in the event-integral identity.

The Borel sigma-algebra of a topological space: specifies the Borel sets quantified over in the definition.

Guo et al., full paper, Section 2: defines confidence as the probability attached to the predicted label and motivates calibration by correctness frequency.