Binned expected and maximum calibration error

Let $K\ge 2$ and let $({\widehat{p}}_i,y_i{)}_{i=1}^n$ be a nonempty finite evaluation sample, where ${\widehat{p}}_i\in {\mathrm{\Delta }}_K$ and $y_i\in \{1,\dots ,K\}$. For each $i$, let ${\widehat{y}}_i=\min \arg {\max }_j{\widehat{p}}_{ij}$, $c_i={\max }_j{\widehat{p}}_{ij}$, and $a_i={\mathbf{1}}_{\{{\widehat{y}}_i=y_i\}}$. Fix before evaluating the sample a finite Borel partition $(B_1,\dots ,B_M)$ of $[0,1]$. For $I_b=\{i:c_i\in B_b\}$ and $n_b=\mathrm{\mid }{I}_b\mathrm{\mid }>0$, define $\mathrm{acc}(B_b)=\frac{1}{n_b}{\sum }_{i\in I_b}{a}_i,\mathrm{conf}(B_b)=\frac{1}{n_b}{\sum }_{i\in I_b}{c}_i\mathrm{.}$ The binned expected calibration error and binned maximum calibration error for this sample and partition are ${\mathrm{ECE}}_{(B_b)}={\sum }_{b:n_b>0}\frac{n_b}{n}\mid \mathrm{acc}(B_b)-\mathrm{conf}(B_b)\mid ,{\mathrm{MCE}}_{(B_b)}={\max }_{b:n_b>0}\mid \mathrm{acc}(B_b)-\mathrm{conf}(B_b)\mid \mathrm{.}$ These are finite-sample, bin-dependent summaries, not the population expected calibration error. They compare observed top-label accuracy with average confidence within bins; log loss and Brier loss instead score probability forecasts directly and are strictly proper.

Top-label calibration: fixes the top-confidence and smallest-index tie conventions used for each sample prediction.

The Borel sigma-algebra of a topological space: specifies the measurable sets used in the fixed partition of $[0,1]$.

Probability calibration and calibration error: defines a population expected calibration error using a supplied calibration function and discrepancy, distinct from the empirical binned statistic here.

Strictly proper scoring rules on a finite label set: defines proper scoring rules and the finite-label log and Brier losses.

Log and Brier losses are strictly proper: proves strict propriety of the log and Brier losses.

Guo et al., full paper, Section 2: defines binned ECE and MCE from within-bin accuracy and confidence.