Predictive calibration and uncertainty under shift

Let $K\ge 2$, let the feature space be a measurable space $\mathcal{X}$, and let the label set be $\mathcal{Y}=\{1,\dots ,K\}$ with all subsets measurable. Let $H$ be a training statistic valued in a measurable rule space $\mathcal{H}$, and let $s:\mathcal{H}\times \mathcal{X}\times \mathcal{Y}\to \mathbb{R}$ be a jointly measurable score rule. The rule is fixed after training and is used unchanged on calibration and test examples. Given $m\ge 0$ calibration examples $(X_i,Y_i)$ and $0<\alpha <1$, compute $R_i=s(H,X_i,Y_i),k=\lceil (m+1)(1-\alpha )\rceil \mathrm{.}$ If $k\le m$, sort the calibration scores by the ordered keys $(R_i,i)$ and let $q$ be the score value at position $k$. If $k=m+1$, set $q=+\mathrm{\infty }$. For a feature value $x$, return the prediction set

For the optional tie-randomized version, use one random variable $U$ that is uniform on $[0,1]$ and independent of the training information and all calibration and test examples, shared across the candidate-label scan. For $r_y=s(H,x,y)$, define $p_U(y)=\frac{\mathrm{\#}\{i\le m:R_i>r_y\}+U(1+\mathrm{\#}\{i\le m:R_i=r_y\})}{m+1},C_U(x)=\{y\in \mathcal{Y}:p_U(y)>\alpha \}\mathrm{.}$ The candidate is included in its own tie group. The algorithm defines these sets; any coverage claim requires separate distributional hypotheses.

Statistics and estimators: a training statistic is a measurable map into its stated measurable rule space.

Conformal scores and rank p-values: fixes the split-score convention, upper-rank formula, tie randomization, and independent uniform input.

Split conformal prediction set: supplies the general split-conformal quantile construction, including the $+\mathrm{\infty }$ sentinel.

Finite order statistics and rank maps are measurable: gives measurability of the finite score order statistic and rank maps.

Independent sigma-algebras and independent events: defines independence of the optional uniform variable from the data.

Lei et al., full paper, Section 2.2 and Appendix A.1: gives the split-conformal order-statistic construction and its finite-sample exchangeable-rank argument for regression.

Tibshirani, full lecture notes, Sections 1.2–2: derives the adjusted rank, split calibration score, sentinel convention, and optional exact tie randomization.

Let $(\mathrm{\Omega },\mathcal{F},\mathbb{P})$ be a probability space, let $K\ge 2$, let $Y:\mathrm{\Omega }\to \{1,\dots ,K\}$ be a random variable, with the label set carrying the full power-set sigma-algebra, and let $G=(G_1,\dots ,G_K):\mathrm{\Omega }\to {\mathrm{\Delta }}_K$ be a measurable random forecast, where the probability simplex is ${\mathrm{\Delta }}_K=\{p\in [0,1{]}^K:{\sum }_{j=1}^K{p}_j=1\}$. Define the predicted label and top confidence by $\widehat{Y}=\min \{j:G_j={\max }_{\mathrm{\ell }\in \{1,\dots ,K\}}{G}_{\mathrm{\ell }}\},C={\max }_{\mathrm{\ell }\in \{1,\dots ,K\}}{G}_{\mathrm{\ell }}\mathrm{.}$ The smallest-index rule resolves ties. The forecast is top-label calibrated if for every Borel set $B\subseteq [0,1]$, $\mathbb{P}(Y=\widehat{Y},C\in B)=\mathbb{E}[{\mathbf{1}}_{\{C\in B\}}C]\mathrm{.}$ Thus, among predictions whose top confidence lies in any specified set of confidence values, the frequency of a correct top label equals the average reported top confidence.

Probability calibration and calibration error: its top-confidence condition is the same event-integral identity, with the same smallest-index tie rule.

Probability measures and probability spaces: fixes the probability law and expectation used in the event-integral identity.

The Borel sigma-algebra of a topological space: specifies the Borel sets quantified over in the definition.

Guo et al., full paper, Section 2: defines confidence as the probability attached to the predicted label and motivates calibration by correctness frequency.

Fix $K\ge 2$. The input is a trained classifier with a measurable, finite-valued logit map $z:\mathcal{X}\to {\mathbb{R}}^K$, a nonempty finite validation sample $(x_i,y_i{)}_{i=1}^{n_v}$ with $n_v\ge 1$ and $y_i\in \{1,\dots ,K\}$, and a finite nonempty set of candidate temperatures $\mathcal{T}\subset (0,\mathrm{\infty })$. The validation labels were not used to fit the logits. For $T\in \mathcal{T}$, compute $p_T(j\mid x)=\frac{\exp (z_j(x)\mathrm{/}T)}{{\sum }_{\mathrm{\ell }=1}^K\exp (z_{\mathrm{\ell }}(x)\mathrm{/}T)}(j=1,\dots ,K),L_{\mathrm{v}\mathrm{a}\mathrm{l}}(T)=-\frac{1}{n_v}{\sum }_{i=1}^{n_v}\log p_T(y_i\mid x_i)\mathrm{.}$ Choose the smallest temperature among the minimizers, $T^{\ast }=\min \arg {\min }_{T\in \mathcal{T}}{L}_{\mathrm{v}\mathrm{a}\mathrm{l}}(T)\mathrm{.}$ and return the forecasts $p_{T^{\ast }}(\cdot \mid x)$ and predicted label $\min \arg {\max }_j{p}_{T^{\ast }}(j\mid x)$ for each input $x$. This is a finite-candidate version of validation negative-log-likelihood temperature scaling. It selects only by empirical validation loss; it makes no finite-sample calibration guarantee for future data, particularly when the validation and future data laws differ.

Strictly proper scoring rules on a finite label set: defines the finite-label logarithmic loss used by the validation objective.

Log and Brier losses are strictly proper: establishes that log loss is strictly proper; this does not itself give a finite-sample calibration guarantee.

The natural logarithm as the inverse of the exponential function: fixes the logarithm and its domain on positive reals.

The real exponential function and the number $e$ by a power series: defines the exponential used in the softmax formula.

A measurable function between measurable spaces: specifies the measurability condition on the logit map.

Positive temperature scaling preserves the maximizing class ↗: proves normalization, finite validation loss, existence of the finite-grid minimizer, and preservation of the maximizing class.

Guo et al., full paper, Section 4.2: introduces multiclass temperature scaling, validation NLL selection, and preservation of the class prediction.

Let $K\ge 2$ and let $({\widehat{p}}_i,y_i{)}_{i=1}^n$ be a nonempty finite evaluation sample, where ${\widehat{p}}_i\in {\mathrm{\Delta }}_K$ and $y_i\in \{1,\dots ,K\}$. For each $i$, let ${\widehat{y}}_i=\min \arg {\max }_j{\widehat{p}}_{ij}$, $c_i={\max }_j{\widehat{p}}_{ij}$, and $a_i={\mathbf{1}}_{\{{\widehat{y}}_i=y_i\}}$. Fix before evaluating the sample a finite Borel partition $(B_1,\dots ,B_M)$ of $[0,1]$. For $I_b=\{i:c_i\in B_b\}$ and $n_b=\mathrm{\mid }{I}_b\mathrm{\mid }>0$, define $\mathrm{acc}(B_b)=\frac{1}{n_b}{\sum }_{i\in I_b}{a}_i,\mathrm{conf}(B_b)=\frac{1}{n_b}{\sum }_{i\in I_b}{c}_i\mathrm{.}$ The binned expected calibration error and binned maximum calibration error for this sample and partition are ${\mathrm{ECE}}_{(B_b)}={\sum }_{b:n_b>0}\frac{n_b}{n}\mid \mathrm{acc}(B_b)-\mathrm{conf}(B_b)\mid ,{\mathrm{MCE}}_{(B_b)}={\max }_{b:n_b>0}\mid \mathrm{acc}(B_b)-\mathrm{conf}(B_b)\mid \mathrm{.}$ These are finite-sample, bin-dependent summaries, not the population expected calibration error. They compare observed top-label accuracy with average confidence within bins; log loss and Brier loss instead score probability forecasts directly and are strictly proper.

Top-label calibration: fixes the top-confidence and smallest-index tie conventions used for each sample prediction.

The Borel sigma-algebra of a topological space: specifies the measurable sets used in the fixed partition of $[0,1]$.

Probability calibration and calibration error: defines a population expected calibration error using a supplied calibration function and discrepancy, distinct from the empirical binned statistic here.

Strictly proper scoring rules on a finite label set: defines proper scoring rules and the finite-label log and Brier losses.

Log and Brier losses are strictly proper: proves strict propriety of the log and Brier losses.

Guo et al., full paper, Section 2: defines binned ECE and MCE from within-bin accuracy and confidence.

Let $K\ge 2$, let $z\in {\mathbb{R}}^K$, and let $T>0$. Define $p_j(T)=\frac{\exp (z_j\mathrm{/}T)}{{\sum }_{\mathrm{\ell }=1}^K\exp (z_{\mathrm{\ell }}\mathrm{/}T)}(j=1,\dots ,K)\mathrm{.}$ Then $p(T)$ lies in the relative interior of ${\mathrm{\Delta }}_K$ in the affine hyperplane $\{p\in {\mathbb{R}}^K:{\sum }_j{p}_j=1\}$ and $\arg {\max }_{j\in \{1,\dots ,K\}}{p}_j(T)=\arg {\max }_{j\in \{1,\dots ,K\}}{z}_j$. For the finite validation inputs and candidate temperatures specified in the validation temperature-scaling algorithm, every validation loss is finite and the stated smallest-minimizer rule returns a unique candidate.

Validation temperature scaling: defines the finite candidate set, validation NLL, and tie-breaking rule whose well-definedness is also proved here.

The real exponential function and the number $e$ by a power series: defines the real exponential used in the softmax formula.

The exponential is positive and satisfies $\exp (-x)=1\mathrm{/}\exp (x)$: gives positivity of the exponential.

The exponential function is strictly increasing: supplies strict monotonicity for pairwise logit comparisons.

The natural logarithm as the inverse of the exponential function: defines the logarithm on positive reals, the domain of the validation losses.

Guo et al., full paper, Section 4.2: states that the scalar temperature leaves the maximizing class unchanged; the argument here verifies the claim under the stated positive-temperature hypotheses.