Calibration, proper losses, and surrogate classification — Examples

Each forecast row and each conditional-label row in Models A, B, and C has nonnegative coordinates summing to one, so all entries define simplex forecasts and conditional label distributions. In each model the three forecast vectors are distinct. Therefore full reliability would require $p_i=g_i$ at every row by [F1], but Model A row 1, Model B row 1, and Model C row 1 each have $p_i\mathrm{\ne }{g}_i$. Full reliability fails in all three models.

In Model A, for each label the groups with forecast coordinate $0.4$ have conditional frequency $0.4$, and the groups with coordinate $0.3$ have average conditional frequency $0.3$: for labels 1, 2, and 3 the latter averages are respectively $(0.2+0.4)\mathrm{/}2$, $(0.4+0.2)\mathrm{/}2$, and $(0.2+0.4)\mathrm{/}2$. Thus Model A is classwise calibrated. Its top confidence is always $0.4$, its predicted labels are $1,2,3$ in the three rows, and each corresponding conditional correctness probability is $0.4$, so it is top-confidence calibrated.

In Model B the top confidence is always $0.5$, the predicted labels are $1,2,3$, and the average conditional correctness is $(0.7+0.5+0.3)\mathrm{/}3=0.5$, so top-confidence calibration holds. Classwise calibration fails because forecast coordinate $G_1=0.5$ occurs only in row 1, where $P(Y=1\mid X=1)=0.7\mathrm{\ne }0.5$.

In Model C, for each label the forecast coordinate $0.5$ group has average conditional frequency $0.5$, and the coordinate $0$ group has frequency $0$: the three nonzero-group averages are $(0.8+0.2)\mathrm{/}2$, $(0.2+0.8)\mathrm{/}2$, and $(0.8+0.2)\mathrm{/}2$. Thus classwise calibration holds. All top confidences equal $0.5$; the tie-broken predicted labels are $1,1,2$, with conditional correctness probabilities $0.8,0.2,0.8$. Their average is $0.6\mathrm{\ne }0.5$, so top-confidence calibration fails. Steps 1.1, 1.2, and 1.3 establish the other model claims; the Model C calculation completes the pairwise distinctions.

Independence gives $P(Y=j,X\in A)=P(Y=j)P(X\in A)=p_{j}P(X\in A)$ for every feature event $A$ and class $j$. Thus the conditional label distribution given either $X=a$ or $X=b$ is $p=(0.7,0.2,0.1)$, and $G$ is exactly that conditional forecast.

For any Borel set $B\subseteq {\mathrm{\Delta }}_3$, if $p\in B$ then $P(Y=j,G\in B)=p_j=\mathbb{E}[{\mathbf{1}}_{\{G\in B\}}{G}_j]$; if $p\mathrm{\notin }B$, both sides are zero. By [F1], $G$ is fully reliable; its constancy means it has the same output at both feature values and so does not distinguish them.

The forecast values identify the two feature values, so the calibration function is $r(0.6,0.4)=(0.8,0.2)$ and $r(0.8,0.2)=(0.6,0.4)$. Therefore $\mathbb{E}[d(r(G),G)]=\frac{1}{2}\mathrm{\mid }0.8-0.6\mathrm{\mid }+\frac{1}{2}\mathrm{\mid }0.6-0.8\mathrm{\mid }=0.2$.

In the single bin $B={\mathrm{\Delta }}_2$, the mean forecast first coordinate is $(0.6+0.8)\mathrm{/}2=0.7$, and the mean label-indicator first coordinate is $(0.8+0.6)\mathrm{/}2=0.7$. Hence the population binned discrepancy is $\mathrm{\mid }0.7-0.7\mathrm{\mid }=0$. By [F2] the empirical one-bin estimate converges almost surely to zero; equivalently, [F3] gives convergence of both sample averages to $0.7$.

The probability of $T=u$ is $2\mathrm{/}3$, and the conditional probability of $Y=1$ there is $(0.1+0.9)\mathrm{/}2=0.5$; at $T=v$ it is $0.75$. Thus the forecast is exactly the conditional label distribution given $T$, and [F1] implies full reliability. Directly, each forecast value identifies $T$, so the conditional distribution of $Y$ given $G=(0.5,0.5)$ is $(0.5,0.5)$ and given $G=(0.75,0.25)$ is $(0.75,0.25)$, which is [F2].

The original feature values $a$ and $b$ have different conditional label laws, with success probabilities $0.1$ and $0.9$, but both map to the same forecast. Hence observing the forecast does not determine which of those feature values occurred, so the forecast discards information about $X$ while remaining calibrated.