KMR's finite group-conditioned risk-score assignment and calibration

Let $\mathrm{\Sigma }$ and $\mathcal{B}$ be nonempty finite sets of feature types and score bins. For each group $t\in \{1,2\}$, let $N_t>0$ be its total mass and let $a_{t\sigma }\ge 0$ satisfy ${\sum }_{\sigma \in \mathrm{\Sigma }}{a}_{t\sigma }=1$. Let $p_{\sigma }\in [0,1]$ be the probability of the positive binary outcome $Y=1$ for feature type $\sigma$, with the same $p_{\sigma }$ in both groups. A randomized risk assignment consists of scores $v_b\in [0,1]$ for $b\in \mathcal{B}$ and a single matrix $X_{\sigma b}\ge 0$ with ${\sum }_{b\in \mathcal{B}}{X}_{\sigma b}=1$ for each $\sigma$; conditional on feature type $\sigma$, the assignment chooses bin $b$ with probability $X_{\sigma b}$, independently of group and outcome. Thus, for a group-$t$ individual chosen according to its feature distribution, ${\mathrm{Pr}}_t(\mathrm{\Sigma }=\sigma ,Y=1,B=b)=a_{t\sigma }{p}_{\sigma }{X}_{\sigma b}$ and ${\mathrm{Pr}}_t(\mathrm{\Sigma }=\sigma ,Y=0,B=b)=a_{t\sigma }(1-p_{\sigma })X_{\sigma b}$, and the assigned score is $S=v_B$. The expected positive mass is ${\mu }_t=N_t{\sum }_{\sigma }{a}_{t\sigma }{p}_{\sigma }$, and the group base rate is ${\rho }_t={\mu }_t\mathrm{/}{N}_t$.

The assignment is calibrated within group $t$ if for every $b\in \mathcal{B}$,

$$\sum _{\sigma \in \mathrm{\Sigma }}{a}_{t\sigma }{p}_{\sigma }{X}_{\sigma b}=v_b\sum _{\sigma \in \mathrm{\Sigma }}{a}_{t\sigma }{X}_{\sigma b}\mathrm{.}$$

This cross-multiplied condition also covers bins with zero assigned mass. The instance permits perfect prediction when $p_{\sigma }\in \{0,1\}$ for every feature type with $a_{1\sigma }+a_{2\sigma }>0$, with the bin set and scores allowed to be chosen for this existence question. Assign the occupied types with $p_{\sigma }=0$ to a score-$0$ bin and those with $p_{\sigma }=1$ to a score-$1$ bin, omitting an unused outcome bin. For a prescribed $\mathcal{B}$, at least two bins are needed when both deterministic outcomes occur among occupied types; one bin suffices when only one occurs. A score assignment is perfectly predictive when $S=Y$ almost surely in each group.

Probability calibration and calibration error: within each group, full reliability of the binary forecast $(1-S,S)$ gives the calibration identity for each distinct score value. This is the per-bin identity above when every score value occurs in only one bin; if distinct bins share a score, full reliability gives only the aggregate identity across those bins.

Kleinberg, Mullainathan, and Raghavan, complete 23-page paper: §§1.1–1.2 define the finite feature-type, group, randomized-bin model and calibration within each group.

Applied Statistics: From Data to Decisions, full chapter: the section “The impossibility: you cannot satisfy them all” treats calibration within groups for real-valued scores and distinguishes the score theorem from its binary-classifier companion.