Conformal scores and rank p-values

Let $\mathcal{Z}=\mathcal{X}\times \mathcal{Y}$ be a feature-response measurable space, let $m\ge 0$, and put $n=m+1$. A split score rule is a jointly measurable map $s:\mathcal{H}\times \mathcal{Z}\to \mathbb{R}$, where $H$ is a training statistic with values in the specified measurable rule space $(\mathcal{H},\mathcal{H})$. The measurable-statistic convention is that of Statistics and estimators. Once $H=h$ is fixed, calibration and test scores are $R_i=s(h,Z_i)$; the rule does not change when it scores the calibration or test example.

A full score assignment of length $n$ is a measurable map $A=(A_1,\dots ,A_n):{\mathcal{Z}}^n\to {\mathbb{R}}^n$ that is permutation-equivariant: $A({\tau }_{\pi }z)={\tau }_{\pi }A(z)\text{for every }z\in {\mathcal{Z}}^n\text{ and permutation }\pi \mathrm{.}$ For a candidate response $y$, append $(x,y)$ as the $n$th example and write $R_i(y)=A_i(Z_1,\dots ,Z_m,(x,y))$ and $r=R_n(y)$.

For either a split score vector or a full augmented score vector, the nonrandomized upper-rank p-value of the candidate is $p(y)=\frac{1}{m+1}{\sum }_{i=1}^{m+1}\mathbf{1}\{R_i(y)\ge R_{m+1}(y)\}\mathrm{.}$ An optional tie-randomized rank p-value is defined when a supplied random variable $U$ has Lebesgue probability law on $[0,1]$ and is independent of the data and training information: $p_U(y)=\frac{\mathrm{\#}\{i\le m:R_i(y)>r\}+U(1+\mathrm{\#}\{i\le m:R_i(y)=r\})}{m+1}\mathrm{.}$ The added $1$ counts the candidate itself in its tie group. A single such $U$ is shared across a candidate scan. This is an explicit randomization; the general Markov-kernel definition of a randomized procedure Randomized procedures as probability kernels does not assert that arbitrary kernels have a random-seed representation.

Here “uniform on $[0,1]$” means $\mathbb{P}(U\le u)=u$ for $0\le u\le 1$.

Smaller scores are treated as more conforming. The rank formula counts all ties conservatively in the nonrandomized version; the randomized formula spreads the candidate uniformly over its entire tie block.

For a split candidate scan at a fixed feature value $x$, the notation in these formulas is $R_i(y)=s(h,Z_i)$ for $1\le i\le m$ and $R_n(y)=s(h,(x,y))$, with $r=R_n(y)$. Thus the calibration scores do not depend on $y$, and the candidate is scored by the same fixed rule.