Upper-tail p-values under an explicit null calibration

Let $(S,\mathcal{S})$ be a measurable data space, let $\mathrm{\Theta }$ be a nonempty parameter set, and let $({\mathbb{P}}_{\theta }{)}_{\theta \in \mathrm{\Theta }}$ be a family of probability measures on $(S,\mathcal{S})$; these data form the statistical model. Let ${\mathrm{\Theta }}_0\subseteq \mathrm{\Theta }$ be a nonempty null set and let $T:S\to \mathbb{R}$ be a measurable statistic, meaning $T^{-1}(A)\in \mathcal{S}$ for every $A\in \mathcal{B}(\mathbb{R})$; in particular every upper-tail set is measurable. Fix an explicit calibration probability measure $Q_0$ on $(\mathbb{R},\mathcal{B}(\mathbb{R}))$ for the statistic under the null procedure. For a simple null ${\mathrm{\Theta }}_0=\{{\theta }_0\}$, the exact calibration is $Q_0={\mathbb{P}}_{{\theta }_0}\circ T^{-1}$, meaning $Q_0(A)={\mathbb{P}}_{{\theta }_0}(T^{-1}(A))$ for every Borel set $A$. For a composite null or an asymptotic analysis, $Q_0$ is supplied by the specified calibration procedure, whose nuisance-parameter treatment and approximation assumptions are part of the data.

For an observed datum $s\in S$, with the convention that larger $T$ values are more extreme, the upper-tail p-value is $p_0(s):=Q_0(\{t\in \mathbb{R}:t\ge T(s)\})\mathrm{.}$ It is a probability under the stated null calibration for a test statistic at least as extreme as the observed one. It is not a probability assigned to the null hypothesis conditional on the data; no probability distribution on $\mathrm{\Theta }$ is supplied by this definition. The definition alone establishes neither type-I error control nor super-uniformity; any such validity claim requires a separate argument for the specified sampling model and calibration. A lower-tail or two-sided procedure must specify its own measurable extremeness set instead of using the displayed upper-tail convention.

Model families, null sets, and statistical tests

Borel sets used for the calibration law

Probability measures

Wasserman, full statistics textbook, Chapter 10 §§10.2 and 10.12

Cowan et al., full likelihood-based tests paper, §§2.1 and 2.3