Valid p-values, Bonferroni, simultaneous coverage

Let $\mathsf{E}=((\mathcal{X},\mathcal{A}),\mathrm{\Theta },(P_{\theta }{)}_{\theta \in \mathrm{\Theta }})$ be a statistical experiment (Statistical experiments), and fix an integer $m\ge 1$ before observing the data. Write $[m]=\{1,\dots ,m\}$. A finite family of null hypotheses is a fixed list $H_{01},\dots ,H_{0m}$ with each $H_{0i}\subseteq \mathrm{\Theta }$. For each $i\in [m]$, let $p_i:\mathcal{X}\to [0,1]$ be a p-value statistic (Valid p-values). It is valid for $H_{0i}$ when

$P_{\theta }\{x:p_i(x)\le u\}\le u(\theta \in H_{0i},0\le u\le 1)\mathrm{.}$

If $H_{0i}=\mathrm{\varnothing }$, this universal condition is vacuous and $i$ never belongs to a true-null index set. For a nonempty null it agrees with the published valid-p-value definition. The family may have any joint dependence among its p-values. Validity is required only under the corresponding null laws; a p-value statistic alone does not imply validity.

For $\theta \in \mathrm{\Theta }$, the true-null index set is $I_0(\theta ):=\{i\in [m]:\theta \in H_{0i}\}\mathrm{.}$ A fixed deterministic multiple-testing rule is a family of measurable rejection events $D_i\in \mathcal{A}$, one for each $i\in [m]$. Its total number of rejections and number of false rejections at parameter $\theta$ are $R:={\sum }_{i=1}^m{\mathbf{1}}_{D_i},V_{\theta }:={\sum }_{i\in I_0(\theta )}{\mathbf{1}}_{D_i}\mathrm{.}$ Its familywise error rate (FWER) at $\theta$ is $P_{\theta }(V_{\theta }\ge 1)$, and its false discovery rate (FDR) at $\theta$ is $E_{\theta }\text{\hspace{0.17em}⁣}\left[\frac{V_{\theta }}{R\vee 1}\right],$ where $R\vee 1=\max \{R,1\}$. The random variable $V_{\theta }\mathrm{/}(R\vee 1)$ is the false discovery proportion (FDP). Strong FWER control at level $\alpha$ means that this FWER is at most $\alpha$ for every $\theta \in \mathrm{\Theta }$, including parameters for which only some of the null hypotheses are true. The denominator convention makes the FDP zero when no hypothesis is rejected.

If the testing rule uses auxiliary randomization, include that randomizer in the observation and its sigma-algebra before defining the rejection events $D_i$.

These quantities are well-defined: the finite sums of indicators are measurable, $0\le V_{\theta }\le R$, and $0\le V_{\theta }\mathrm{/}(R\vee 1)\le 1$, so the ratio is integrable under each probability law (Expectation of a nonnegative or integrable random variable).

Let $\mathsf{E}=((\mathcal{X},\mathcal{A}),\mathrm{\Theta },(P_{\theta }{)}_{\theta \in \mathrm{\Theta }})$ be a statistical experiment (Statistical experiments) with nonempty parameter set $\mathrm{\Theta }$. Fix $m\ge 1$ coordinate targets ${\psi }_i:\mathrm{\Theta }\to {\mathrm{\Psi }}_i$ and procedures $C_i:\mathcal{X}\to \mathcal{P}({\mathrm{\Psi }}_i)$, $i\in [m]$, where $\mathcal{P}({\mathrm{\Psi }}_i)$ denotes the set of all subsets of ${\mathrm{\Psi }}_i$. For every $\theta \in \mathrm{\Theta }$, suppose each membership event $A_i(\theta ):=\{x\in \mathcal{X}:{\psi }_i(\theta )\in C_i(x)\}$ belongs to $\mathcal{A}$ and has marginal coverage $P_{\theta }(A_i(\theta ))\ge 1-{\alpha }_i,$ where $0\le {\alpha }_i\le 1$ and ${\sum }_{i=1}^m{\alpha }_i\le \alpha \le 1$. Then, for every $\theta \in \mathrm{\Theta }$, $P_{\theta }\{{\psi }_i(\theta )\in C_i(X)\text{ for every }i\in [m]\}\ge 1-\alpha \mathrm{.}$ Thus the family of marginal confidence sets has simultaneous coverage at least $1-\alpha$. No independence assumption is needed.

Let $H_{01},\dots ,H_{0m}$ be a fixed finite family with $m\ge 1$ in a statistical experiment, and let $p_i$ be a valid p-value for $H_{0i}$ as in Null-valid p-values and familywise and false-discovery error rates. For $\alpha \in [0,1]$, reject $H_{0i}$ exactly when $p_i(X)\le \alpha \mathrm{/}m$. Then, for every $\theta$ in the parameter space, $P_{\theta }\{\text{at least one true null is rejected}\}=P_{\theta }(V_{\theta }\ge 1)\le \alpha \mathrm{.}$ The bound holds for every true-null configuration, including the global null and configurations with false nulls, and for arbitrary dependence among the p-values. This is strong FWER control; it is not an FDR guarantee.