Missingness and nonidentification

Let $(\mathrm{\Omega },\mathcal{F},P)$ be a probability space and let $W=(W_1,\dots ,W_k)$, for a finite $k\ge 1$, be a random element of the product measurable space $\mathcal{W}={\prod }_{j=1}^k{\mathcal{W}}_j$ (Random elements and real random variables, Measurable spaces and measurable sets). The law of $W$ is the pushforward probability measure defined in Law or distribution of a random element. Let $\mathcal{R}$ be a nonempty finite set of observation patterns with its discrete sigma-algebra, and let $R:\mathrm{\Omega }\to \mathcal{R}$ be a random element.

For each $r\in \mathcal{R}$, specify the recorded coordinate set $I_r\subseteq \{1,\dots ,k\}$ and its complement $J_r=\{1,\dots ,k\}\setminus I_r$. The observed projection $W_{\mathrm{o}\mathrm{b}\mathrm{s}}(r)$ is the coordinate projection into ${\mathcal{W}}_{\mathrm{o}\mathrm{b}\mathrm{s}}(r):={\prod }_{j\in I_r}{\mathcal{W}}_j$, and the missing projection $W_{\mathrm{m}\mathrm{i}\mathrm{s}}(r)$ is the projection into ${\mathcal{W}}_{\mathrm{m}\mathrm{i}\mathrm{s}}(r):={\prod }_{j\in J_r}{\mathcal{W}}_j$. An empty coordinate tuple takes values in a one-point measurable space. The observed record is $O=(R,W_{\mathrm{o}\mathrm{b}\mathrm{s}}(R))$ in the disjoint union ${⨆}_{r\in \mathcal{R}}(\{r\}\times {\mathcal{W}}_{\mathrm{o}\mathrm{b}\mathrm{s}}(r))$, where a set in the disjoint-union sigma-algebra is measurable exactly when each pattern section is measurable. The pattern records which coordinates are available; it does not replace an unrecorded full-data value by zero.

Fix the full-data/observed-data partition of Full and observed data under a missingness pattern. A response mechanism is a specified conditional law $g(r\mid w)$ of the pattern $R$ given full data $W=w$, represented by a probability kernel from the full-data space to the finite pattern space (Conditional law given a random element, Measure kernel and probability kernel). In a parameterized model, write the response kernel as $g_{\psi }$; it must be a probability distribution over patterns for each full-data value.

Relative to a specified full-data law, the mechanism is missing completely at random (MCAR) when, for every pattern $r$, $g(r\mid w)$ is constant in $w$ almost surely. It is missing at random (MAR) when, for every pattern $r$, there is a measurable function $h_r$ on that pattern's observed-data space such that $g(r\mid w)=h_r(W_{\mathrm{o}\mathrm{b}\mathrm{s}}(r))\text{almost surely}\mathrm{.}$ For a family of full-data laws, the condition is required under each law with the same response mechanism and observed-coordinate function. It is missing not at random (MNAR) when MAR fails for at least one pattern, for the declared full-data and observed-data partition. MCAR is a special case of MAR, since a constant is a function of the observed coordinates.

These conditions concern a supplied version of the conditional law; values off the full-data model's support do not affect the classification. This definition does not assert that a response mechanism is empirically true or that every joint law comes with a selected conditional kernel.

When the full-data and pattern spaces are standard Borel and a kernel must be obtained from a joint law rather than supplied as part of the model, the library's RCD existence and factor-through results provide that construction under AC: Existence of regular conditional distributions for standard borel targets and Regular conditional kernels factor through a standard borel conditioning variable. The finite examples on the companion page specify their response probabilities directly and do not use that existence argument.

Assume AC. Let $W=(W_1,\dots ,W_k)$ have a full-data model dominated by a common sigma-finite positive measure $\mu$ on the finite product measurable space $\mathcal{W}={\prod }_{j=1}^k{\mathcal{W}}_j$, with selected finite-valued density representatives $p_{\theta }$ as in Dominated statistical experiments and Likelihood densities. Let $\mathcal{R}$ be a nonempty finite pattern set, let $I_r$ and $J_r$ be the observed and missing coordinate sets for pattern $r$, and write ${\pi }_r(w)=W_{\mathrm{o}\mathrm{b}\mathrm{s}}(r)$ for its measurable coordinate projection. Let $g_{\psi }(r\mid w)$ be a supplied response probability kernel as in MCAR, MAR, and MNAR relative to an observation pattern, and define the joint model by $P_{\theta ,\psi }(dw,\{r\})=g_{\psi }(r\mid w)P_{\theta }(dw)\mathrm{.}$

Fix a finite positive measure $\rho$ equivalent to $\mu$; such a measure exists for every sigma-finite $\mu$. For each pattern put ${\kappa }_r(B)=\rho ({\pi }_r^{-1}(B))$ for measurable $B\subseteq {\mathcal{W}}_{\mathrm{o}\mathrm{b}\mathrm{s}}(r)$, and define the finite measure on the observed disjoint union by ${\lambda }_O(A)={\sum }_{r\in \mathcal{R}}{\kappa }_r(\{x:(r,x)\in A\})\mathrm{.}$ The full-data marginal law of $W_{\mathrm{o}\mathrm{b}\mathrm{s}}(r)$ and the pattern-$r$ observed sublaw $B\mapsto P_{\theta ,\psi }(R=r,{\pi }_r(W)\in B)$ are absolutely continuous with respect to ${\kappa }_r$. Choose finite-valued nonnegative Radon--Nikodym density representatives $q_{\theta ,r}$ and ${\mathrm{\ell }}_{\theta ,\psi ,r}$ for these measures. Then the observed record $O=(R,W_{\mathrm{o}\mathrm{b}\mathrm{s}}(R))$ is dominated by ${\lambda }_O$ and has density ${\mathrm{\ell }}_{\theta ,\psi }(r,x)={\mathrm{\ell }}_{\theta ,\psi ,r}(x)$ at $(r,x)$.

In the product-reference special case where $\mu =\nu ={⨂}_{j=1}^k{\nu }_j$, each ${\nu }_j$ is sigma-finite, and the coordinate reordering $w=(x,z)=(W_{\mathrm{o}\mathrm{b}\mathrm{s}}(r),W_{\mathrm{m}\mathrm{i}\mathrm{s}}(r))$ identifies $\nu$ with ${\nu }_r^{\mathrm{o}\mathrm{b}\mathrm{s}}\times {\nu }_r^{\mathrm{m}\mathrm{i}\mathrm{s}}$, where ${\nu }_r^{\mathrm{o}\mathrm{b}\mathrm{s}}={⨂}_{j\in I_r}{\nu }_j$ and ${\nu }_r^{\mathrm{m}\mathrm{i}\mathrm{s}}={⨂}_{j\in J_r}{\nu }_j$, one may instead use the pattern reference ${\nu }_r^{\mathrm{o}\mathrm{b}\mathrm{s}}$, with disjoint-union reference ${\lambda }_O^{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{d}}(A)={\sum }_r{\nu }_r^{\mathrm{o}\mathrm{b}\mathrm{s}}(\{x:(r,x)\in A\})$. The observed record is dominated by this sigma-finite measure, and Tonelli's theorem gives the density versions ${\bar{q}}_{\theta ,r}(x)=\int p_{\theta }(x,z){\nu }_r^{\mathrm{m}\mathrm{i}\mathrm{s}}(dz),{\mathrm{\ell }}_{\theta ,\psi ,r}(x)=\int p_{\theta }(x,z)g_{\psi }(r\mid x,z){\nu }_r^{\mathrm{m}\mathrm{i}\mathrm{s}}(dz)\mathrm{.}$ An empty coordinate block is a one-point space with unit mass.

Let $Y,R\in \{0,1\}$, with $R=1$ meaning that $Y$ is observed, and encode the observed record as $O=(R,RY)$. When $R=0$, the zero in the second recorded coordinate is a placeholder and does not assign the missing full-data value $Y=0$. Put $a=P(R=1),b=P(R=1,Y=1),0\le b\le a\le 1.$ Let ${\mathrm{\Theta }}_{a,b}$ be the set of all joint laws $P$ of $(Y,R)$ with these values of $a$ and $b$, and let the statistical model map $P\in {\mathrm{\Theta }}_{a,b}$ to the law of $O$. For each individual $P$, use its finite conditional response kernel specified below when assessing the MAR clause; arbitrary null-state values do not affect that classification. The full-data mean $m(P)=P(Y=1)$ is an estimand on this model. Its set of values over ${\mathrm{\Theta }}_{a,b}$ is exactly $[b,b+1-a]\mathrm{.}$ Consequently, without a MAR restriction, the mean is not identifiable when $a<1$; it is identified as $b$ when $a=1$. If MAR also holds, then for $a>0$ the mean is $b\mathrm{/}a$, while for $a=0$ it remains unidentified.

Assume AC and the hypotheses and notation of Observed-data likelihood under a response model. Suppose the full-data parameter and response-mechanism parameter range independently over the nonempty Cartesian product $\mathrm{\Theta }\times \mathrm{\Psi }$, so the full-data law is $P_{\theta }$ and the response kernel is $g_{\psi }$. For each pattern let $m_r$ denote the pattern reference selected in the lemma: $m_r={\kappa }_r$ in general, and one may take $m_r={\nu }_r^{\mathrm{o}\mathrm{b}\mathrm{s}}$ in the product-reference special case. Let $Q_{\theta ,r}$ be the law of $W_{\mathrm{o}\mathrm{b}\mathrm{s}}(r)$ under $P_{\theta }$, and choose a finite-valued nonnegative density representative $q_{\theta ,r}=d{Q}_{\theta ,r}\mathrm{/}d{m}_r$. Suppose MAR holds for every $\psi$ and $r$ with one measurable factor $h_{\psi ,r}$ valid $P_{\theta }$-almost surely for every $\theta$. Clipping this factor to $[0,1]$ preserves those almost-sure identities, so take $0\le h_{\psi ,r}(x)\le 1$ for every $x$. Then ${\mathrm{\ell }}_{\theta ,\psi }(r,x)=h_{\psi ,r}(x)q_{\theta ,r}(x)$ is a selected observed-record likelihood version relative to the disjoint union reference ${\lambda }_O(A)={\sum }_r{m}_r(\{x:(r,x)\in A\})$. In the product-reference special case, the missing-coordinate integrals ${\bar{q}}_{\theta ,r}(x)=\int p_{\theta }(x,z){\nu }_r^{\mathrm{m}\mathrm{i}\mathrm{s}}(dz),{\mathrm{\ell }}_{\theta ,\psi }(r,x)=\int p_{\theta }(x,z)g_{\psi }(r\mid x,z){\nu }_r^{\mathrm{m}\mathrm{i}\mathrm{s}}(dz)$ are density versions, and may be chosen in the displayed factorized form after selecting finite-valued representatives.

For a fixed response parameter, if $h_{\psi ,r}(x)>0$ and $q_{{\theta }_2,r}(x)>0$, the factor cancels from the likelihood ratio in $\theta$. Profiling over the same set $\mathrm{\Psi }$ for each $\theta$ multiplies $q_{\theta ,r}(x)$ by the factor ${\sup }_{\psi \in \mathrm{\Psi }}{h}_{\psi ,r}(x)$, which is independent of $\theta$; when this factor is zero the observed record has zero likelihood for every $\theta$. This licenses omission of the response factor for likelihood inference about $\theta$ on its support. It does not license omission when the target depends on $\psi$, or when the allowed parameter pairs are not a Cartesian product.

The density-class identities are for each fixed parameter pair. As in Likelihood densities, no common exceptional null set across an uncountable parameter family is asserted; the displayed pointwise likelihood claims use the selected versions stated above.