M- and Z-Estimators

Let $P$ be a probability measure on a measurable observation space $(\mathcal{X},\mathcal{A})$, and let $X_1,X_2,\dots$ be IID with law $P$; IID means a common law and mutual independence of the whole family (Probability measures and probability spaces, Identical distribution and IID families). Let $(\mathrm{\Theta },d)$ be a metric candidate space with its Borel sigma-algebra (Metric space: $d(x,y)=0$ iff $x=y$, symmetry, and the triangle inequality; pseudometric and ultrametric, The Borel sigma-algebra of a topological space). For each $\theta \in \mathrm{\Theta }$, let $m_{\theta }:\mathcal{X}\to \mathbb{R}$ be measurable and assume $P\mathrm{\mid }{m}_{\theta }\mathrm{\mid }<\mathrm{\infty }$. Define $P_n{m}_{\theta }=\frac{1}{n}{\sum }_{i=1}^n{m}_{\theta }(X_i),P{m}_{\theta }=\int m_{\theta }dP\mathrm{.}$

A measurable estimator sequence ${\widehat{\theta }}_n$ into the Borel target $\mathrm{\Theta }$ is an M-estimator if it maximizes or minimizes $\theta \mapsto P_n{m}_{\theta }$; measurability is as in Estimator sequences in IID experiments. It is an exact maximizer when $P_n{m}_{{\widehat{\theta }}_n}={\sup }_{\theta \in \mathrm{\Theta }}{P}_n{m}_{\theta }$, and an $o_P(1)$-approximate maximizer when the finite measurable gap ${\sup }_{\theta \in \mathrm{\Theta }}{P}_n{m}_{\theta }-P_n{m}_{{\widehat{\theta }}_n}$ is $o_P(1)$. For minimization use the gap $P_n{m}_{{\widehat{\theta }}_n}-{\inf }_{\theta \in \mathrm{\Theta }}{P}_n{m}_{\theta }$. The notation $o_P(1)$ is that of Stochastic order and convergence notation for finite-dimensional random vectors.

The maximizing and minimizing forms are both included; replacing $m_{\theta }$ by $-m_{\theta }$ converts one to the other. This definition does not assert that a maximizer exists, nor that the population criterion $P{m}_{\theta }$ has a unique maximizer. Whenever a random candidate is substituted into the empirical or population criterion, assume that evaluation is measurable and finite.

Let $P$ be a probability measure on $(\mathcal{X},\mathcal{A})$, let $X_1,X_2,\dots$ be IID with law $P$ (Probability measures and probability spaces, Identical distribution and IID families), and let $(\mathrm{\Theta },d)$ be a metric candidate space with its Borel sigma-algebra (Metric space: $d(x,y)=0$ iff $x=y$, symmetry, and the triangle inequality; pseudometric and ultrametric, The Borel sigma-algebra of a topological space). Fix a finite integer $q\ge 1$ and a norm on ${\mathbb{R}}^q$. For each $\theta \in \mathrm{\Theta }$, let ${\psi }_{\theta }:\mathcal{X}\to {\mathbb{R}}^q$ be measurable with $P\mathrm{\parallel }{\psi }_{\theta }\mathrm{\parallel }<\mathrm{\infty }$. Define ${\mathrm{\Psi }}_n(\theta )=P_n{\psi }_{\theta }=\frac{1}{n}{\sum }_{i=1}^n{\psi }_{\theta }(X_i),\mathrm{\Psi }(\theta )=P{\psi }_{\theta }\mathrm{.}$

A measurable estimator sequence ${\widehat{\theta }}_n$ into the Borel target $\mathrm{\Theta }$ is a Z-estimator if it is an exact or $o_P(1)$-approximate root: ${\mathrm{\Psi }}_n({\widehat{\theta }}_n)=0$ for every $n$, or $\mathrm{\parallel }{\mathrm{\Psi }}_n({\widehat{\theta }}_n)\mathrm{\parallel }=o_P(1)$. Measurability is as in Estimator sequences in IID experiments, and $o_P(1)$ has the convention of Stochastic order and convergence notation for finite-dimensional random vectors. Whenever a random candidate is substituted into an estimating equation, assume the evaluation is measurable.

Because $q\ge 1$ is finite, For $n\ge 1$ all norms on ${\mathbb{R}}^n$ are equivalent gives constants $c,C>0$ with $c\mathrm{\parallel }v{\mathrm{\parallel }}_2\le \mathrm{\parallel }v\mathrm{\parallel }\le C\mathrm{\parallel }v{\mathrm{\parallel }}_2$ for every $v\in {\mathbb{R}}^q$. Thus for every ${\mathbb{R}}^q$-valued sequence, convergence of the chosen-norm tails to zero is equivalent to the Euclidean $o_P(1)$ convention in Stochastic order and convergence notation for finite-dimensional random vectors.

This definition does not infer a Z-estimating equation from an M-criterion. Differentiating an M-criterion is justified only under additional differentiability and interior-point hypotheses; a nonsmooth criterion need not have a score equation.

Let $(\mathrm{\Theta },d)$ be a metric space, let $X_1,\dots ,X_n$ be IID with law $P$, and let $m_{\theta }$ be a jointly measurable, $P$-integrable real criterion for every $\theta \in \mathrm{\Theta }$. Write $M_n(\theta )=P_n{m}_{\theta }$ and $M(\theta )=P{m}_{\theta }$. Fix ${\theta }_0\in \mathrm{\Theta }$. Assume the displayed suprema and estimator evaluations below are measurable and finite where used. In the maximization case, suppose

1. $U_n={\sup }_{\theta \in \mathrm{\Theta }}\mathrm{\mid }{M}_n(\theta )-M(\theta )\mathrm{\mid }=o_P(1)$;

2. for every $\epsilon >0$, ${\gamma }_{\epsilon }=M({\theta }_0)-\sup \{M(\theta ):d(\theta ,{\theta }_0)\ge \epsilon \}>0$, with $\sup \mathrm{\varnothing }=-\mathrm{\infty }$;

3. the measurable estimator ${\widehat{\theta }}_n$ satisfies $M_n({\widehat{\theta }}_n)\ge M_n({\theta }_0)-{\rho }_n$ for a nonnegative ${\rho }_n=o_P(1)$.

Then ${\widehat{\theta }}_n$ is consistent in probability for ${\theta }_0$, meaning $P^{\otimes n}\{d({\widehat{\theta }}_n,{\theta }_0)>\epsilon \}\to 0$ for every $\epsilon >0$ (Pointwise consistency in probability). The same conclusion holds for a minimizing estimator if, for every $\epsilon >0$, $\inf \{M(\theta ):d(\theta ,{\theta }_0)\ge \epsilon \}-M({\theta }_0)>0$ and $M_n({\widehat{\theta }}_n)\le M_n({\theta }_0)+{\sigma }_n$ for a nonnegative measurable ${\sigma }_n=o_P(1)$. The relative criterion-gap premise in the maximizing case is implied by the global exact or $o_P(1)$-approximate M-estimator definition, as proved in the final proof step.

The uniform-convergence premise is an explicit hypothesis. This theorem does not replace it with pointwise convergence or a pointwise law of large numbers.

Let $(\mathrm{\Theta },d)$ be a metric space, let $X_1,\dots ,X_n$ be IID with law $P$, and let ${\psi }_{\theta }$ be jointly measurable and integrable as an ${\mathbb{R}}^q$-valued function for a fixed finite $q\ge 1$. Write ${\mathrm{\Psi }}_n(\theta )=P_n{\psi }_{\theta }$ and $\mathrm{\Psi }(\theta )=P{\psi }_{\theta }$, using a fixed norm on ${\mathbb{R}}^q$. Fix ${\theta }_0\in \mathrm{\Theta }$. Assume the displayed suprema and estimator evaluations are measurable. If

1. ${\sup }_{\theta \in \mathrm{\Theta }}\mathrm{\parallel }{\mathrm{\Psi }}_n(\theta )-\mathrm{\Psi }(\theta )\mathrm{\parallel }=o_P(1)$;

2. $\mathrm{\Psi }({\theta }_0)=0$ and, for every $\epsilon >0$, $b_{\epsilon }=\inf \{\mathrm{\parallel }\mathrm{\Psi }(\theta )\mathrm{\parallel }:d(\theta ,{\theta }_0)\ge \epsilon \}>0$, with $\inf \mathrm{\varnothing }=+\mathrm{\infty }$;

3. $\mathrm{\parallel }{\mathrm{\Psi }}_n({\widehat{\theta }}_n)\mathrm{\parallel }=o_P(1)$ for a measurable estimator ${\widehat{\theta }}_n$,

then ${\widehat{\theta }}_n$ is consistent in probability for ${\theta }_0$.