M-estimators

Let $P$ be a probability measure on a measurable observation space $(\mathcal{X},\mathcal{A})$, and let $X_1,X_2,\dots$ be IID with law $P$; IID means a common law and mutual independence of the whole family (Probability measures and probability spaces, Identical distribution and IID families). Let $(\mathrm{\Theta },d)$ be a metric candidate space with its Borel sigma-algebra (Metric space: $d(x,y)=0$ iff $x=y$, symmetry, and the triangle inequality; pseudometric and ultrametric, The Borel sigma-algebra of a topological space). For each $\theta \in \mathrm{\Theta }$, let $m_{\theta }:\mathcal{X}\to \mathbb{R}$ be measurable and assume $P\mathrm{\mid }{m}_{\theta }\mathrm{\mid }<\mathrm{\infty }$. Define $P_n{m}_{\theta }=\frac{1}{n}{\sum }_{i=1}^n{m}_{\theta }(X_i),P{m}_{\theta }=\int m_{\theta }dP\mathrm{.}$

A measurable estimator sequence ${\widehat{\theta }}_n$ into the Borel target $\mathrm{\Theta }$ is an M-estimator if it maximizes or minimizes $\theta \mapsto P_n{m}_{\theta }$; measurability is as in Estimator sequences in IID experiments. It is an exact maximizer when $P_n{m}_{{\widehat{\theta }}_n}={\sup }_{\theta \in \mathrm{\Theta }}{P}_n{m}_{\theta }$, and an $o_P(1)$-approximate maximizer when the finite measurable gap ${\sup }_{\theta \in \mathrm{\Theta }}{P}_n{m}_{\theta }-P_n{m}_{{\widehat{\theta }}_n}$ is $o_P(1)$. For minimization use the gap $P_n{m}_{{\widehat{\theta }}_n}-{\inf }_{\theta \in \mathrm{\Theta }}{P}_n{m}_{\theta }$. The notation $o_P(1)$ is that of Stochastic order and convergence notation for finite-dimensional random vectors.

The maximizing and minimizing forms are both included; replacing $m_{\theta }$ by $-m_{\theta }$ converts one to the other. This definition does not assert that a maximizer exists, nor that the population criterion $P{m}_{\theta }$ has a unique maximizer. Whenever a random candidate is substituted into the empirical or population criterion, assume that evaluation is measurable and finite.