M- and Z-Estimators — Examples

Write ${\bar{x}}_n=n^{-1}{\sum }_i{x}_i$. Expanding both sides gives ${\sum }_i(x_i-\theta {)}^2={\sum }_i(x_i-{\bar{x}}_n{)}^2+n(\theta -{\bar{x}}_n{)}^2$, because ${\sum }_i(x_i-{\bar{x}}_n)=0$. Thus $P_n{m}_{\theta }=P_n{m}_{{\bar{x}}_n}-(\theta -{\bar{x}}_n{)}^2$, which is uniquely maximized at ${\bar{x}}_n$.

Let $\mu =PX$. Expanding $X-\theta =(X-\mu )+(\mu -\theta )$ and using $P(X-\mu )=0$ gives $P{m}_{\theta }=-P[(X-\mu {)}^2]-(\theta -\mu {)}^2$. The moment assumptions make the mean and centered second moment finite, so this is uniquely maximized at $\mu$.

The IID sequence in [F4] gives the repeated-sampling context. If every coordinate of $x,y\in {\mathbb{R}}^n$ differs by less than $\delta$, then $\mathrm{\mid }{\bar{x}}_n(x)-{\bar{x}}_n(y)\mathrm{\mid }\le n^{-1}{\sum }_i\mathrm{\mid }{x}_i-y_i\mathrm{\mid }<\delta$. Hence the sample-mean map is continuous for the product topology. Inverse images of open sets are open, so the map is Borel measurable under [A1]; it is an estimator sequence by [F2] and is an exact M-estimator by [F1] and step 1.1. The target metric is the usual metric in [F3].

The IID data context is as in [F4], and the candidate set is a metric space by [F3]. Each order statistic has the finite min-max representation $x_{(j)}={\min }_{S\subseteq \{1,\dots ,n\},\mathrm{\mid }S\mathrm{\mid }=j}{\max }_{i\in S}{x}_i$: every j-element subset contains a value at least $x_{(j)}$, while the subset of the first j ordered entries has maximum $x_{(j)}$. Coordinate projections are continuous. For finite vectors $a,b$, the maxima differ by at most ${\max }_i\mathrm{\mid }{a}_i-b_i\mathrm{\mid }$: each $a_i\le b_i+\delta$ for $\delta ={\max }_i\mathrm{\mid }{a}_i-b_i\mathrm{\mid }$, and the reverse inequality follows by swapping $a,b$; minima satisfy the same bound by negating the vectors. Thus finite maxima and minima, each order statistic, and the stated middle-point selection are continuous, hence Borel measurable under [A1]. The selection is an estimator sequence by [F2].

Put $Q(\theta )={\sum }_i\mathrm{\mid }{x}_i-\theta \mathrm{\mid }$. For $n=2k+1$ and $\theta <x_{(k+1)}$, let $\delta =x_{(k+1)}-\theta >0$. Moving to $x_{(k+1)}$ decreases each of the k+1 distances for indices $i\ge k+1$ by exactly $\delta$; each of the other k distances can increase by at most $\delta$ by the triangle inequality. Therefore $Q(x_{(k+1)})-Q(\theta )\le k\delta -(k+1)\delta =-\delta <0$. For $\theta >x_{(k+1)}$, the k+1 distances for indices $i\le k+1$ decrease by $\delta =\theta -x_{(k+1)}$, while each of the other k distances can increase by at most $\delta$, giving the same strict inequality. Thus the odd-sample median uniquely minimizes $Q$.

For $n=2k$ and $\theta \in [x_{(k)},x_{(k+1)}]$, the first k observations lie at or below $\theta$ and the last k lie at or above it, so $Q(\theta )=k\theta -{\sum }_{i=1}^k{x}_{(i)}+{\sum }_{i=k+1}^{2k}{x}_{(i)}-k\theta$ is constant on this interval. If $\theta <x_{(k)}$, moving to $x_{(k)}$ decreases the k+1 distances for $i\ge k$ by $\delta =x_{(k)}-\theta$, while the other k-1 distances can increase by at most $\delta$; hence $Q(x_{(k)})-Q(\theta )\le (k-1)\delta -(k+1)\delta =-2\delta <0$. If $\theta >x_{(k+1)}$, moving to $x_{(k+1)}$ decreases the k+1 distances for $i\le k+1$ by $\delta =\theta -x_{(k+1)}$, while the other k-1 distances can increase by at most $\delta$, so $Q(x_{(k+1)})-Q(\theta )\le -2\delta <0$. Therefore the interval is exactly the minimizing set of $Q$ and the maximizing set of $P_n{m}_{\theta }$ by [F1].

The population criterion is $M(0)=0$ and $M(1)=E{X}_1=p>0$, so its unique maximum is ${\theta }_0=1$. The empirical criteria are $M_n(0)=0$ and $M_n(1)={\bar{X}}_n\ge 0$; the displayed selection therefore attains their maximum for every sample. By [F5], the sample space is $\{0,1{\}}^n$ with its product sigma-algebra. Each singleton vector is a measurable rectangle, and every subset is a finite union of singletons, so every data map is measurable. The estimator is exact by [F1].

Independence and the Bernoulli finite sample law imply that expectations of products of functions of distinct coordinates factor: the joint mass at each $(x_1,\dots ,x_n)\in \{0,1{\}}^n$ is ${\prod }_{i}P(X_i=x_i)$, and summing this product over the finite sample space factors. Expanding $({\sum }_i{Y}_i{)}^4$, every term with an index occurring once has expectation zero by $E{Y}_i=0$ and this factorization; the remaining terms are ${\sum }_{i}E{Y}_i^4+6{\sum }_{i<j}E{Y}_i^{2}E{Y}_j^2\le n+6\left(\genfrac{}{}{0px}{}{n}{2}\right)\le 3n^2$. For $Z\ge 0$ and $t>0$, $t{\mathbf{1}}_{\{Z>t\}}\le Z$, so integrating gives $P(Z>t)\le EZ\mathrm{/}t$. Applying this to $Z=({\sum }_i{Y}_i{)}^4$ gives $P(\mathrm{\mid }{\bar{X}}_n-p\mathrm{\mid }>\epsilon )\le 3\mathrm{/}(n^2{\epsilon }^4)$.

For fixed $\epsilon >0$, the bound in step 1.2 is summable in $n$. By the union bound, $P({\bigcup }_{n\ge N}\{\mathrm{\mid }{\bar{X}}_n-p\mathrm{\mid }>\epsilon \})\le {\sum }_{n\ge N}3\mathrm{/}(n^2{\epsilon }^4)\to 0$; hence only finitely many such deviations occur almost surely. Applying this to $\epsilon =1\mathrm{/}j$ for every positive integer $j$ gives ${\bar{X}}_n\to p$ almost surely.

For $0<\epsilon <1$, the metric-error event is $\{{\widehat{\theta }}_n=0\}=\{{\bar{X}}_n=0\}$, which is contained in $\{\mathrm{\mid }{\bar{X}}_n-p\mathrm{\mid }>p\mathrm{/}2\}$. Step 1.2 bounds its probability by $3\mathrm{/}(n^2(p\mathrm{/}2{)}^4)\to 0$; for $\epsilon \ge 1$ the error event is empty. Hence ${\widehat{\theta }}_n$ is consistent by [F3] in the metric of [F4]. Also, since $p>0$, step 2.1 implies that ${\widehat{\theta }}_n=1$ eventually almost surely, and ${\sup }_{\theta \in \mathrm{\Theta }}\mathrm{\mid }{M}_n(\theta )-M(\theta )\mathrm{\mid }=\mathrm{\mid }{\bar{X}}_n-p\mathrm{\mid }\to 0$ almost surely, displaying the finite-class uniform law.

By [A1] and [F4], every observation equals $0$ almost surely and the sequence is IID. Thus $P_n{m}_0=P_n{m}_1=P{m}_0=P{m}_1=0$ for every $n$. Both parameters are population maximizers, and every data-measurable choice is an exact M-estimator by [F1]. The uniform error is identically zero, but the population maximum is not separated from the other parameter.

Choose the deterministic estimator ${\widehat{\theta }}_n=1$ for odd $n$ and ${\widehat{\theta }}_n=0$ for even $n$. For each $n$ it is a constant function of the data, so every inverse image is empty or the full sample space and it is measurable; by step 1.1 it is exact. For the fixed population maximizer ${\theta }_0=0$ and $\epsilon =1\mathrm{/}2$, $P\{d({\widehat{\theta }}_n,{\theta }_0)>\epsilon \}$ equals $1$ for odd $n$ and $0$ for even $n$, so the error probability has no zero limit.

This exact M-estimator is not consistent for ${\theta }_0=0$ by [F2]. Thus the unseparated population maximum does not guarantee consistency, even when the empirical criterion equals its population version exactly.