Nonregular and Misspecified Limits

Put ${\mathrm{\Delta }}_n={\sup }_{\theta \in \mathrm{\Theta }}\mathrm{\mid }{P}_n{\mathrm{\ell }}_{\theta }-M(\theta )\mathrm{\mid }$ and $r_n={\sup }_{\theta \in \mathrm{\Theta }}{P}_n{\mathrm{\ell }}_{\theta }-P_n{\mathrm{\ell }}_{{\widehat{\theta }}_n}$. By assumption ${\mathrm{\Delta }}_n{\to }_{P}0$ and $r_n{\to }_{P}0$. Since ${\widehat{\theta }}_n$ is an approximate maximizer by [F8], $P_n{\mathrm{\ell }}_{{\widehat{\theta }}_n}\ge P_n{\mathrm{\ell }}_{{\theta }^{\ast }}-r_n$, and therefore $0\le M({\theta }^{\ast })-M({\widehat{\theta }}_n)\le 2{\mathrm{\Delta }}_n+r_n$.

For any fixed $\epsilon >0$, separation gives a positive gap ${\eta }_{\epsilon }$ whenever $\mathrm{\parallel }{\widehat{\theta }}_n-{\theta }^{\ast }{\mathrm{\parallel }}_2\ge \epsilon$. Step 1.1 then gives $P(\mathrm{\parallel }{\widehat{\theta }}_n-{\theta }^{\ast }{\mathrm{\parallel }}_2\ge \epsilon )\le P(2{\mathrm{\Delta }}_n+r_n\ge {\eta }_{\epsilon })\to 0$. Thus ${\widehat{\theta }}_n-{\theta }^{\ast }=o_P(1)$ by [F6], so with probability tending to one the segment from ${\theta }^{\ast }$ to ${\widehat{\theta }}_n$ lies in any fixed neighborhood on which the score is differentiable.

Choose a ball about ${\theta }^{\ast }$ inside the differentiability neighborhood; the measurable event that ${\widehat{\theta }}_n$ lies in this ball ensures its segment from ${\theta }^{\ast }$ stays in that neighborhood. On this event define $A_n={\int }_0^1{P}_n{\mathrm{\nabla }}_{\theta }{S}_{{\theta }^{\ast }+t({\widehat{\theta }}_n-{\theta }^{\ast })}dt$; set $A_n=H$ otherwise. For each fixed parameter in a smaller ball, density measurability makes the log-density measurable, and difference quotients make the sections of $S_{\theta }$ and ${\mathrm{\nabla }}_{\theta }{S}_{\theta }$ measurable. On compact subballs, rounding parameters to rational grids gives countable-valued Borel approximations; continuity in $\theta$ makes the score and Hessian fields jointly measurable as pointwise limits. The segment integrand is then measurable, and its continuous sample paths make $A_n$ the limit of measurable Riemann sums. Given $\epsilon >0$, continuity of $D$ at ${\theta }^{\ast }$ supplies a sufficiently small fixed $\delta$ for which $\mathrm{\parallel }D(\theta )-H{\mathrm{\parallel }}_{op}<\epsilon \mathrm{/}2$ throughout the $\delta$-ball. Consistency from step 2.1 and the assumed uniform Hessian law imply $P(\mathrm{\parallel }{A}_n-H{\mathrm{\parallel }}_{op}>\epsilon )\to 0$.

On the same high-probability event, the componentwise fundamental theorem of calculus along the segment gives $P_n{S}_{{\widehat{\theta }}_n}=P_n{S}_{{\theta }^{\ast }}+A_n({\widehat{\theta }}_n-{\theta }^{\ast })$. By step 3.1, every entry of $A_n-H$ tends to zero in probability. The determinant and adjugate are polynomials in the entries, so $\det A_n{\to }_P\det H\mathrm{\ne }0$ and $\mathrm{adj}(A_n){\to }_P\mathrm{adj}(H)$; on the event $\det A_n\mathrm{\ne }0$, $A_n^{-1}=\mathrm{adj}(A_n)\mathrm{/}\det A_n$, hence every entry of $A_n^{-1}-H^{-1}$ tends to zero in probability. The finite-dimensional matrix bound $\mathrm{\parallel }B{\mathrm{\parallel }}_{op}\le d{\max }_{j,k}\mathrm{\mid }{B}_{jk}\mathrm{\mid }$ then gives $\mathrm{\parallel }{A}_n^{-1}-H^{-1}{\mathrm{\parallel }}_{op}{\to }_{P}0$. Rearranging gives $\sqrt{n}({\widehat{\theta }}_n-{\theta }^{\ast })=-A_n^{-1}\sqrt{n}{P}_n{S}_{{\theta }^{\ast }}+A_n^{-1}\sqrt{n}{P}_n{S}_{{\widehat{\theta }}_n}$ on an event whose probability tends to one.

By [F2], the score vectors $S_{{\theta }^{\ast }}(X_i)$ are IID; their mean is zero and their covariance is $J$. Their second moment is finite by the Statement, so [F4] and [A1] give $\sqrt{n}{P}_n{S}_{{\theta }^{\ast }}\Rightarrow G$ with $G\sim N_d(0,J)$. The matrix $A_n^{-1}$ converges in probability in operator norm to $H^{-1}$, and the residual vector $\sqrt{n}{P}_n{S}_{{\widehat{\theta }}_n}=o_P(1)$ by the Statement. Flattening the matrix entries and concatenating the vectors gives a finite-dimensional vector converging in probability to $(H^{-1},0)$. Applying [F5] to the continuous map $(x,A,r)\mapsto -Ax+Ar$ gives the limit $-H^{-1}G$. Its covariance is $H^{-1}J{H}^{-T}$, proving the assertion.

On the unit sphere $S^{d-1}$ of [F2], the function $u\mapsto u^{T}Iu$ is continuous and strictly positive. The sphere is nonempty, closed and bounded, hence compact by [F4]; [F5] gives a minimum $\lambda >0$, since a zero minimum would be attained at a unit vector and contradict positive definiteness. Rescaling any nonzero $h$ to the unit sphere gives $h^{T}Ih\ge \lambda \mathrm{\parallel }h{\mathrm{\parallel }}_2^2$, also valid for $h=0$.

Completing the square gives $q_z(h)=\frac{1}{2}{z}^T{I}^{-1}z-\frac{1}{2}(h-I^{-1}z{)}^{T}I(h-I^{-1}z)$. By [F1] and [F6], $q_z(h)\le \mathrm{\parallel }z{\mathrm{\parallel }}_2\mathrm{\parallel }h{\mathrm{\parallel }}_2-\frac{1}{2}\lambda \mathrm{\parallel }h{\mathrm{\parallel }}_2^2$, which tends to $-\mathrm{\infty }$ as $\mathrm{\parallel }h{\mathrm{\parallel }}_2\to \mathrm{\infty }$.

Choose $c_0\in C$. By step 1.2 there is $R>\mathrm{\parallel }{c}_0{\mathrm{\parallel }}_2$ such that $q_z(h)<q_z(c_0)$ whenever $\mathrm{\parallel }h{\mathrm{\parallel }}_2\ge R$. The set $C\cap {\bar{B}}_2(0,R)$ is nonempty, closed and bounded, hence compact by [F2]–[F4]. The continuous quadratic $q_z$ attains a maximum there by [F5]; values outside are below $q_z(c_0)$, so this is a global maximum on $C$.

The identity of step 1.2 says that maximizing $q_z$ over $C$ is equivalent to minimizing $(h-I^{-1}z{)}^{T}I(h-I^{-1}z)$. Step 2.1 proves that this minimum is attained; by [F7] it equals $d_I^2(I^{-1}z,C)$. Substitution gives the stated supremum formula.

With the supplied QMD information $I$ of [F1], write $q_z(h)=h^{T}z-\frac{1}{2}{h}^{T}Ih$ and $m_A(z)={\sup }_{h\in A}{q}_z(h)$ for $A=C,C_0$. By [F6], $m_A(z)=\frac{1}{2}{z}^T{I}^{-1}z-\frac{1}{2}{d}_I^2(I^{-1}z,A)$. If $h_A(z)$ is a maximizer, then $q_z(h_A(z))\ge q_z(0)=0$. The lower bound in [F6] and [F5] imply $\mathrm{\parallel }{h}_A(z){\mathrm{\parallel }}_2\le 2\mathrm{\parallel }z{\mathrm{\parallel }}_2\mathrm{/}\lambda$ (with the bound immediate when $h_A(z)=0$).

Let $\lambda$ be from [F6], fix $M>0$, and choose $R>2M\mathrm{/}\lambda +1$. For $A_n=C_n$, $A=C$ (and likewise for $A_n=C_{0n}$, $A=C_0$), write $m_{A_n,R}(z)={\sup }_{h\in A_n\cap {\bar{B}}_2(0,R)}{q}_z(h)$. For every $\mathrm{\parallel }z{\mathrm{\parallel }}_2\le M$, [F6] gives a maximizer $h(z)$ of $q_z$ on $A$ with $\mathrm{\parallel }h(z){\mathrm{\parallel }}_2\le 2M\mathrm{/}\lambda <R-1$. Local set convergence supplies points of $A_n$ within ${\delta }_{n,R-1}+1\mathrm{/}n$ of $h(z)$, hence inside the radius-$R$ ball for all large $n$. Conversely, every point of $A_n\cap {\bar{B}}_2(0,R)$ is within ${\delta }_{n,R}+1\mathrm{/}n$ of a point of $A$. With $B_I={\sum }_{j,k=1}^d\mathrm{\mid }{I}_{jk}\mathrm{\mid }$, [F3] and [F5] give $\mathrm{\mid }{q}_z(h)-q_z(h^{\mathrm{\prime }})\mathrm{\mid }\le (M+B_I(R+1))\mathrm{\parallel }h-h^{\mathrm{\prime }}{\mathrm{\parallel }}_2$ for $\mathrm{\parallel }h{\mathrm{\parallel }}_2,\mathrm{\parallel }{h}^{\mathrm{\prime }}{\mathrm{\parallel }}_2\le R+1$ and $\mathrm{\parallel }z{\mathrm{\parallel }}_2\le M$. These two directions show ${\sup }_{\mathrm{\parallel }z{\mathrm{\parallel }}_2\le M}\mathrm{\mid }{m}_{A_n,R}(z)-m_A(z)\mathrm{\mid }\to 0$. The same bounds apply to both full and null sets.

The sequence $Z_n$ is tight. Indeed, for any $\eta >0$ choose $R$ with $P(\mathrm{\parallel }Z{\mathrm{\parallel }}_2\ge R)<\eta$, possible because [F7] supplies a Borel probability law on finite-dimensional ${\mathbb{R}}^d$. The set $\{z:\mathrm{\parallel }z{\mathrm{\parallel }}_2\ge R\}$ is closed, so [F8] bounds ${\mathrm{lim\hspace{0.17em}sup}}_{n}P(\mathrm{\parallel }{Z}_n{\mathrm{\parallel }}_2\ge R)$ by $\eta$. step 1.1 then makes the maximizers of $q_{Z_n}$ over $C$ and $C_0$ tight as well.

For $A_n=C_n$ or $C_{0n}$, put $L_{A_n,R}:={\sup }_{h\in A_n\cap {\bar{B}}_2(0,R)}\{{\mathrm{\ell }}_n({\theta }_0+h\mathrm{/}\sqrt{n})-{\mathrm{\ell }}_n({\theta }_0)\}$. The uniform expansion gives $\mathrm{\mid }{L}_{A_n,R}-m_{A_n,R}(Z_n)\mathrm{\mid }\le {\sup }_{h\in C_n,\mathrm{\parallel }h{\mathrm{\parallel }}_2\le R}\mathrm{\mid }{r}_n(h)\mathrm{\mid }{\to }_{P}0$ for each fixed $R$. For fixed $M>0$ and $R>2M\mathrm{/}\lambda +1$, step 1.2 therefore gives, for every $\epsilon >0$, $P(\mathrm{\mid }{L}_{A_n,R}-m_A(Z_n)\mathrm{\mid }>\epsilon ,\mathrm{\parallel }{Z}_n{\mathrm{\parallel }}_2\le M)\to 0$, with $A=C$ or $C_0$, respectively.

Given $\eta >0$, tightness of $Z_n$ lets us choose $M$ so that $\mathrm{\parallel }{Z}_n{\mathrm{\parallel }}_2\le M$ with limiting probability at least $1-\eta \mathrm{/}3$. Choose $R>2M\mathrm{/}\lambda +1$ large enough that the joint event that both local likelihood maximizers lie in the radius-$R$ ball has limiting probability at least $1-\eta \mathrm{/}3$. By step 1.1 both quadratic maximizers also lie in that ball whenever $\mathrm{\parallel }{Z}_n{\mathrm{\parallel }}_2\le M$. Thus all four full and truncated suprema agree except on events with total limiting upper probability at most $2\eta \mathrm{/}3$. step 2.2 and arbitrariness of $\eta$ give $T_n-2(m_C(Z_n)-m_{C_0}(Z_n)){\to }_{P}0$.

The form $⟨u,v{⟩}_I=u^{T}Iv$ is an inner product; its Cauchy–Schwarz inequality follows by requiring $(u-tv{)}^{T}I(u-tv)\ge 0$ for every real $t$. Thus the induced norm satisfies the triangle inequality, and taking infima gives $\mathrm{\mid }{d}_I(x,C)-d_I(y,C)\mathrm{\mid }\le \mathrm{\parallel }x-y{\mathrm{\parallel }}_I$. Also $v^{T}Iv\le B_I\mathrm{\parallel }v{\mathrm{\parallel }}_2^2$, where $B_I$ is the finite entry sum from step 1.2, so the distance map is Euclidean Lipschitz after composition with the fixed linear map $I^{-1}$. Therefore $g(z):=d_I^2(I^{-1}z,C_0)-d_I^2(I^{-1}z,C)$ is continuous. The formula in step 1.1 gives $2[m_C(z)-m_{C_0}(z)]=g(z)$. By [F9], $g(Z_n)\Rightarrow g(Z)$. step 3.1 and [F10] then give $T_n\Rightarrow g(Z)$, which is the stated distance-difference law.

On the probability-one event $0<X_{(n)}\le \theta$, the likelihood at a candidate $u>0$ is $L_n(u)={\prod }_{i=1}^n{p}_u(X_i)=u^{-n}\mathbf{1}\{X_{(n)}\le u\}$. It is zero for $u<X_{(n)}$, while for $u\ge X_{(n)}$ it equals $u^{-n}\le X_{(n)}^{-n}=L_n(X_{(n)})$, with strict inequality if $u>X_{(n)}$. Thus $X_{(n)}$ maximizes the likelihood almost surely; on the null event $X_{(n)}=0$, use the arbitrary positive assignment specified above.

For $0\le t<n\theta$, independence and the uniform CDF give $P_{\theta }\{n(\theta -X_{(n)})>t\}=P_{\theta }\{X_i<\theta -t\mathrm{/}n\text{ for every }i\}=(1-t\mathrm{/}(n\theta ){)}^n$. For fixed $t\ge 0$ this formula applies for all sufficiently large $n$, and its limit is $e^{-t\mathrm{/}\theta }$.

For every $h>0$ small enough that $\theta +h>0$, the gained support interval $(\theta ,\theta +h]$ has $p_{\theta }=0$ and $p_{\theta +h}=1\mathrm{/}(\theta +h)$. If the model were QMD at $\theta$, the score term in [F2] would vanish on this interval and the squared remainder integral would be at least ${\int }_{\theta }^{\theta +h}{p}_{\theta +h}(x)dx=h\mathrm{/}(\theta +h)$. But $[h\mathrm{/}(\theta +h)]\mathrm{/}{h}^2=1\mathrm{/}[h(\theta +h)]\to \mathrm{\infty }$, contradicting the required $o(h^2)$ remainder. Thus the model is not QMD at $\theta$.

Therefore the nonnegative variable $n(\theta -X_{(n)})$ converges in distribution to the exponential law with rate $1\mathrm{/}\theta$, whose CDF is $0$ for $t<0$ and $1-e^{-t\mathrm{/}\theta }$ for $t\ge 0$. The limiting CDF is continuous, so [F3] applies at every real $t$.

For any $\epsilon >0$ and all $n>{\epsilon }^2\mathrm{/}{\theta }^2$, the exact maximum formula gives $P_{\theta }\{\sqrt{n}(\theta -X_{(n)})>\epsilon \}=(1-\epsilon \mathrm{/}(\theta \sqrt{n}){)}^n\to 0$. Thus $\sqrt{n}(\theta -X_{(n)}){\to }_{P}0$ by [F4], refuting the stated nonzero root-$n$ fluctuation and confirming the faster $n$ rate in step 2.1.