Asymptotic linearity and normality of a regular MLE

Write $P_0=P_{{\theta }_0}$, $S_0=\{p_{{\theta }_0}>0\}$, and $r_t=\sqrt{p_{{\theta }_0+t}}-\sqrt{p_{{\theta }_0}}-\frac{1}{2}{t}^T{s}_{{\theta }_0}\sqrt{p_{{\theta }_0}}$. By [F3], $\mathrm{\parallel }{r}_t{\mathrm{\parallel }}_{L^2(\mu )}=o(\mathrm{\parallel }t\mathrm{\parallel })$, and [F12] gives ${\mathbb{E}}_0{s}_{{\theta }_0}=0$. On $S_0^c$ both $\sqrt{p_{{\theta }_0}}$ and $s_{{\theta }_0}\sqrt{p_{{\theta }_0}}$ vanish, so the probability mass ${\alpha }_t={\int }_{S_0^c}{p}_{{\theta }_0+t}d\mu$ is $o(\mathrm{\parallel }t{\mathrm{\parallel }}^2)$. Expanding the QMD approximation in [F3] and applying [F9] to its cross term gives $\mathrm{\parallel }\sqrt{p_{{\theta }_0+t}}-\sqrt{p_{{\theta }_0}}{\mathrm{\parallel }}_2^2=\frac{1}{4}{t}^{T}It+o(\mathrm{\parallel }t{\mathrm{\parallel }}^2)$.

Normalize the absolutely continuous part on $S_0$ by $q_t=p_{{\theta }_0+t}{\mathbf{1}}_{S_0}\mathrm{/}(1-{\alpha }_t)$, and put $a_t=\sqrt{q_t\mathrm{/}{p}_{{\theta }_0}}-1$ on $S_0$. Since ${\alpha }_t=o(\mathrm{\parallel }t{\mathrm{\parallel }}^2)$, [F3] gives $a_t=\frac{1}{2}{t}^T{s}_{{\theta }_0}+o_{L^2(P_0)}(\mathrm{\parallel }t\mathrm{\parallel })$, so ${\mathbb{E}}_0{a}_t^2=\frac{1}{4}{t}^{T}It+o(\mathrm{\parallel }t{\mathrm{\parallel }}^2)$ and $2{\mathbb{E}}_0{a}_t+{\mathbb{E}}_0{a}_t^2=0$ from ${\mathbb{E}}_0(1+a_t{)}^2=1$. The local envelope makes $f_t=\log (p_{{\theta }_0+t}\mathrm{/}{p}_{{\theta }_0})$ square-integrable on $S_0$ and $\mathrm{\parallel }{f}_t{\mathrm{\parallel }}_2=O(\mathrm{\parallel }t\mathrm{\parallel })$. Along every sequence $t\to 0$ with $t\mathrm{/}\mathrm{\parallel }t\mathrm{\parallel }\to u$, the $L^2$ convergence of $a_t\mathrm{/}\mathrm{\parallel }t\mathrm{\parallel }$ has an almost-everywhere convergent subsequence. On it, $2\log (1+a_t)\mathrm{/}\mathrm{\parallel }t\mathrm{\parallel }\to u^T{s}_{{\theta }_0}$ pointwise. Since $\mathrm{\mid }{f}_t\mathrm{\mid }\mathrm{/}\mathrm{\parallel }t\mathrm{\parallel }\le L$ and $s_{{\theta }_0}\in L^2(P_0)$, $\mathrm{\mid }{f}_t\mathrm{/}\mathrm{\parallel }t\mathrm{\parallel }-u^T{s}_{{\theta }_0}{\mathrm{\mid }}^2\le 2L^2+2\mathrm{\parallel }{s}_{{\theta }_0}{\mathrm{\parallel }}^2$; dominated convergence [F10] gives convergence in $L^2$ along that subsequence. Every subsequence has such a subsubsequence, so convergence holds for the full sequence. Sequential compactness of the unit sphere therefore yields $f_t=t^T{s}_{{\theta }_0}+o_{L^2(P_0)}(\mathrm{\parallel }t\mathrm{\parallel })$. Moreover, the squares $a_t^2\mathrm{/}\mathrm{\parallel }t{\mathrm{\parallel }}^2$ are uniformly integrable because $a_t\mathrm{/}\mathrm{\parallel }t\mathrm{\parallel }$ converges in $L^2$ along every convergent direction. Taking the logarithm expansion in expectation requires a tail check. Fix $\epsilon >0$. On $\mathrm{\mid }{a}_t\mathrm{\mid }\le \epsilon$, the Taylor remainder $2\log (1+a_t)-2a_t+a_t^2$ is bounded by $\omega (\epsilon )a_t^2$, where $\omega (\epsilon )\to 0$. Uniform integrability gives ${\mathbb{E}}_0[a_t^2{\mathbf{1}}_{\mathrm{\mid }{a}_t\mathrm{\mid }>\epsilon }]=o(\mathrm{\parallel }t{\mathrm{\parallel }}^2)$ and therefore $P_0(\mathrm{\mid }{a}_t\mathrm{\mid }>\epsilon )=o(\mathrm{\parallel }t{\mathrm{\parallel }}^2)$. On this tail, $\mathrm{\mid }2\log (1+a_t)\mathrm{\mid }\le \mathrm{\mid }{f}_t\mathrm{\mid }+\mathrm{\mid }\log (1-{\alpha }_t)\mathrm{\mid }$; its expectation is $o(\mathrm{\parallel }t{\mathrm{\parallel }}^2)$ by the envelope, Cauchy–Schwarz [F9], and the tail probability. The tail expectations of $\mathrm{\mid }{a}_t\mathrm{\mid }$ and $a_t^2$ are also $o(\mathrm{\parallel }t{\mathrm{\parallel }}^2)$ by [F9] and uniform integrability. Letting $\epsilon \downarrow 0$ proves that the expected Taylor remainder is $o(\mathrm{\parallel }t{\mathrm{\parallel }}^2)$. Taking expectations then gives ${\mathbb{E}}_0{f}_t=2{\mathbb{E}}_0{a}_t-{\mathbb{E}}_0{a}_t^2+o(\mathrm{\parallel }t{\mathrm{\parallel }}^2)=-\frac{1}{2}{t}^{T}It+o(\mathrm{\parallel }t{\mathrm{\parallel }}^2);$ the normalization term $\log (1-{\alpha }_t)$ is $o(\mathrm{\parallel }t{\mathrm{\parallel }}^2)$.

The squared Hellinger distance $H^2(P_{{\theta }_0+t},P_0)=\int (\sqrt{p_{{\theta }_0+t}}-\sqrt{p_{{\theta }_0}}{)}^{2}d\mu$ equals $\frac{1}{4}{t}^{T}It+o(\mathrm{\parallel }t{\mathrm{\parallel }}^2)$ by step 1.1. The affinity ${\rho }_t=\int \sqrt{p_{{\theta }_0+t}{p}_{{\theta }_0}}d\mu =1-H^2\mathrm{/}2$. On $S_0$ let $Y=\sqrt{p_{{\theta }_0+t}\mathrm{/}{p}_{{\theta }_0}}$, so ${\mathbb{E}}_{0}Y={\rho }_t$. For $y>0$, $\log y\le \log {\rho }_t+(y-{\rho }_t)\mathrm{/}{\rho }_t$ follows from $z-1-\log z\ge 0$ for $z=y\mathrm{/}{\rho }_t$, since this function has derivative $1-1\mathrm{/}z$ and its minimum is $0$ at $z=1$; integrating gives $P_0{f}_t=2{\mathbb{E}}_0\log Y\le 2\log {\rho }_t$. Since $\log (1-x)\le -x$ for $0\le x<1$ because $-x-\log (1-x)$ has derivative $x\mathrm{/}(1-x)\ge 0$ and value $0$ at $x=0$. Thus $2\log {\rho }_t\le -H^2$, so for some $c,\delta >0$, $P_0{f}_t\le -c\mathrm{\parallel }t{\mathrm{\parallel }}^2$ whenever $\mathrm{\parallel }t\mathrm{\parallel }\le {\delta }_0$.

Let ${\mathbb{G}}_{n}f=\sqrt{n}(P_{n}f-P_{0}f)$ for the empirical measure $P_n$. The pointwise Lipschitz envelope gives $\mathrm{\parallel }{f}_t-f_v{\mathrm{\parallel }}_{L^2(P_0)}\le \mathrm{\parallel }L{\mathrm{\parallel }}_2\mathrm{\parallel }t-v\mathrm{\parallel }$. The sample and mean processes are continuous in $t$ on compact local balls, so their suprema equal suprema over countable dense subsets and are measurable. To bound the local supremum, take an independent ghost sample $X_1^{\mathrm{\prime }},\dots ,X_n^{\mathrm{\prime }}$. For each fixed original sample, the supremum of absolute values is convex in the ghost sample vector, so Jensen's inequality replaces $P_{0}f$ by $n^{-1}{\sum }_{i}f(X_i^{\mathrm{\prime }})$ in the expected supremum. Swapping each pair $(X_i,X_i^{\mathrm{\prime }})$ with an independent sign and then using the triangle inequality bounds that supremum by twice the expected conditional Rademacher supremum. Condition on the observations for the latter. For the conditional Rademacher process $R_n(f)=n^{-1\mathrm{/}2}{\sum }_i{ϵ}_{i}f(X_i)$, Hoeffding's exponential bound ${\mathbb{E}}_{ϵ}{e}^{\lambda (R_n(f)-R_n(g))}={\prod }_i\cosh (\lambda (f-g)(X_i)\mathrm{/}\sqrt{n})\le \exp \{{\lambda }^2{P}_n(f-g{)}^2\mathrm{/}2\}$, since $\log \cosh u\le u^2\mathrm{/}2$ (integrate $\tanh u\le u$ for $u\ge 0$ and use evenness), makes each increment sub-Gaussian with variance proxy $P_n(f-g{)}^2\le P_n{L}^2\mathrm{\parallel }t-v{\mathrm{\parallel }}^2$. Choose successively finer dyadic Euclidean nets of the radius-$r$ parameter ball. At level $k$ the net has at most $(C{2}^k{)}^d$ points, each increment from its level-$(k-1)$ parent has scale at most $Cr{2}^{-k}(P_n{L}^2{)}^{1\mathrm{/}2}$, and the expected maximum of $N$ centered sub-Gaussian variables of scale $\sigma$ is at most $\sigma \sqrt{2\log (2N)}$, by the exponential-moment bound and optimizing $\lambda$. Summing the resulting bounds uses the convergent series ${\sum }_{k\ge 1}{2}^{-k}\sqrt{k}$ and gives ${\mathbb{E}}_{ϵ}{\sup }_{\mathrm{\parallel }t\mathrm{\parallel }\le r}\mathrm{\mid }{R}_n(f_t-f_0)\mathrm{\mid }\le C_{d}r(P_n{L}^2{)}^{1\mathrm{/}2}$. Symmetrization and $\mathbb{E}(P_n{L}^2{)}^{1\mathrm{/}2}\le (P_0{L}^2{)}^{1\mathrm{/}2}$ therefore give $\mathbb{E}{\sup }_{\mathrm{\parallel }t\mathrm{\parallel }\le r}\mathrm{\mid }{\mathbb{G}}_n(f_t-f_0)\mathrm{\mid }\le C_{d}r\mathrm{\parallel }L{\mathrm{\parallel }}_2\mathrm{.}$ For $r_t=f_t-t^T{s}_{{\theta }_0}$, step 2.1 gives ${\sup }_{\mathrm{\parallel }t\mathrm{\parallel }\le M\mathrm{/}\sqrt{n}}{P}_0{r}_t^2=o(n^{-1})$, and increments have the square-integrable pointwise envelope $L+\mathrm{\parallel }{s}_{{\theta }_0}\mathrm{\parallel }$. Fix a mesh $\epsilon >0$ and cover the bounded $h$-ball by finitely many mesh-$\epsilon$ points. At each center $h_j$, the variance of $\sqrt{n}{\mathbb{G}}_n{r}_{h_j\mathrm{/}\sqrt{n}}$ is $n{\mathrm{Var}}_0(r_{h_j\mathrm{/}\sqrt{n}})\le n{P}_0{r}_{h_j\mathrm{/}\sqrt{n}}^2=o(1)$, so Chebyshev and finiteness of the net make the maximum over centers $o_P(1)$. For increments within cells, form $g_h=r_{h\mathrm{/}\sqrt{n}}-r_{h_j\mathrm{/}\sqrt{n}}$, where $h_j$ is the center of the cell containing $h$. Each $g_h$ is pointwise bounded by $\epsilon (L+\mathrm{\parallel }{s}_{{\theta }_0}\mathrm{\parallel })\mathrm{/}\sqrt{n}$. Apply conditional Rademacher chaining to the union of cell classes: at dyadic level $k$ there are at most $C_M{\epsilon }^{-d}{2}^{kd}$ links, each with conditional scale at most $C\epsilon 2^{-k}(P_n(L+\mathrm{\parallel }{s}_{{\theta }_0}\mathrm{\parallel }{)}^2{)}^{1\mathrm{/}2}$ after the $\sqrt{n}$ normalization. The exponential maximum bound and summation of ${\sum }_{k\ge 0}{2}^{-k}\sqrt{\log (C_M{\epsilon }^{-d})+C_{d}k}$ give an expected supremum at most $C_M\epsilon \sqrt{\log (C_M\mathrm{/}\epsilon )}$ after symmetrization and $\mathbb{E}(P_n(L+\mathrm{\parallel }{s}_{{\theta }_0}\mathrm{\parallel }{)}^2{)}^{1\mathrm{/}2}\le \mathrm{\parallel }L+\mathrm{\parallel }{s}_{{\theta }_0}\mathrm{\parallel }{\mathrm{\parallel }}_2$. Markov's inequality then shows that these cell increments vanish in probability as $\epsilon \downarrow 0$. Letting $n\to \mathrm{\infty }$ and then $\epsilon \downarrow 0$ proves ${\sup }_{\mathrm{\parallel }h\mathrm{\parallel }\le M}\mathrm{\mid }\sqrt{n}{\mathbb{G}}_n{r}_{h\mathrm{/}\sqrt{n}}\mathrm{\mid }=o_P(1)$. Thus, for each fixed $M$, uniformly over $\mathrm{\parallel }h\mathrm{\parallel }\le M$, ${\mathrm{\ell }}_n({\theta }_0+h\mathrm{/}\sqrt{n})-{\mathrm{\ell }}_n({\theta }_0)=h^T{\mathrm{\Delta }}_n-\frac{1}{2}{h}^{T}Ih+o_{P_0}(1),{\mathrm{\Delta }}_n=n^{-1\mathrm{/}2}{\sum }_{i=1}^n{s}_{{\theta }_0}(X_i)\mathrm{.}$ The expectation term is step 2.1; the centered term follows by writing $f_t=t^T{s}_{{\theta }_0}+(f_t-t^T{s}_{{\theta }_0})$ and using the two net bounds.

Reduce ${\delta }_0$ if needed so the log envelope holds on the radius $2{\delta }_0$ ball, and put $\delta ={\delta }_0\mathrm{/}2$. Consistency makes ${\widehat{\theta }}_n$ lie within $\delta$ of ${\theta }_0$ with probability tending to one. On the dyadic shells $r_j\le \mathrm{\parallel }t\mathrm{\parallel }\le 2r_j$ with $r_j=2^{j}M\mathrm{/}\sqrt{n}$, exact maximization and ${\mathrm{\ell }}_n({\widehat{\theta }}_n)\ge {\mathrm{\ell }}_n({\theta }_0)$ require $\sqrt{n}{\sup }_{\mathrm{\parallel }t\mathrm{\parallel }\le 2r_j}\mathrm{\mid }{\mathbb{G}}_n{f}_t\mathrm{\mid }\ge cn{r}_j^2$. The expectation bound in step 3.1 and Markov's inequality [F11] bound the probability of this event by $C\mathrm{/}(\sqrt{n}{r}_j)=C\mathrm{/}(2^{j}M)$. Summing the geometric series and then letting $M\to \mathrm{\infty }$ proves $\sqrt{n}({\widehat{\theta }}_n-{\theta }_0)=O_{P_0}(1)$ using [F4]. The argument used only consistency, feasibility of ${\theta }_0$, and the maximizing inequality; it therefore proves the same bound for any exact maximizer over a subset containing ${\theta }_0$.

Put ${\widehat{h}}_n=\sqrt{n}({\widehat{\theta }}_n-{\theta }_0)$ and $h_n^{\ast }=I^{-1}{\mathrm{\Delta }}_n$. The multivariate CLT [F7] and [F5] make $h_n^{\ast }=O_{P_0}(1)$; step 4.1 makes ${\widehat{h}}_n=O_{P_0}(1)$. For $Q_n(h)=h^T{\mathrm{\Delta }}_n-\frac{1}{2}{h}^{T}Ih$, $Q_n(h_n^{\ast })-Q_n(h)=\frac{1}{2}(h-h_n^{\ast }{)}^{T}I(h-h_n^{\ast })\ge \frac{1}{2}{\lambda }_{\min }(I)\mathrm{\parallel }h-h_n^{\ast }{\mathrm{\parallel }}^2\mathrm{.}$ Uniform convergence on bounded balls in step 3.1, feasibility of bounded local $h$ because ${\theta }_0$ is interior, and exact MLE maximization therefore force ${\widehat{h}}_n-h_n^{\ast }=o_{P_0}(1)$.

The CLT [F7] gives ${\mathrm{\Delta }}_n\Rightarrow N_d(0,I)$; the limiting law exists by [F8]. Since $I$ is invertible, $h_n^{\ast }=I^{-1}{\mathrm{\Delta }}_n\Rightarrow N_d(0,I^{-1})$ by the continuous linear map. Combining this with step 5.1 by [F6] proves both the asymptotic linear representation and the stated normal limit. AC is used only through [F7]–[F8].