Regression Estimation

For the general assertion truncate $L$ to $L_A=L{\mathbf{1}}_{\{\mathrm{\mid }u\mathrm{\mid }\le A,\mathrm{\mid }L(u)\mathrm{\mid }\le A\}}$. The expectation of $(nh{)}^{-1}{\sum }_i{L}_A(U_i)$ is $\int L_A(u)f(x-hu)du\to f(x)\int L_A$ by uniform density convergence on its fixed compact support. Its variance is at most $(nh{)}^{-1}\int L_A(u{)}^{2}f(x-hu)du=O_A((nh{)}^{-1})\to 0$. The elementary bound $\mathbb{P}(\mathrm{\mid }V-\mathbb{E}V\mathrm{\mid }>ϵ)\le \mathrm{Var}(V)\mathrm{/}{ϵ}^2$ follows by integrating ${ϵ}^2{\mathbf{1}}_{\{\mathrm{\mid }V-\mathbb{E}V\mathrm{\mid }>ϵ\}}\le (V-\mathbb{E}V{)}^2$. Thus the truncated average converges in probability.

The expectation of the absolute difference between the full and truncated averages is at most $C\int \mathrm{\mid }L-L_A\mathrm{\mid }$, using the global density bound or its bound on the shrinking image of the compact support of $L$. This tends to zero as $A\to \mathrm{\infty }$. Markov's bound, obtained from $ϵ{\mathbf{1}}_{\{Z>ϵ\}}\le Z$ for $Z\ge 0$, makes this difference uniformly small in probability. Also $\int L_A\to \int L$. First choosing $A$ large and then $n$ large proves the general assertion.

For the rapid-tail case and $L(u)=u^{j}K(u)$, $j=0,1,2$, restrict first to $\mathrm{\mid }x-X_i\mathrm{\mid }\le \delta$, where the density is bounded. The same truncation proof works there. The expected absolute contribution of the remaining observations is at most ${ϵ}_h{h}^{2-j}{\int }_{\mathrm{\mid }z-x\mathrm{\mid }>\delta }\mathrm{\mid }z-x{\mathrm{\mid }}^{j-3}f(z)dz\to 0$, where ${ϵ}_h={\sup }_{\mathrm{\mid }u\mathrm{\mid }\ge \delta \mathrm{/}h}\mathrm{\mid }u{\mathrm{\mid }}^{3}K(u)\to 0$. Thus the same scalar limits hold for these three moments without a global density bound.

Apply steps 1.1–3.1 to $L(u)=K(u)u^j$ for $j=0,1,2$, all integrable by the finite second moment. Symmetry gives $\int uK=0$. The entries of $M_n$ therefore have the displayed limits. Its determinant tends in probability to $f(x{)}^2{\mu }_2>0$, since it is a polynomial in the entries. A zero determinant lies outside a fixed neighborhood of this positive limit, so its probability tends to zero.

Write $T_x-m_{\theta }(x)=(T_x-{\mathbb{E}}_{\theta }{T}_x)+b_{\theta }(x)$. Squaring and taking expectations gives ${\mathbb{E}}_{\theta }[(T_x-m_{\theta }(x){)}^2]={\mathbb{E}}_{\theta }[(T_x-{\mathbb{E}}_{\theta }{T}_x{)}^2]+b_{\theta }(x{)}^2$, because ${\mathbb{E}}_{\theta }[T_x-{\mathbb{E}}_{\theta }{T}_x]=0$.

By [F1] the left side of step 1.1 is the pointwise risk, and by [F2] its first term on the right is the variance. This is the asserted identity.

Put $g=mf$. For $v=f,g$, twice applying the fundamental theorem of calculus gives $v(x-t)=v(x)-t{v}^{\mathrm{\prime }}(x)+t^2{\int }_0^1(1-s)v^{\mathrm{\prime }\mathrm{\prime }}(x-st)ds$. Thus $v(x-hu)=v(x)-hu{v}^{\mathrm{\prime }}(x)+\frac{1}{2}{h}^2{u}^2{v}^{\mathrm{\prime }\mathrm{\prime }}(x)+h^2{u}^2{r}_{v,h}(u)$. On compact kernel support the remainder tends uniformly to zero. Under the global alternative it is uniformly bounded and tends to zero for each fixed $u$: split the remainder integral at $\mathrm{\mid }u\mathrm{\mid }\le A$, use uniform continuity near $x$ on that fixed set, and bound the remaining part by $C{\int }_{\mathrm{\mid }u\mathrm{\mid }>A}{u}^{2}K(u)du\to 0$ as $A\to \mathrm{\infty }$. In the rapid-tail alternative [F4] gives $\int \mathrm{\mid }g\mathrm{\mid }<\mathrm{\infty }$. Put ${ϵ}_h={\sup }_{\mathrm{\mid }u\mathrm{\mid }\ge \delta \mathrm{/}h}\mathrm{\mid }u{\mathrm{\mid }}^{3}K(u)\to 0$. For $v=f,mf$, $\int \mathrm{\mid }v\mathrm{\mid }<\mathrm{\infty }$ and $h^{-1}{\int }_{\mathrm{\mid }z-x\mathrm{\mid }>\delta }K((x-z)\mathrm{/}h)\mathrm{\mid }v(z)\mathrm{\mid }dz\le {ϵ}_h{h}^2{\delta }^{-3}\int \mathrm{\mid }v\mathrm{\mid }=o(h^2)$. On $\mathrm{\mid }z-x\mathrm{\mid }\le \delta$ use the local Taylor remainder; its normalized integral is bounded by ${\mu }_2$ times the modulus of continuity of $v^{\mathrm{\prime }\mathrm{\prime }}$ at $x$. The omitted constant, linear and quadratic Taylor tails are $o(h^2)$ because ${\int }_{\mathrm{\mid }u\mathrm{\mid }>\delta \mathrm{/}h}{u}^{2}K\to 0$. First let $h\to 0$, then $\delta \to 0$. In every alternative the integrated remainder is $o(h^2)$. Symmetry and the finite second moment give $\int uK(u)du=0$, so $\int K(u)v(x-hu)du=v(x)+\frac{1}{2}{h}^2{\mu }_2{v}^{\mathrm{\prime }\mathrm{\prime }}(x)+o(h^2)$. All integrals are finite by localization or the global bounds.

Apply step 1.1 to numerator $g$ and denominator $f$. The latter tends to $f(x)>0$. Taking their quotient and subtracting $m(x)$ gives $\frac{1}{2}{h}^2{\mu }_2\{(mf{)}^{\mathrm{\prime }\mathrm{\prime }}\mathrm{/}f-m{f}^{\mathrm{\prime }\mathrm{\prime }}\mathrm{/}f\}(x)+o(h^2)=\frac{1}{2}{h}^2{\mu }_2(m^{\mathrm{\prime }\mathrm{\prime }}+2m^{\mathrm{\prime }}{f}^{\mathrm{\prime }}\mathrm{/}f)(x)+o(h^2)$, as asserted.

Write $r_h(u)=m(x-hu)-m(x)+hu{m}^{\mathrm{\prime }}(x)$. Taylor's integral formula bounds $\mathrm{\mid }{r}_h(u)\mathrm{\mid }\le C{h}^2{u}^2$ and gives $r_h(u)\mathrm{/}{h}^2\to \frac{1}{2}{m}^{\mathrm{\prime }\mathrm{\prime }}(x)u^2$ pointwise. It also gives $\mathrm{\mid }{r}_h(u)\mathrm{\mid }\le Ch\mathrm{\mid }u\mathrm{\mid }$ and $r_h(u)\mathrm{/}(h\mathrm{\mid }u\mathrm{\mid })\to 0$ for each nonzero $u$. These bounds hold on compact kernel support for small $h$, or globally under the alternative assumptions. With $R_j=\int K(u)u^{j}f(x-hu)r_h(u)du$, splitting at a fixed $\mathrm{\mid }u\mathrm{\mid }\le A$ and controlling the tails by ${\int }_{\mathrm{\mid }u\mathrm{\mid }>A}{u}^{2}K(u)du$ proves $R_0=\frac{1}{2}{h}^2{m}^{\mathrm{\prime }\mathrm{\prime }}(x)f(x){\mu }_2+o(h^2)$ and $R_1=o(h)$. The same split yields $S_0\to f(x)$, $S_2\to f(x){\mu }_2$; symmetry and the first-order expansion of $f$ give $S_1=-h{f}^{\mathrm{\prime }}(x){\mu }_2+o(h)=O(h)$. In the rapid-tail alternative [F4] gives $\int \mathrm{\mid }m\mathrm{\mid }f<\mathrm{\infty }$. Write $z=x-hu$ and split at $\mathrm{\mid }z-x\mathrm{\mid }=\delta$. The far parts of $S_0,S_1,S_2$ are respectively $o(h^2),o(h),o(1)$, since $K((x-z)\mathrm{/}h)\le {ϵ}_h{h}^3\mathrm{/}\mathrm{\mid }x-z{\mathrm{\mid }}^3$ and $\int f=1$. The far parts of $R_0,R_1$ are $o(h^2),o(h)$: expand $r=m(z)-m(x)-m^{\mathrm{\prime }}(x)(z-x)$ and use $\int \mathrm{\mid }m\mathrm{\mid }f<\mathrm{\infty }$ with $\mathrm{\mid }z-x\mathrm{\mid }\ge \delta$. On the near region the preceding local bounds and the integrable $u^{2}K$ apply; letting $\delta \to 0$ after $h\to 0$ gives the same remainder limits. These estimates also make every stated integral finite.

The determinant tends to $f(x{)}^2{\mu }_2>0$. Since $T_j=m(x)S_j-h{m}^{\mathrm{\prime }}(x)S_{j+1}+R_j$ for $j=0,1$, the affine terms in the intercept cancel exactly and $m_{\mathrm{L}\mathrm{L},h}(x)-m(x)=(S_2{R}_0-S_1{R}_1)\mathrm{/}{D}_h$. Step 1.1 makes $S_1{R}_1=o(h^2)$ and gives the asserted coefficient after division by the positive limiting determinant.

Put $h=h_n$, $U_i=(x-X_i)\mathrm{/}h$, $S_j=(nh{)}^{-1}{\sum }_{i}K(U_i)U_i^j$ and $r_h(u)=m(x-hu)-m(x)+hu{m}^{\mathrm{\prime }}(x)$. Then $S_0\to f(x)$, $S_2\to f(x){\mu }_2$ by [F2]. Symmetry and the first-order density expansion give $\mathbb{E}{S}_1=-h{f}^{\mathrm{\prime }}(x){\mu }_2+o(h)$; the tail is controlled by $u^{2}K$ under the global alternative. Also $\mathrm{Var}(S_1)\le C(nh{)}^{-1}\int u^2{K}^2$, using localization or the global density bound. In the rapid-tail alternative its local part has this bound, and the far contribution to $\mathrm{Var}(S_1)$ is at most $C{h}^2{n}^{-1}{\int }_{\mathrm{\mid }z-x\mathrm{\mid }>\delta }f(z)\mathrm{\mid }z-x{\mathrm{\mid }}^{-4}dz$, hence no larger than $O((nh{)}^{-1})$. Since $n{h}^3\to \mathrm{\infty }$, $(S_1-\mathbb{E}{S}_1)\mathrm{/}h\to 0$ in probability by the variance bound, so $S_1=O_P(h)$.

Set $R_j=(nh{)}^{-1}{\sum }_{i}K(U_i)U_i^j{r}_h(U_i)$. Taylor's integral formula gives $\mathrm{\mid }{r}_h(u)\mathrm{\mid }\le C{h}^2{u}^2$ and $r_h(u)\mathrm{/}{h}^2\to \frac{1}{2}{m}^{\mathrm{\prime }\mathrm{\prime }}(x)u^2$. The truncation proof of [F2] applies to this varying integrand: on any fixed set where $\mathrm{\mid }u\mathrm{\mid }$ and $K(u)$ are bounded it converges uniformly, its average has variance $O_A((nh{)}^{-1})$, and the absolute tail expectation is at most $C{\int }_{\mathrm{t}\mathrm{a}\mathrm{i}\mathrm{l}}{u}^{2}K$. Hence $R_0\mathrm{/}{h}^2\to \frac{1}{2}{m}^{\mathrm{\prime }\mathrm{\prime }}(x)f(x){\mu }_2$ in probability. Furthermore $\mathrm{\mid }{r}_h(u)\mathrm{\mid }\le Ch\mathrm{\mid }u\mathrm{\mid }$ and $r_h(u)\mathrm{/}(h\mathrm{\mid }u\mathrm{\mid })\to 0$ pointwise. Splitting the integral as before gives $\mathbb{E}\mathrm{\mid }{R}_1\mathrm{\mid }\mathrm{/}h\le \int K(u)\mathrm{\mid }u\mathrm{\mid }\mathrm{\mid }{r}_h(u)\mathrm{\mid }f(x-hu)\mathrm{/}hdu\to 0$. Markov's bound yields $R_1=o_P(h)$. Under the rapid-tail alternative [F5] gives $\int \mathrm{\mid }m\mathrm{\mid }f<\mathrm{\infty }$; first apply these arguments on $\mathrm{\mid }{X}_i-x\mathrm{\mid }\le \delta$. For the far part put ${ϵ}_h={\sup }_{\mathrm{\mid }u\mathrm{\mid }\ge \delta \mathrm{/}h}\mathrm{\mid }u{\mathrm{\mid }}^{3}K(u)\to 0$ and expand $r=m(z)-m(x)-m^{\mathrm{\prime }}(x)(z-x)$. Its expected absolute contributions to $R_0\mathrm{/}{h}^2$ and $R_1\mathrm{/}h$ are bounded by $C_{\delta }{ϵ}_h\int (1+\mathrm{\mid }m\mathrm{\mid })f\to 0$. The same tail estimate gives the stated expectation expansion for $S_1$. Thus all limits hold without global smoothness.

On a nonsingular design, [F1] and [F3] give $b_n-m(x)=(S_2{R}_0-S_1{R}_1)\mathrm{/}(S_0{S}_2-S_1^2)$, since the constant and slope terms cancel exactly in the intercept. Steps 1.1–2.1 give $S_1{R}_1=o_P(h^2)$, and the denominator tends to $f(x{)}^2{\mu }_2>0$. Division gives the claimed coefficient. The singular-design event has probability tending to zero by [F2]; even after scaling by $h^{-2}$ the probability of any nonzero fallback contribution is at most that event probability, so it does not change convergence in probability. This argument integrates the response law separately for each design and never interchanges an undefined unconditional signed expectation.

By [F3], $M_n=(nh{)}^{-1}{\sum }_{i}K(U_i)(1,U_i{)}^{\mathsf{T}}(1,U_i)\to f(x)\mathrm{diag}(1,{\mu }_2)$ in probability. Put $V_n=(nh{)}^{-1}{\sum }_{i}K(U_i{)}^2{\sigma }^2(X_i)(1,U_i{)}^{\mathsf{T}}(1,U_i)$. The scalar generalization proved in [F3] applies also to a continuous multiplier ${\sigma }^2(x-hu)$: on fixed truncated support this multiplier converges uniformly to ${\sigma }^2(x)$ and is bounded, and the tail expectation is bounded by $C{\int }_{\mathrm{t}\mathrm{a}\mathrm{i}\mathrm{l}}\mathrm{\mid }L\mathrm{\mid }$. Thus, for $L=K^2$ the $(0,0)$ entry tends to $f(x){\sigma }^2(x)R(K)$. In the rapid-tail alternative, local truncation remains valid. For $j=0,1,2$ the expected absolute far contribution to $V_{n,j}$ is bounded by $C{h}^{5-j}{\int }_{\mathrm{\mid }z-x\mathrm{\mid }>\delta }\mathrm{\mid }z-x{\mathrm{\mid }}^{j-6}{\sigma }^2(z)f(z)dz=o(1)$, since [F7] gives $\int {\sigma }^{2}f\le \mathbb{E}{Y}^2$. Also $\int (1+u^2)K^2<\mathrm{\infty }$ follows from boundedness and the rapid tail. For the local-linear conclusion, $\int (1+u^2)K^2<\mathrm{\infty }$ permits $L=K^2{u}^j$, $j=0,1,2$, by the same truncation argument. All entries then converge to $f(x){\sigma }^2(x)\int K^2{u}^{j+k}$; symmetry makes the off-diagonal limit zero. Terms with $K(U_i)=0$ are interpreted as zero. Every active response-law variance is finite almost surely; these are variances under the specified conditional response laws, without an assumption of unconditional square integrability.

On the event ${\sum }_{i}K(U_i)>0$, [F1], conditional response independence, and [F4] give $\mathrm{Var}({\widehat{m}}_{\mathrm{N}\mathrm{W},h_n}(x)\mid X_{1:n})={\displaystyle \frac{\sum _{i}K(U_i{)}^2{\sigma }^2(X_i)}{(\sum _{i}K(U_i){)}^2}}$. Writing $M_{n,00}=(n{h}_n{)}^{-1}{\sum }_{i}K(U_i)$ and $V_{n,00}=(n{h}_n{)}^{-1}{\sum }_{i}K(U_i{)}^2{\sigma }^2(X_i)$ makes the scaled variance $V_{n,00}\mathrm{/}{M}_{n,00}^2$, which converges to $f(x){\sigma }^2(x)R(K)\mathrm{/}f(x{)}^2$. The zero-denominator event has probability tending to zero by [F3]'s $(0,0)$ entry, so the same convergence holds with the fallback.

On a nonsingular local-linear design matrix, weighted least squares gives conditional variance $(n{h}_n{)}^{-1}{e}_0^{\mathsf{T}}{M}_n^{-1}{V}_n{M}_n^{-1}{e}_0$, where $e_0=(1,0{)}^{\mathsf{T}}$. The limit matrix is diagonal with entries $f(x)$ and $f(x){\mu }_2$, so its inverse has first diagonal entry $1\mathrm{/}f(x)$; the $(0,0)$ limit of $V_n$ is $f(x){\sigma }^2(x)R(K)$. Matrix inversion is continuous at this invertible limit, because its entries are quotients of entries of the adjugate by the nonzero determinant. Thus the scaled conditional variance converges to $R(K){\sigma }^2(x)\mathrm{/}f(x)$. By [F3] the singular-design event has probability tending to zero, and the fallback does not change convergence in probability.

For the extra Nadaraya–Watson clause, choose $A,\kappa >0$ with $J=\{\mathrm{\mid }u\mathrm{\mid }\le A,K(u)\ge \kappa \}$ of positive measure, possible since $\int K=1$. Its predictor count is binomial with success probability $p_h\asymp h$. On $G_n=\{N\ge n{p}_h\mathrm{/}2\}$, $T_n=n^{-1}{\sum }_{i}K(U_i)\ge c_{1}h$, whereas Markov's bound applied to $e^{-N}$ gives $\mathbb{P}(G_n^c)\le \exp (-c_{2}nh)$. The truncation argument of step 1.1 also proves $V_{n,00}\to f(x){\sigma }^2(x)R(K)$ in $L^1$: the bounded truncated average converges in $L^2$, while the expected absolute tail is uniformly small. On $G_n$, $(T_n\mathrm{/}h{)}^{-2}$ is bounded and converges in probability to $f(x{)}^{-2}$. It follows that the expectation of the scaled conditional variance on $G_n$ has the stated limit: separate $V_{n,00}$ into its constant limit and an $L^1$ error, and use bounded convergence in probability for the reciprocal factor. On $G_n^c$, the conditional variance is at most the active-region bound on ${\sigma }^2$, since the weights are nonnegative and sum to one, or the fallback is deterministic. Hence its scaled expectation is $O(nh{e}^{-c_{2}nh})\to 0$ in those alternatives. Under rapid tails, the conditional variance is at most ${\sum }_i{\sigma }^2(X_i)$; the count among the other $n-1$ observations is independent of $X_i$ and is below $n{p}_h\mathrm{/}2$ on $G_n^c$. Its binomial lower-tail bound gives $\mathbb{E}[{\sigma }^2(X_i);G_n^c]\le \mathbb{E}{Y}^2{e}^{-c_{3}nh}$. Thus the contribution to the expected scaled conditional variance is at most $n^{2}h\mathbb{E}{Y}^2{e}^{-c_{3}nh}$. Under the stated $n{h}^3\to \mathrm{\infty }$ condition, $nh\ge n^{2\mathrm{/}3}$ eventually, so this polynomial factor times the exponential tends to zero.

Let $A_n$ be the response-law mean of the Nadaraya–Watson fit, including its fallback, and let $m_h=\int Kmf\mathrm{/}\int Kf$. The kernel integrals and localization or global bounds imply $m_h\to m(x)$. Put $Z_i=K(U_i)$ and $R_n=n^{-1}{\sum }_i{Z}_i(m(X_i)-m_h)$. Then $\mathbb{E}{R}_n=0$ and $\mathbb{E}{R}_n^2\le n^{-1}\mathbb{E}[Z_1^2(m(X_1)-m_h{)}^2]=o(h\mathrm{/}n)$: after change of variables, continuity at $x$ handles a fixed compact $u$-set and the integrable $K^2$ tail is dominated by bounded $f,m$. On $G_n$, $A_n-m_h=R_n\mathrm{/}{T}_n$, so its expected square is $o((nh{)}^{-1})$. On $G_n^c$, $A_n$ and $m_h$ are bounded in the compact/global alternatives. Under rapid tails, [F7] gives $\int m^{2}f<\mathrm{\infty }$; the far part of the second-moment integral is $O(h^6)\int m^{2}f=o(h)$, and the leave-one-out binomial argument in step 3.1 gives $\mathbb{E}[A_n^2;G_n^c]\le n\mathbb{E}{Y}^2{e}^{-c_{4}nh}+a_0^2\mathbb{P}(G_n^c)$. Its expected squared bad-event contribution is therefore negligible in all stated cases, and $\mathrm{Var}(A_n)\le \mathbb{E}(A_n-m_h{)}^2=o((nh{)}^{-1})$. Bounded active-region means and variances, or the bound ${\widehat{m}}^2\le {\sum }_i{Y}_i^2+a_0^2$ under rapid tails, make the Nadaraya–Watson fit square-integrable. Expanding its centered square and integrating first under the response laws gives total variance equal to the expected conditional variance plus $\mathrm{Var}(A_n)$. Step 3.1 therefore proves the unconditional Nadaraya–Watson expansion.