Density Estimation

Each coordinate projection ${\pi }_i:({\mathbb{R}}^d{)}^n\to {\mathbb{R}}^d$ is measurable by the definition of the finite product sigma-algebra. The map $z\mapsto (x-z)\mathrm{/}h$ is continuous, hence Borel; composing it with ${\pi }_i$ and then with $K$ gives a real measurable function on the sample space by [F1] and [F3].

By [F4], the finite sum of these functions and its scalar multiple $(n{h}^d{)}^{-1}$ are measurable. This is exactly $T_x$, so [F2] makes it a statistic with target $\mathbb{R}$. Since the estimand ${\psi }_x$ also has target space $\mathbb{R}$, $T_x$ is an estimator of ${\psi }_x$.

Induct on the number of factors using the finite product recursion in [F1] and the two-factor equality [F4]. Starting with $\mathcal{B}({\mathbb{R}}^d)$, this identifies the specified $(n+1)$-factor sigma-algebra with $\mathcal{B}({\mathbb{R}}^{(n+1)d})$. For each $i$, the affine map $(z_1,\dots ,z_n,x)\mapsto (x-z_i)\mathrm{/}h$ is continuous on this Euclidean space: inverse images of open sets are open, hence inverse images of Borel sets are Borel by closure under complements and countable unions. It is therefore measurable for the specified product sigma-algebra. Composing with the Borel map $K$ gives a measurable real function by [F2].

Applying [F3] to the finite sum of these functions and multiplying by the constant $(n{h}^d{)}^{-1}$ proves that $Q_h$ is Borel measurable.

Every summand is nonnegative because $K\ge 0$. For fixed $z_i$, the map $T_i(u)=z_i+hu$ is a $C^1$ diffeomorphism of ${\mathbb{R}}^d$ with $\mathrm{\mid }\det D{T}_i\mathrm{\mid }=h^d$. Applying [F2] to $K$ gives ${\int }_{{\mathbb{R}}^d}{h}^{-d}K\text{\hspace{0.17em}⁣}\left(\frac{x-z_i}{h}\right)dx={\int }_{{\mathbb{R}}^d}{h}^{-d}K(u)h^{d}du=1.$

Each scaled translate is in $L^1$ by step 1.1. Linearity [F3] therefore gives ${\int }_{{\mathbb{R}}^d}\frac{1}{n{h}^d}{\sum }_{i=1}^{n}K\text{\hspace{0.17em}⁣}\left(\frac{x-z_i}{h}\right)dx=\frac{1}{n}{\sum }_{i=1}^{n}1=1.$ This proves the claim.

Set $C=\max (1,H)$. For $x\in {\mathbb{R}}^d$, write $M=f(x)$ and $g=\mathrm{\nabla }f(x)$. For $q(t)=f(x+ty)$, the chain rule [F4] gives $q^{\mathrm{\prime }}(0)=g\cdot y$ and $q^{\mathrm{\prime }\mathrm{\prime }}(t)=y^T{D}^{2}f(x+ty)y$. Integration by parts and the second fundamental theorem [F5] give $q(1)-q(0)-q^{\mathrm{\prime }}(0)={\int }_0^1(1-t)q^{\mathrm{\prime }\mathrm{\prime }}(t)dt\mathrm{.}$ Since $\mathrm{\mid }{q}^{\mathrm{\prime }\mathrm{\prime }}(t)\mathrm{\mid }\le C\mathrm{\parallel }y{\mathrm{\parallel }}^2$ and ${\int }_0^1(1-t)dt=1\mathrm{/}2$, this yields $\mid f(x+y)-M-g\cdot y\mid \le \frac{C}{2}\mathrm{\parallel }y{\mathrm{\parallel }}^2,$ so both the upper and lower Taylor bounds with this remainder hold. Taking $y=-g\mathrm{/}C$ and using $f\ge 0$ yields $0\le f(x-g\mathrm{/}C)\le M-\mathrm{\parallel }g{\mathrm{\parallel }}^2\mathrm{/}(2C)$; hence $\mathrm{\parallel }g\mathrm{\parallel }\le \sqrt{2CM}$.

If $M>0$, put $r=\frac{1}{8}\sqrt{M\mathrm{/}C}$. For every $y$ with $\mathrm{\parallel }y\mathrm{\parallel }\le r$, the corresponding lower Taylor bound and step 1.1 give $f(x+y)\ge M-\mathrm{\parallel }g\mathrm{\parallel }r-\frac{C}{2}{r}^2\ge M(1-\frac{\sqrt{2}}{8}-\frac{1}{128})\ge \frac{1}{2}M\mathrm{.}$ The cube $Q=x+[-r\mathrm{/}\sqrt{d},r\mathrm{/}\sqrt{d}{]}^d$ lies in that ball and has measure ${\lambda }_d(Q)=(2r\mathrm{/}\sqrt{d}{)}^d=(M\mathrm{/}(16Cd){)}^{d\mathrm{/}2}$ by [A1] and [F2].

Monotonicity [F3] and $\int f=1$ now imply $1\ge {\int }_{Q}fd{\lambda }_d\ge \frac{1}{2}M{\left(\frac{M}{16Cd}\right)}^{d\mathrm{/}2}\mathrm{.}$ Thus $M^{1+d\mathrm{/}2}\le 2(16Cd{)}^{d\mathrm{/}2}$ for every $x$ with $f(x)>0$; points where $f(x)=0$ satisfy the same resulting finite bound. Therefore $f$ is bounded.

First take $g\in C_c^{\mathrm{\infty }}({\mathbb{R}}^d)$. For each $x,u$, set $q(t)=g(x-thu)$. The chain rule [F6] gives $q^{\mathrm{\prime }}(0)=-h\mathrm{\nabla }g(x)\cdot u$ and $q^{\mathrm{\prime }\mathrm{\prime }}(t)=h^2{u}^T{D}^{2}g(x-thu)u$. Integration by parts and the second fundamental theorem [F7] give $g(x-hu)=g(x)-h\mathrm{\nabla }g(x)\cdot u+h^2{\int }_0^1(1-t)u^T{D}^{2}g(x-thu)udt\mathrm{.}$ The first moments are absolutely integrable since $\mathrm{\mid }{u}_j\mathrm{\mid }\le 1+\mathrm{\parallel }u{\mathrm{\parallel }}^2$ and $K\in L^1$ with $M_2(K)<\mathrm{\infty }$. Integrating against $K(u)$ therefore cancels the constant and linear terms by $\int K=1$ and $\int u_{j}K(u)du=0$. Euclidean Cauchy–Schwarz gives $\mathrm{\mid }{u}^T{D}^{2}g(y)u\mathrm{\mid }\le \mathrm{\parallel }u\mathrm{\parallel }\mathrm{\parallel }{D}^{2}g(y)u\mathrm{\parallel }\le \mathrm{\parallel }u{\mathrm{\parallel }}^2\mathrm{\parallel }{D}^{2}g(y){\mathrm{\parallel }}_F$. Apply Holder [F5] with the finite measure $\mathrm{\mid }K(u)\mathrm{\mid }(1-t)\mathrm{\parallel }u{\mathrm{\parallel }}^{2}dudt$, whose total mass is $M_2(K)\mathrm{/}2$, to obtain $\mathrm{\mid }{S}_{h}g(x)-g(x){\mathrm{\mid }}^2\le h^4\left(\frac{M_2(K)}{2}\right){\int }_{{\mathbb{R}}^d}\text{\hspace{0.17em}⁣}{\int }_0^1\mathrm{\mid }K(u)\mathrm{\mid }(1-t)\mathrm{\parallel }u{\mathrm{\parallel }}^2\mathrm{\parallel }{D}^{2}g(x-thu){\mathrm{\parallel }}_F^{2}dtdu\mathrm{.}$ Tonelli [F3] and translation invariance from [F4] then give $\mathrm{\parallel }{S}_{h}g-g{\mathrm{\parallel }}_2^2\le h^4{\left(\frac{M_2(K)}{2}\right)}^2\mathrm{\parallel }{D}^{2}g{\mathrm{\parallel }}_{2,F}^2\mathrm{.}$

For every Borel representative $q\in L^2$, the integrand $(x,u)\mapsto K(u)q(x-hu)$ is product-measurable. Holder [F5] and Tonelli [F3] show its defining integral exists absolutely for almost every $x$ and $\mathrm{\parallel }{S}_{h}q{\mathrm{\parallel }}_2^2\le \mathrm{\parallel }K{\mathrm{\parallel }}_1{\int }_{{\mathbb{R}}^d}\mathrm{\mid }K(u)\mathrm{\mid }{\int }_{{\mathbb{R}}^d}\mathrm{\mid }q(x-hu){\mathrm{\mid }}^{2}dxdu=\mathrm{\parallel }K{\mathrm{\parallel }}_1^2\mathrm{\parallel }q{\mathrm{\parallel }}_2^2,$ by Holder [F5], Tonelli [F3], and translation invariance [F4]. If two Borel representatives agree almost everywhere, translation invariance makes their translates agree almost everywhere for each $u$; Tonelli [F3] then makes their convolution integrals agree for almost every $x$. Thus $S_h$ is well-defined as an $L^2$ equivalence class on the classes used here. Given $ϵ>0$, choose $g\in C_c^{\mathrm{\infty }}$ as in [F2]. The finite sum defining $\mathrm{\parallel }{D}^2(g-f){\mathrm{\parallel }}_{2,F}$ is at most $d\mathrm{\parallel }g-f{\mathrm{\parallel }}_{W^{2,2}}$, so step 1.1 and the operator bound just proved imply $\mathrm{\parallel }{S}_{h}f-f{\mathrm{\parallel }}_2\le (\mathrm{\parallel }K{\mathrm{\parallel }}_1+1)ϵ+\frac{h^2}{2}{M}_2(K)(\mathrm{\parallel }{D}^{2}f{\mathrm{\parallel }}_{2,F}+dϵ)\mathrm{.}$ Letting $ϵ\downarrow 0$ proves the asserted bound.

By [F2], $Z_i:=K((x-X_i)\mathrm{/}h)$ is a measurable real random variable. By [F3], $f$ is bounded. Applying [F5] to $\mathrm{\mid }{Z}_i\mathrm{\mid }$, then [F6] to $u\mapsto x-hu$, gives $\mathbb{E}\mathrm{\mid }{Z}_i\mathrm{\mid }=\int \mathrm{\mid }K((x-t)\mathrm{/}h)\mathrm{\mid }f(t)dt=h^d\int \mathrm{\mid }K(u)\mathrm{\mid }f(x-hu)du\le h^d\mathrm{\parallel }f{\mathrm{\parallel }}_{\mathrm{\infty }}\mathrm{\parallel }K{\mathrm{\parallel }}_1<\mathrm{\infty }\mathrm{.}$ Thus every $Z_i$ is integrable. Put $g(t)=K((x-t)\mathrm{/}h)$ and $g^{+}=\max (g,0)$, $g^{-}=\max (-g,0)$. Both parts are nonnegative and their density integrals are finite, being bounded by the absolute integral just established. Apply [F5] separately to $g^{+}$ and $g^{-}$, then subtract their finite integrals using [F7]: $\mathbb{E}{Z}_i=\int g^{+}f-\int g^{-}f=\int gf$. This derives the signed density identity without applying the nonnegative density theorem directly to $g$. Since the $X_i$ have the same law, the $Z_i$ have the same expectation. Linearity [F7] and the signed substitution in [F6] therefore yield $\mathbb{E}{\widehat{f}}_h(x)=\frac{1}{h^d}\mathbb{E}{Z}_1={\int }_{{\mathbb{R}}^d}K(u)f(x-hu)du\mathrm{.}$

For fixed $u$, set $q(t)=f(x-thu)$ on $[0,1]$. By [F9], $q^{\mathrm{\prime }}(0)=-h\mathrm{\nabla }f(x)\cdot u$ and $q^{\mathrm{\prime }\mathrm{\prime }}(t)=h^2{u}^T{D}^{2}f(x-thu)u$. Integration by parts with $1-t$ and $q^{\mathrm{\prime }}$, then the second fundamental theorem, gives $f(x-hu)=f(x)-h\mathrm{\nabla }f(x)\cdot u+h^2{\int }_0^1(1-t)u^T{D}^{2}f(x-thu)udt\mathrm{.}$ For $u\mathrm{\ne }0$, subtracting $\frac{h^2}{2}{u}^T{D}^{2}f(x)u$ defines a remainder $h^2\mathrm{\parallel }u{\mathrm{\parallel }}^2\rho (h,u)$, where $\rho (h,u)={\int }_0^1(1-t)\frac{u^T(D^{2}f(x-thu)-D^{2}f(x))u}{\mathrm{\parallel }u{\mathrm{\parallel }}^2}dt\mathrm{.}$ Continuity of the Hessian at $x$ gives $\rho (h,u)\to 0$ for each fixed $u$; the global Hessian bound gives $\mathrm{\mid }\rho (h,u)\mathrm{\mid }\le H$ uniformly in $h,u$. Set $\rho (h,0)=0$.

Substitute the expansion from step 1.2 into the convolution in step 1.1. The first moments are absolutely integrable because $\mathrm{\mid }{u}_j\mathrm{\mid }\le 1+\mathrm{\parallel }u{\mathrm{\parallel }}^2$ and $K\in L^1$ with finite second absolute moment, so unit mass cancels $f(x)$ and the zero first moments cancel the linear term. The remaining quadratic term is $\frac{h^2}{2}{\sum }_{j,k}{\mu }_{jk}{\mathrm{\partial }}_{jk}f(x)$. Moreover, $\mathrm{\mid }K(u)\mathrm{\mid }\mathrm{\parallel }u{\mathrm{\parallel }}^{2}H$ is integrable and dominates the scaled remainder. Applying [F8] as $h\downarrow 0$ shows that the remainder divided by $h^2$ tends to zero. The second moments ${\mu }_{jk}$ are absolutely finite by the same integrable moment condition.

Put $Z_i=K((x-X_i)\mathrm{/}h)$. By [F2] and [F7], and using the nonnegative substitution $u\mapsto x-hu$ from [F8], $\mathbb{E}[Z_i^2]=h^d\int K(u{)}^{2}f(x-hu)du\le h^d\mathrm{\parallel }f{\mathrm{\parallel }}_{\mathrm{\infty }}\mathrm{\parallel }K{\mathrm{\parallel }}_2^2<\mathrm{\infty }\mathrm{.}$ Likewise $\mathbb{E}\mathrm{\mid }{Z}_i\mathrm{\mid }\le h^d\mathrm{\parallel }f{\mathrm{\parallel }}_{\mathrm{\infty }}\mathrm{\parallel }K{\mathrm{\parallel }}_1<\mathrm{\infty }$, so each $Z_i$ is integrable and square-integrable. The maps $z\mapsto K((x-z)\mathrm{/}h)$ preserve independence by [F4]; identical laws of $X_i$ give identical laws of $Z_i$.

Let $W_i=Z_i-\mathbb{E}{Z}_i$. For $i\mathrm{\ne }j$, [F5] applied to the independent integrable variables $W_i,W_j$ gives $\mathbb{E}[W_i{W}_j]=\mathbb{E}[W_i]\mathbb{E}[W_j]=0$. Expanding the finite square and using expectation linearity [F11] therefore gives $\mathrm{Var}({\widehat{f}}_h(x))=\frac{1}{n{h}^{2d}}(\mathbb{E}[Z_1^2]-(\mathbb{E}{Z}_1{)}^2)\mathrm{.}$

Step 1.1 shows that $t\mapsto K((x-t)\mathrm{/}h)f(t)$ is integrable. Put $g(t)=K((x-t)\mathrm{/}h)$ and decompose $g=g^{+}-g^{-}$ into its nonnegative parts. The density integrals of both parts are finite because $\int \mathrm{\mid }g\mathrm{\mid }f<\mathrm{\infty }$ by step 1.1. Apply the nonnegative density identity [F7] to each part and subtract using [F11], obtaining $\mathbb{E}{Z}_1=\int gf$. Applying the signed substitution [F9] to this integrable product gives $\mathbb{E}{Z}_1=h^d\int K(u)f(x-hu)du$, so $\mathrm{\mid }\mathbb{E}{Z}_1\mathrm{\mid }\le h^d\mathrm{\parallel }f{\mathrm{\parallel }}_{\mathrm{\infty }}\mathrm{\parallel }K{\mathrm{\parallel }}_1$. Along any sequence $h_n\to 0$, [F10] gives $\int K(u{)}^{2}f(x-h_{n}u)du⟶f(x)\mathrm{\parallel }K{\mathrm{\parallel }}_2^2\mathrm{.}$ Thus the first term in step 2.1 is $f(x)\mathrm{\parallel }K{\mathrm{\parallel }}_2^2\mathrm{/}(n{h}_n^d)+o((n{h}_n^d{)}^{-1})$. The subtracted mean-square term is $O(n^{-1})=o((n{h}_n^d{)}^{-1})$ because $h_n^d\to 0$. The additional $n{h}_n^d\to \mathrm{\infty }$ condition places the expansion in the vanishing-variance regime. This proves the formula.

For fixed $h$, [F2] makes $(z,x)\mapsto {\widehat{f}}_h(x)$ Borel, and [F1] makes $f(x)$ Borel as a function of $(z,x)$. Arithmetic measurability in [F2] therefore makes the squared error product-measurable and nonnegative. Tonelli [F6] identifies MISE with the integral over $x$ of its expected pointwise squared error, even before finiteness has been established.

Write $q_h(x,z)=h^{-d}K((x-z)\mathrm{/}h)$. By Tonelli and the substitution $z+hu$ in [F8], ${\int }_{{\mathbb{R}}^d}\mathbb{E}\mathrm{\mid }{q}_h(x,X_1)\mathrm{\mid }dx=\mathrm{\parallel }K{\mathrm{\parallel }}_1,{\int }_{{\mathbb{R}}^d}\mathbb{E}[q_h(x,X_1{)}^2]dx=h^{-d}\mathrm{\parallel }K{\mathrm{\parallel }}_2^2\mathrm{.}$ Consequently, [F9] makes the summands integrable and square-integrable for almost every $x$. For such $x$, put $g(z)=q_h(x,z)$ and $g^{+}=\max (g,0)$, $g^{-}=\max (-g,0)$. The nonnegative density formula [F7] applied to $\mathrm{\mid }g\mathrm{\mid }$ shows $\int \mathrm{\mid }g\mathrm{\mid }f<\mathrm{\infty }$. Thus both parts have finite density integrals, and [F7] applied separately to them, followed by subtraction using [F11], gives $\mathbb{E}{q}_h(x,X_1)=\int g(z)f(z)dz$. Signed substitution [F8] then identifies this integral with $S_{h}f(x)$, and identical laws plus finite-sum linearity [F11] give $\mathbb{E}{\widehat{f}}_h(x)=S_{h}f(x)$.

At almost every $x$ from step 2.1, the variables $q_h(x,X_i)$ are iid and square-integrable. Set $W_i=q_h(x,X_i)-\mathbb{E}{q}_h(x,X_i)$. These are independent, integrable and square-integrable by [F5] and step 2.1; for $i\mathrm{\ne }j$, [F5] makes $W_i{W}_j$ integrable and gives $\mathbb{E}[W_i{W}_j]=\mathbb{E}{W}_i\mathbb{E}{W}_j=0$. Expanding the finite square in the variance definition [F10] and applying [F11] gives $\mathrm{Var}({\widehat{f}}_h(x))=n^{-2}{\sum }_i\mathbb{E}{W}_i^2=\mathrm{Var}(q_h(x,X_1))\mathrm{/}n$. Expanding one centered square using [F10]–[F11] gives $\mathrm{Var}(q_h(x,X_1))=\mathbb{E}[q_h(x,X_1{)}^2]-(\mathbb{E}{q}_h(x,X_1){)}^2\le \mathbb{E}[q_h(x,X_1{)}^2]$. Integrating and using step 2.1 bounds the integrated variance by $\mathrm{\parallel }K{\mathrm{\parallel }}_2^2\mathrm{/}(n{h}^d)$.

For almost every $x$, step 2.1 identifies the bias with $S_{h}f(x)-f(x)$. The $L^2$ bound [F3] therefore gives ${\int }_{{\mathbb{R}}^d}(\mathbb{E}{\widehat{f}}_h(x)-f(x){)}^{2}dx\le \frac{h^4}{4}{M}_2(K{)}^2\mathrm{\parallel }{D}^{2}f{\mathrm{\parallel }}_{2,F}^2\mathrm{.}$