Density Estimation — Examples

Put $I={\int }_{\mathbb{R}}{e}^{-t^2}dt$. Tonelli [F2] and [F1] give $I^2={\int }_{{\mathbb{R}}^2}{e}^{-(x^2+y^2)}d(x,y)=\pi$. Since $I\ge 0$, $I=\sqrt{\pi }$. Let $\varphi (t)=(2\pi {)}^{-1\mathrm{/}2}{e}^{-t^2\mathrm{/}2}$. The substitution $s=t\mathrm{/}\sqrt{2}$ in [F3] then gives $\int \varphi =1$. By [F5], ${\varphi }^{\mathrm{\prime }}(t)=-t\varphi (t)$. Integration by parts [F6] gives ${\int }_0^{R}t\varphi (t)dt=\varphi (0)-\varphi (R)$. By [F7], $\varphi (R)\to 0$ and $R\varphi (R)\to 0$, so $\int \mathrm{\mid }t\mathrm{\mid }\varphi (t)dt=2\varphi (0)<\mathrm{\infty }$. Oddness then gives $\int t\varphi (t)dt=0$. Integration by parts [F6] on $[-R,R]$ also yields ${\int }_{-R}^R{t}^2\varphi (t)dt={\int }_{-R}^R\varphi (t)dt-[t\varphi (t){]}_{-R}^R\mathrm{.}$ As $R\to \mathrm{\infty }$, $R\varphi (R)\to 0$ by [F7] (put $s=R^2\mathrm{/}4$ and use $R\le 2s$ for $R\ge 2$), and the right side tends to $1$, so $\int t^2\varphi (t)dt=1$ by [F9].

Since $K_d(u)={\prod }_{j=1}^d\varphi (u_j)$, repeated use of Tonelli [F2] shows $\int K_d=1$. Fubini [F4], the one-dimensional moments from step 1.1, and absolute integrability give ${\mu }_{jj}=1$ and ${\mu }_{jk}=0$ for $j\mathrm{\ne }k$. Also $\int K_d(u{)}^{2}du=(2\pi {)}^{-d}{\prod }_{j=1}^d{\int }_{\mathbb{R}}{e}^{-u_j^2}d{u}_j=(2\pi {)}^{-d}{\pi }^{d\mathrm{/}2}=(4\pi {)}^{-d\mathrm{/}2}\mathrm{.}$

Substituting ${\mu }_{jk}={\delta }_{jk}$ and $\mathrm{\parallel }{K}_d{\mathrm{\parallel }}_2^2=(4\pi {)}^{-d\mathrm{/}2}$ into [F8] yields the two displayed leading terms. The Gaussian kernel is nonnegative and integrates to one by step 2.1, so A nonnegative normalized kernel estimate integrates to one also confirms that each finite estimate is a density.

Put $I={\int }_{\mathbb{R}}{e}^{-t^2}dt$. Tonelli [F5] and [F4] give $I^2=\pi$, so $I=\sqrt{\pi }$. Scaling by $s=t\mathrm{/}\sqrt{2}$ with [F6] yields $\int (2\pi {)}^{-1\mathrm{/}2}{e}^{-t^2\mathrm{/}2}dt=1$; repeated Tonelli gives $\int K_d=1$. Since $f_{\tau }(x)={\tau }^{-d}{K}_d(x\mathrm{/}\tau )$, [F6] with $x=\tau u$ gives $\int f_{\tau }=1$.

By [F1] and [F2], expectation of one KDE summand is integration of its Gaussian kernel against $f_{\tau }$. The summand is bounded for fixed $h$, hence integrable. The integrand is nonnegative; using [F6] with $T(u)=x-hu$ changes variables to $\int K_d(u)f_{\tau }(x-hu)du$. All summands have this expectation, so linearity [F3] gives $\mathbb{E}{\widehat{f}}_h(x)={\int }_{{\mathbb{R}}^d}{K}_d(u)f_{\tau }(x-hu)du\mathrm{.}$ The integral is finite because both Gaussian densities are bounded and integrable.

In the integrand of step 2.1, combine exponents using $\mathrm{\parallel }u{\mathrm{\parallel }}^2+{\tau }^{-2}\mathrm{\parallel }x-hu{\mathrm{\parallel }}^2=\frac{{\tau }^2+h^2}{{\tau }^2}{\parallel u-\frac{hx}{{\tau }^2+h^2}\parallel }^2+\frac{\mathrm{\parallel }x{\mathrm{\parallel }}^2}{{\tau }^2+h^2}\mathrm{.}$ The remaining translated Gaussian integral is $(2\pi {\tau }^2\mathrm{/}({\tau }^2+h^2){)}^{d\mathrm{/}2}$ by [F4], [F5], and affine scaling [F6]. Multiplying the normalizing constants gives the displayed formula.

At $x=0$, $\mathbb{E}{\widehat{f}}_h(0)=(2\pi ({\tau }^2+h^2){)}^{-d\mathrm{/}2}<(2\pi {\tau }^2{)}^{-d\mathrm{/}2}=f(0),$ because $h>0$. Therefore a finite positive bandwidth has nonzero bias in this example.

The function $f={\mathbf{1}}_{[0,1]}$ is a Borel density version: [F8] gives ${\lambda }_1([0,1])=1$. Put $I={\int }_{\mathbb{R}}{e}^{-t^2}dt$. Tonelli [F5] and [F4] give $I^2=\pi$, so $I=\sqrt{\pi }$. Scaling with [F7] shows ${\int }_{\mathbb{R}}(2\pi {)}^{-1\mathrm{/}2}{e}^{-t^2\mathrm{/}2}dt=1$. Since this kernel is even, ${\int }_{-\mathrm{\infty }}^{0}K(u)du=1\mathrm{/}2$.

By [F1]–[F3] and the density formula, each summand is integrable because $K$ is bounded and the sample law is a probability measure. Thus $\mathbb{E}{\widehat{f}}_h(0)=h^{-1}{\int }_0^{1}K(-t\mathrm{/}h)dt\mathrm{.}$ Apply [F7] on $\mathbb{R}$ to $T(u)=-hu$ and $g(t)={\mathbf{1}}_{[0,1]}(t)K(-t\mathrm{/}h)$. It gives ${\int }_0^{1}K(-t\mathrm{/}h)dt=h{\int }_{-1\mathrm{/}h}^{0}K(u)du,$ so the displayed expectation equals ${\int }_{-1\mathrm{/}h}^{0}K(u)du$.

Along any sequence $h_m\downarrow 0$, the intervals $[-1\mathrm{/}{h}_m,0]$ increase to $(-\mathrm{\infty },0]$. Monotone convergence [F6] and the half-mass calculation in step 1.1 give ${\int }_{-1\mathrm{/}{h}_m}^{0}K(u)du⟶{\int }_{-\mathrm{\infty }}^{0}K(u)du=\frac{1}{2}\mathrm{.}$

The density version was chosen with $f(0)=1$, so step 3.1 gives $\mathbb{E}{\widehat{f}}_h(0)-f(0)\to -1\mathrm{/}2$. The selected density is discontinuous at $0$, so this calculation does not satisfy the smooth interior-point theorem's hypotheses.