IID bootstrap and jackknife for smooth statistics

For each coordinate $j$, the sequence $(X_{ij}{)}_i$ is IID by [F1] and [F4], and is integrable since $\mathbb{E}\mathrm{\mid }{X}_{1j}\mathrm{\mid }\le 1+\mathbb{E}\mathrm{\mid }{X}_{1j}{\mathrm{\mid }}^2<\mathrm{\infty }$. For each pair $j,k$, $(X_{ij}{X}_{ik}{)}_i$ is also IID by [F1] and [F4], and is integrable because $2\mathrm{\mid }{X}_{1j}{X}_{1k}\mathrm{\mid }\le X_{1j}^2+X_{1k}^2$. Apply [F3] to each of these finitely many sequences. This gives ${\bar{X}}_{n,j}\to {\mu }_j$ and $n^{-1}{\sum }_i{X}_{ij}{X}_{ik}\to \mathbb{E}[X_{1j}{X}_{1k}]$ in probability.

For each $j,k$, expand ${\widehat{\mathrm{\Sigma }}}_{n,jk}=\frac{1}{n}{\sum }_i{X}_{ij}{X}_{ik}-{\bar{X}}_{n,j}{\bar{X}}_{n,k}\mathrm{.}$ Step 1.1 makes the right side converge in probability to $\mathbb{E}[X_{1j}{X}_{1k}]-{\mu }_j{\mu }_k={\mathrm{\Sigma }}_{jk}$; the same step gives the claimed mean convergence. Since $d$ is finite, entrywise convergence holds simultaneously in probability.

Formula [F1] gives $P_n(\mathrm{\varnothing })=0$ and $P_n({\mathbb{R}}^d)=n^{-1}{\sum }_{i=1}^{n}1=1$.

Let $(A_k{)}_{k\ge 1}$ be pairwise disjoint Borel sets. For each fixed $i$, at most one indicator $1\{x_i\in A_k\}$ is nonzero. Therefore $P_n\text{\hspace{0.17em}⁣}\left({\bigcup }_{k\ge 1}{A}_k\right)=\frac{1}{n}{\sum }_{i=1}^n{\sum }_{k\ge 1}1\{x_i\in A_k\}={\sum }_{k\ge 1}\frac{1}{n}{\sum }_{i=1}^{n}1\{x_i\in A_k\}={\sum }_{k\ge 1}{P}_n(A_k)\mathrm{.}$ The interchange is between a finite sum and a nonnegative series; by [F4] both are the supremum of the corresponding finite partial sums, so finite distributivity gives the equality.

Steps 1.1 and 1.2 verify the empty-set and countable-additivity clauses in [F2], and step 1.1 gives total mass one. Thus [F3] makes $P_n$ a probability measure.

By [F1], ${\bar{X}}_n\to \mu$ and $n^{-1}{\sum }_i\mathrm{\parallel }{X}_i-{\bar{X}}_n{\mathrm{\parallel }}^2=\mathrm{tr}({\widehat{\mathrm{\Sigma }}}_n)$ is bounded in probability. For each $\epsilon >0$, [F2] and the finite second moment give $\mathbb{P}\text{\hspace{0.17em}⁣}({\max }_{i\le n}\mathrm{\parallel }{X}_i\mathrm{\parallel }>\epsilon n)\le n\mathbb{P}(\mathrm{\parallel }{X}_1\mathrm{\parallel }>\epsilon n)\le \frac{\mathbb{E}\mathrm{\parallel }{X}_1{\mathrm{\parallel }}^2}{{\epsilon }^{2}n}⟶0.$ Therefore ${\max }_i\mathrm{\parallel }{X}_i-{\bar{X}}_n\mathrm{\parallel }\mathrm{/}(n-1)\to 0$ in probability. Choose $\delta >0$ so the closed Euclidean ball $\bar{B}(\mu ,2\delta )$ is contained in $U$, and let $E_n=\{\mathrm{\parallel }{\bar{X}}_n-\mu \mathrm{\parallel }\le \delta ,{\max }_i\mathrm{\parallel }{X}_i-{\bar{X}}_n\mathrm{\parallel }\mathrm{/}(n-1)\le \delta \}$. This is measurable and has probability tending to one by the preceding bounds. Extend $\mathrm{\nabla }g$ by zero outside $U$ to a Borel map $\widetilde{\mathrm{\nabla }g}:{\mathbb{R}}^d\to {\mathbb{R}}^d$, and define ${\omega }_n={\max }_{i\le n}{\sup }_{t\in \mathbb{Q}\cap [0,1]}{1}_{E_n}\parallel \widetilde{\mathrm{\nabla }g}\text{\hspace{0.17em}⁣}({\bar{X}}_n-t\frac{X_i-{\bar{X}}_n}{n-1})-a\parallel \mathrm{.}$ Each rational-indexed term is measurable, and the countable supremum is measurable because its strict superlevel sets are countable unions; the finite maximum is measurable as well. On $E_n$, every segment lies in $\bar{B}(\mu ,2\delta )\subset U$, where $\mathrm{\nabla }g$ is bounded by continuity on this compact ball, and continuity along the segment makes the rational supremum equal the supremum over $[0,1]$. On $E_n^c$ every term inside the supremum is zero. The greatest distance of any segment point from $\mu$ on $E_n$ is at most $\mathrm{\parallel }{\bar{X}}_n-\mu \mathrm{\parallel }+{\max }_i\mathrm{\parallel }{X}_i-{\bar{X}}_n\mathrm{\parallel }\mathrm{/}(n-1)$, which tends to zero in probability. Continuity of $\mathrm{\nabla }g$ at $\mu$ and $\mathbb{P}(E_n)\to 1$ therefore imply ${\omega }_n\to 0$ in probability.

On $E_n$, define ${\phi }_i(t)=g({\bar{X}}_n-t(X_i-{\bar{X}}_n)\mathrm{/}(n-1))$ for $0\le t\le 1$. It is continuous on $[0,1]$ and differentiable on $(0,1)$ by [F5], with derivative $-(n-1{)}^{-1}\mathrm{\nabla }g({\bar{X}}_n-t(X_i-{\bar{X}}_n)\mathrm{/}(n-1){)}^T(X_i-{\bar{X}}_n)$. By [F4] there is ${\tau }_i\in (0,1)$ at which this derivative equals ${\phi }_i(1)-{\phi }_i(0)$. On every outcome define the measurable algebraic remainder $e_{n,i}:={\widehat{\theta }}_{n,(-i)}-{\widehat{\theta }}_n+a^T(X_i-{\bar{X}}_n)\mathrm{/}(n-1)$. On $E_n$ the mean-value identity expresses this remainder as $-\frac{(\mathrm{\nabla }g({\bar{X}}_n-{\tau }_i(X_i-{\bar{X}}_n)\mathrm{/}(n-1))-a{)}^T(X_i-{\bar{X}}_n)}{n-1}$, so $\mathrm{\mid }{e}_{n,i}\mathrm{\mid }\le {\omega }_n\mathrm{\parallel }{X}_i-{\bar{X}}_n\mathrm{\parallel }\mathrm{/}(n-1)$. The points ${\tau }_i$ are used only for this pointwise identity and bound; no measurable selection is needed. Since $\mathbb{P}(E_n)\to 1$, the expansion with the stated bound holds with probability tending to one.

The algebraic definition of $e_{n,i}$ gives $p_{n,i}={\widehat{\theta }}_n+a^T(X_i-{\bar{X}}_n)-(n-1)e_{n,i}$. On $E_n$, the bound in step 2.1 makes the average squared error in the statement at most ${\omega }_n^2\frac{1}{n}{\sum }_i\mathrm{\parallel }{X}_i-{\bar{X}}_n{\mathrm{\parallel }}^2,$ which tends to zero in probability by step 1.1. Since $\mathbb{P}(E_n^c)\to 0$, the stated convergence in probability follows.

The algebraic definition of $e_{n,i}$ gives, after averaging over $i$ and using ${\sum }_i(X_i-{\bar{X}}_n)=0$, $n{V}_{J,n}=\frac{1}{n-1}{\sum }_i{(a^T(X_i-{\bar{X}}_n)-(n-1)(e_{n,i}-{\bar{e}}_n))}^2,$ where ${\bar{e}}_n=n^{-1}{\sum }_i{e}_{n,i}$. The leading quadratic term is $\frac{1}{n-1}{\sum }_i\{a^T(X_i-{\bar{X}}_n){\}}^2=\frac{n}{n-1}{a}^T{\widehat{\mathrm{\Sigma }}}_{n}a⟶a^T\mathrm{\Sigma }a$ by [F1]. On $E_n$, the normalized squared error term is bounded by $(n-1){\sum }_i(e_{n,i}-{\bar{e}}_n{)}^2\le (n-1){\sum }_i{e}_{n,i}^2\le {\omega }_n^2\frac{1}{n-1}{\sum }_i\mathrm{\parallel }{X}_i-{\bar{X}}_n{\mathrm{\parallel }}^2⟶0.$ If ${\psi }_i=a^T(X_i-{\bar{X}}_n)$ and $r_i=(n-1)(e_{n,i}-{\bar{e}}_n)$, then on $E_n$ the absolute cross term is at most $2\sqrt{\left(\frac{1}{n-1}{\sum }_i{\psi }_i^2\right)\left(\frac{1}{n-1}{\sum }_i{r}_i^2\right)},$ which tends to zero because the first factor converges to the finite $a^T\mathrm{\Sigma }a$ and the second to zero. Since $\mathbb{P}(E_n^c)\to 0$, these bounds prove the jackknife variance limit in probability.

Fix an outcome and let $u_1<\cdots <u_m$ be the distinct values among the finite list $(R_{n,\omega }{)}_{\omega \in {\mathsf{I}}_n}$. Write $F_n^{\ast }(u_j^{-})=n^{-n}\mathrm{\#}\{\eta :R_{n,\eta }<u_j\}$. On $[u_j,u_{j+1})$, $F_n^{\ast }$ is constant at $F_n^{\ast }(u_j)$ and $G$ is nondecreasing, so the absolute difference is bounded by the larger of its two endpoint errors, $\mathrm{\mid }{F}_n^{\ast }(u_j)-G(u_j)\mathrm{\mid }$ and $\mathrm{\mid }{F}_n^{\ast }(u_{j+1}^{-})-G(u_{j+1})\mathrm{\mid }$. For $t<u_1$, the error is at most $\mathrm{\mid }{F}_n^{\ast }(u_1^{-})-G(u_1)\mathrm{\mid }$; for $t\ge u_m$, it is at most $\mathrm{\mid }{F}_n^{\ast }(u_m)-G(u_m)\mathrm{\mid }$. Each left-limit error is approached as $t\uparrow u_j$, and each right-value error occurs at $u_j$. Therefore $D_n={\max }_{\omega \in {\mathsf{I}}_n}\max \{\mathrm{\mid }{F}_n^{\ast }(R_{n,\omega })-G(R_{n,\omega })\mathrm{\mid },\mathrm{\mid }{F}_n^{\ast }(R_{n,\omega }^{-})-G(R_{n,\omega })\mathrm{\mid }\}$, where $F_n^{\ast }(r^{-})=n^{-n}\mathrm{\#}\{\eta :R_{n,\eta }<r\}$.

For each fixed tuple $\omega$, $R_{n,\omega }$ is a finite linear combination of measurable coordinates, hence measurable by [F2]. Thus each comparison event $\{R_{n,\eta }\le R_{n,\omega }\}$ or $\{R_{n,\eta }<R_{n,\omega }\}$ is measurable by applying [F2] to the measurable difference and its Borel half-line. Consequently $F_n^{\ast }(R_{n,\omega })$ and $F_n^{\ast }(R_{n,\omega }^{-})$ are measurable finite sums; $G(R_{n,\omega })$ is measurable by [F4]. The finite maximum in step 1.1 is measurable by [F2], so $D_n$ is measurable.

For subsets $C_1,\dots ,C_n\subseteq \{1,\dots ,n\}$, the product-uniform index law gives ${\mathbb{P}}^{\ast }(I_1\in C_1,\dots ,I_n\in C_n)=\frac{{\prod }_{j=1}^n\mathrm{\mid }{C}_j\mathrm{\mid }}{n^n}={\prod }_{j=1}^n\frac{\mathrm{\mid }{C}_j\mathrm{\mid }}{n}\mathrm{.}$ Thus the coordinate indices are independent and each is uniform on $\{1,\dots ,n\}$.

For each $j$, the map $i\mapsto x_i$ from the finite discrete index set to ${\mathbb{R}}^d$ is measurable, and its pushforward of the uniform law assigns $A\in \mathcal{B}({\mathbb{R}}^d)$ mass $n^{-1}{\sum }_{i}1\{x_i\in A\}=P_n(A)$. Applying [F3] to the independent indices shows the values $X_j^{\ast }=x_{I_j}$ are independent; their common law is $P_n$, which is a probability law by [F2].

Put $Y_i=a^T(X_i-\mu )$ and ${\bar{Y}}_n=n^{-1}{\sum }_i{Y}_i$. Its variance is ${\sigma }^2$. By [F2], the empirical variance $s_n^2=\frac{1}{n}{\sum }_{i=1}^n(Y_i-{\bar{Y}}_n{)}^2=a^T{\widehat{\mathrm{\Sigma }}}_{n}a$ converges in probability to ${\sigma }^2>0$, while ${\bar{Y}}_n\to 0$ in probability and $n^{-1}{\sum }_i{Y}_i^2\to {\sigma }^2$ in probability.

For $t>0$, define the unnormalized empirical tail quantity $T_n(t)=\frac{1}{n}{\sum }_{i=1}^n(Y_i-{\bar{Y}}_n{)}^{2}1\{\mathrm{\mid }{Y}_i-{\bar{Y}}_n\mathrm{\mid }>t\sqrt{n}\}\mathrm{.}$ On $\mathrm{\mid }{\bar{Y}}_n\mathrm{\mid }\le t\sqrt{n}\mathrm{/}2$, the indicator implies $\mathrm{\mid }{Y}_i\mathrm{\mid }>t\sqrt{n}\mathrm{/}2$, and $T_n(t)\le 2\frac{1}{n}{\sum }_i{Y}_i^{2}1\{\mathrm{\mid }{Y}_i\mathrm{\mid }>t\sqrt{n}\mathrm{/}2\}+2{\bar{Y}}_n^2\frac{1}{n}{\sum }_{i}1\{\mathrm{\mid }{Y}_i\mathrm{\mid }>t\sqrt{n}\mathrm{/}2\}\mathrm{.}$ The expectation of the first term tends to zero by [F7], so [F8] makes it converge to zero in probability. The second empirical tail fraction is at most $4(t^{2}n{)}^{-1}(n^{-1}{\sum }_i{Y}_i^2)$; hence its product with $2{\bar{Y}}_n^2$ tends to zero in probability by [F2]. The complement event has probability tending to zero by [F2], so $T_n(t)\to 0$ in probability for every $t>0$.

Conditionally on the data, the row variables $Z_{n\mathrm{\ell }}^{\ast }=n^{-1\mathrm{/}2}{a}^T(X_{\mathrm{\ell }}^{\ast }-{\bar{X}}_n)$, $1\le \mathrm{\ell }\le n$, are independent and centered by [F1]. They are finite conditional rows of the type in [F3], with total variance $s_n^2$. By [F4], when $s_n>0$ their normalized Lindeberg quantity at threshold $\epsilon >0$ is $s_n^{-2}{T}_n(\epsilon s_n)$. On $s_n\ge \sigma \mathrm{/}2$, this is at most $4{\sigma }^{-2}{T}_n(\epsilon \sigma \mathrm{/}2)$, which tends to zero in probability by step 1.2; also $s_n^2\to {\sigma }^2$ in probability by [F2].

Take any subsequence, indexed $(n_k{)}_{k\ge 1}$. Recursively, after $k_0=0$, let $k_j$ be the least integer $k>k_{j-1}$ for which $\mathbb{P}(\mathrm{\mid }{s}_{n_k}^2-{\sigma }^2\mathrm{\mid }>1\mathrm{/}j)+{\sum }_{r=1}^j\mathbb{P}(T_{n_k}(1\mathrm{/}r)>1\mathrm{/}j)<2^{-j}$. For fixed $j$, each of these finitely many probabilities tends to zero as $k\to \mathrm{\infty }$ by [F2] and step 1.2, so the qualifying set is nonempty; [F12] gives its least index.

Let $E_j$ be the union of the variance-deviation event and the $j$ tail events at index $k_j$. By [F9], $\mathbb{P}(E_j)<2^{-j}$ and $\mathbb{P}({\bigcup }_{j\ge m}{E}_j)\le {\sum }_{j\ge m}{2}^{-j}=2^{1-m}$. The event that infinitely many $E_j$ occur is contained in every such tail union, so its probability is zero. Off this null event, $s_{n_{k_j}}^2\to {\sigma }^2$ and, for every fixed $r\ge 1$, $T_{n_{k_j}}(1\mathrm{/}r)\to 0$. For any $t>0$, choose $r$ with $1\mathrm{/}r<t$ by [F13]; since $T_n$ is nonincreasing, $0\le T_{n_{k_j}}(t)\le T_{n_{k_j}}(1\mathrm{/}r)\to 0$.

Along almost every selected data path, $s_{n_{k_j}}\to \sigma >0$, and for every $\epsilon >0$ the Lindeberg quantity is eventually bounded by $4{\sigma }^{-2}{T}_{n_{k_j}}(\epsilon \sigma \mathrm{/}2)\to 0$. Thus the deterministic conditional rows satisfy [F5], giving ${\sum }_{\mathrm{\ell }=1}^{n_{k_j}}{Z}_{n_{k_j},\mathrm{\ell }}^{\ast }\mathrm{/}{s}_{n_{k_j}}\Rightarrow N(0,1)$. Every starting subsequence has a further subsequence with this almost-sure conditional convergence.

Along each data sequence from step 4.1, put $m_j=n_{k_j}$ and let $H_j$ be the conditional CDF of ${\sum }_{\mathrm{\ell }=1}^{m_j}{Z}_{m_j,\mathrm{\ell }}^{\ast }\mathrm{/}{s}_{m_j}$, discarding finitely many rows before $s_{m_j}>0$. By [F6], $H_j(t)\to \mathrm{\Phi }(t)$ for every real $t$. Fix $x\in \mathbb{R}$ and $h>0$. Since $s_{m_j}\to \sigma >0$, eventually $x\mathrm{/}\sigma -h\le x\mathrm{/}{s}_{m_j}\le x\mathrm{/}\sigma +h$, so monotonicity gives $H_j(x\mathrm{/}\sigma -h)\le F_{m_j}^{\ast }(x)=H_j(x\mathrm{/}{s}_{m_j})\le H_j(x\mathrm{/}\sigma +h)$. Letting $j\to \mathrm{\infty }$ and then $h\downarrow 0$, continuity in [F10] yields $F_{m_j}^{\ast }(x)\to \mathrm{\Phi }(x\mathrm{/}\sigma )={\mathrm{\Phi }}_{\sigma }(x)$. To make this uniform, fix $\eta >0$, choose $R$ with ${\mathrm{\Phi }}_{\sigma }(-R)<\eta$ and $1-{\mathrm{\Phi }}_{\sigma }(R)<\eta$, and partition $[-R,R]$ into finitely many intervals on which ${\mathrm{\Phi }}_{\sigma }$ varies by less than $\eta$, using [F11]. Monotonicity of each conditional CDF bounds its error between grid points by the endpoint errors plus $\eta$; outside $[-R,R]$ the two tail probabilities add at most $2\eta$ plus endpoint errors. The maximum grid error tends to zero, and then $\eta \downarrow 0$ proves almost-sure uniform CDF convergence along the selected subsequence. Since $D_n$ is measurable by [F14], this is almost-sure convergence of real random variables.