Same-sample coefficient remainder is of order $O_P(n^{-1})$

The entries $h_j(X_i)$, $f(X_i)$, $h_j(X_i)h_k(X_i)$, and $h_j(X_i)f(X_i)$ are measurable by [F11]. Since each $X_i$ has law $P$, applying the same measurable function to each gives a common law; [F10] preserves their mutual independence, so each resulting sequence is IID by [F2]. The fourth-moment hypothesis and [F8] make every product entry integrable, and [F9] identifies its expectation with the corresponding integral under $P$. Applying [F4] to these IID sequences gives ${\bar{h}}_n\to \theta$, ${\bar{f}}_n\to Pf$, $n^{-1}{\sum }_{i}h(X_i)h(X_i{)}^{\mathsf{T}}\to E_P[h{h}^{\mathsf{T}}]$, and $n^{-1}{\sum }_{i}h(X_i)f(X_i)\to E_P[hf]$ almost surely. The centered formulas therefore give ${\widehat{\mathrm{\Sigma }}}_n\to \mathrm{\Sigma }$ and ${\widehat{c}}_n\to c$ almost surely; positive definiteness of $\mathrm{\Sigma }$ and continuity of determinant imply ${\widehat{\mathrm{\Sigma }}}_n$ is eventually invertible almost surely. The fallback definition makes ${\widehat{\beta }}_n$ measurable: on the open nonsingular set the inverse entries are ratios of polynomial functions, and on its closed complement the value is constant.

Put $W_i=h(X_i)-\theta$ and $V_i=f(X_i)-Pf$. The finite fourth-moment assumptions imply finite fourth moments for every component of $W_i$ and for $V_i$; for example, $\mathrm{\mid }x-a{\mathrm{\mid }}^4\le 8(\mathrm{\mid }x{\mathrm{\mid }}^4+\mathrm{\mid }a{\mathrm{\mid }}^4)$. Then [F8] gives finite second moments for every product $W_{ij}{W}_{ik}$ and $W_{ij}{V}_i$. Each centered coordinate and product is a fixed measurable function of $X_i$, so [F2] and [F10] make each sequence IID. By [F9] and [F7], $E{W}_i=0$, $E{V}_i=0$, and the product means are $\mathrm{\Sigma }$ and $c$. Independence and [F6] give variance equal to the one-observation variance divided by $n$ for each sample average. Applying [F5] with $a=M\mathrm{/}\sqrt{n}$ and taking a finite union over the fixed $q$ coordinates and matrix entries proves ${\bar{h}}_n-\theta =O_P(n^{-1\mathrm{/}2})$, ${\bar{f}}_n-Pf=O_P(n^{-1\mathrm{/}2})$, $n^{-1}{\sum }_i{W}_i{W}_i^{\mathsf{T}}-\mathrm{\Sigma }=O_P(n^{-1\mathrm{/}2})$, and $n^{-1}{\sum }_i{W}_i{V}_i-c=O_P(n^{-1\mathrm{/}2})$. The identities ${\widehat{\mathrm{\Sigma }}}_n=\frac{1}{n}{\sum }_i{W}_i{W}_i^{\mathsf{T}}-({\bar{h}}_n-\theta )({\bar{h}}_n-\theta {)}^{\mathsf{T}},{\widehat{c}}_n=\frac{1}{n}{\sum }_i{W}_i{V}_i-({\bar{h}}_n-\theta )({\bar{f}}_n-Pf)$ show that the centering corrections are $O_P(n^{-1})$. Thus ${\widehat{\mathrm{\Sigma }}}_n-\mathrm{\Sigma }$ and ${\widehat{c}}_n-c$ are jointly $O_P(n^{-1\mathrm{/}2})$.

On the event that ${\widehat{\mathrm{\Sigma }}}_n$ is invertible, ${\widehat{\beta }}_n-{\beta }^{\ast }={\widehat{\mathrm{\Sigma }}}_n^{-1}[({\widehat{c}}_n-c)-({\widehat{\mathrm{\Sigma }}}_n-\mathrm{\Sigma }){\beta }^{\ast }]$, since $c=\mathrm{\Sigma }{\beta }^{\ast }$. Step 1.1 and the adjugate-over-determinant formula show that the inverse converges to ${\mathrm{\Sigma }}^{-1}$ almost surely on the eventually nonsingular set; extending it by zero on the singular event gives an $O_P(1)$ matrix sequence. Given $\epsilon >0$, choose bounds for this inverse norm and for the bracket in step 1.2 after scaling by $\sqrt{n}$, with total failure probability less than $\epsilon \mathrm{/}2$. On their intersection the product is bounded after scaling by $\sqrt{n}$, so ${\widehat{\beta }}_n-{\beta }^{\ast }=O_P(n^{-1\mathrm{/}2})$ there. The singular-event probability tends to zero by step 1.1; the fallback value $0$ therefore does not change this order in probability.

By the plug-in definition in the Statement and [F1] for the fixed coefficient ${\beta }^{\ast }$, subtracting the two estimators cancels ${\bar{f}}_n$ and leaves exactly $-({\widehat{\beta }}_n-{\beta }^{\ast }{)}^{\mathsf{T}}({\bar{h}}_n-\theta )$. Given $\epsilon >0$, choose finite bounds for each factor after scaling by $\sqrt{n}$ so that the probability either bound fails is less than $\epsilon \mathrm{/}2$; on their intersection, Cauchy--Schwarz in ${\mathbb{R}}^q$ bounds $n$ times the absolute inner product by the product of those bounds. Thus the inner product is $O_P(n^{-1})$ by definition.