The likelihood-ratio score gradient is unbiased on fixed support

Fix nonzero $h\in {\mathbb{R}}^d$ with ${\theta }_0+h\in U$. For $x\mathrm{\notin }N$, [F3] and [F4] applied to $t\mapsto p_{{\theta }_0+th}(x)$ give $p_{{\theta }_0+h}(x)-p_{{\theta }_0}(x)={\int }_0^1⟨\mathrm{\nabla }{p}_{{\theta }_0+th}(x),h⟩dt\mathrm{.}$ With $R_h(x):=H(x)(p_{{\theta }_0+h}(x)-p_{{\theta }_0}(x)-⟨{\mathrm{\nabla }}_{\theta }{p}_{{\theta }_0}(x),h⟩)$, subtracting the derivative at $t=0$ and applying [F7] gives $\frac{\mathrm{\mid }{R}_h(x)\mathrm{\mid }}{\mathrm{\parallel }h{\mathrm{\parallel }}_2}\le \mathrm{\mid }H(x)\mathrm{\mid }{\sup }_{0\le t\le 1}\mathrm{\parallel }\mathrm{\nabla }{p}_{{\theta }_0+th}(x)-\mathrm{\nabla }{p}_{{\theta }_0}(x){\mathrm{\parallel }}_2\mathrm{.}$

The right side of step 1.1 has a measurable version: outside $N$, continuity in $t$ makes its supremum equal to the supremum over rational $t\in [0,1]$; set it to zero on $N$. This weighted supremum is bounded almost everywhere by $2G(x)$ by the envelope in the Statement and tends to zero for $\nu$-almost every $x$ as $h\to 0$. Thus [F5] shows $\int \mathrm{\mid }{R}_h\mathrm{\mid }d\nu \mathrm{/}\mathrm{\parallel }h{\mathrm{\parallel }}_2\to 0$ along every sequence of nonzero $h\to 0$. The envelope makes each coordinate of $H\mathrm{\nabla }{p}_{{\theta }_0}$ integrable, and the hypotheses make $H{p}_{\eta }$ integrable. By [F9], for $h\mathrm{\ne }0$, $J({\theta }_0+h)-J({\theta }_0)-⟨{\int }_{\mathcal{X}}H\mathrm{\nabla }{p}_{{\theta }_0}d\nu ,h⟩={\int }_{\mathcal{X}}{R}_{h}d\nu$. Therefore the Fréchet derivative is $\mathrm{\nabla }J({\theta }_0)={\int }_{\mathcal{X}}H(x)\mathrm{\nabla }{p}_{{\theta }_0}(x)d\nu (x)$.

On the common support $S$, $p_{{\theta }_0}>0$ and [F1] gives $\mathrm{\nabla }{p}_{{\theta }_0}=p_{{\theta }_0}{S}_{{\theta }_0}$. Off $S$, the density is identically zero on $U$, so its derivative is zero almost everywhere. For each coordinate put ${\phi }_j(x):=H(x)S_{{\theta }_0,j}(x)$. The weighted second-moment hypothesis makes ${\phi }_j$ square-integrable under $P_{{\theta }_0}$, hence integrable since $2\mathrm{\mid }{\phi }_j\mathrm{\mid }\le 1+{\phi }_j^2$ on a probability space. Its positive and negative parts are measurable and nonnegative; by [F2] and [F8], ${\mathbb{E}}_{{\theta }_0}[{\phi }_j]={\mathbb{E}}_{{\theta }_0}[{\phi }_j^{+}]-{\mathbb{E}}_{{\theta }_0}[{\phi }_j^{-}]=\int {\phi }_j^{+}{p}_{{\theta }_0}d\nu -\int {\phi }_j^{-}{p}_{{\theta }_0}d\nu =\int {\phi }_j{p}_{{\theta }_0}d\nu \mathrm{.}$ This explicitly extends the nonnegative density rule to the signed integrand. Substitution in step 2.1 therefore gives $\mathrm{\nabla }J({\theta }_0)={\mathbb{E}}_{{\theta }_0}[H(X)S_{{\theta }_0}(X)]\mathrm{.}$

The weighted second-moment assumption makes each coordinate $H(X)S_{{\theta }_0,j}(X)$ square-integrable. [F6] gives finite variance, while step 3.1 gives its expectation as the corresponding objective derivative.