A finite-dimensional QMD score has mean zero

Write $q=\sqrt{p_{{\theta }_0}}$ and $r_h=\sqrt{p_{{\theta }_0+h}}-q-\frac{1}{2}{h}^T{s}_{{\theta }_0}q$. By [F1], $\mathrm{\parallel }{r}_h{\mathrm{\parallel }}_2=o(\mathrm{\parallel }h\mathrm{\parallel })$. By [F3], each $q_h=\sqrt{p_{{\theta }_0+h}}$ is a probability square-root density, so $\mathrm{\parallel }{q}_h{\mathrm{\parallel }}_2^2=\mathrm{\parallel }q{\mathrm{\parallel }}_2^2=1$. Since $P_{{\theta }_0}$ is a probability and the score is square-integrable, [F2] with the constant function $1$ shows its coordinates are integrable. All products in the following norm expansion are integrable by [F2], since $q$, $r_h$, and $h^T{s}_{{\theta }_0}q$ belong to $L^2(\mu )$. Thus [F4] permits integrating the finite expansion term by term, and [F3] transfers the score term to expectation under $P_{{\theta }_0}$. Expanding $q_h=q+\frac{1}{2}{h}^T{s}_{{\theta }_0}q+r_h$ gives $0=h^T{\mathbb{E}}_{{\theta }_0}{s}_{{\theta }_0}+2⟨q,r_h⟩+\mathrm{\parallel }\frac{1}{2}{h}^T{s}_{{\theta }_0}q+r_h{\mathrm{\parallel }}_2^2$. Cauchy–Schwarz [F2] bounds the inner product by $o(\mathrm{\parallel }h\mathrm{\parallel })$; square-integrability of the score and $\mathrm{\parallel }{r}_h{\mathrm{\parallel }}_2=o(\mathrm{\parallel }h\mathrm{\parallel })$ make the last term $O(\mathrm{\parallel }h{\mathrm{\parallel }}^2)$. Thus $h^T{\mathbb{E}}_{{\theta }_0}{s}_{{\theta }_0}=o(\mathrm{\parallel }h\mathrm{\parallel })$.

For each coordinate vector $e_j$, set $h=t{e}_j$ with $t>0$ small enough that ${\theta }_0+t{e}_j\in \mathrm{\Theta }$. Divide the relation from step 1.1 by $t$ and let $t\downarrow 0$ to obtain ${\mathbb{E}}_{{\theta }_0}{s}_{{\theta }_0,j}=0$. This holds for all $j=1,\dots ,d$, hence the score vector has mean zero.