A policy score has zero conditional mean before the action

Fix $t$ and a positive-probability atom $h$ of ${\mathcal{H}}_t$ contained in $\{T>t\}$, and let $s$ be its current state. By [F3], adjoining action $a$ and next outcome $(s^{\mathrm{\prime }},r)$ multiplies the mass of $h$ by ${\pi }_{\theta }(a\mid s)p(s^{\mathrm{\prime }},r\mid s,a)$. Summing over the finite next-state and reward sets and using [F1] gives $\mathrm{Pr}(A_t=a\mid h)={\pi }_{\theta }(a\mid s)$. On atoms contained in $\{T\le t\}$, ${\psi }_t=0$ by definition; on zero-probability atoms no conditional value is prescribed.

Enumerate the finite support as $B(s)=\{a_0,\dots ,a_{m-1}\}$. By induction on the number of terms, [F5] gives the derivative of this finite sum as the sum of the derivatives: the empty sum has derivative zero, and adjoining one term uses the binary sum rule. For each positive-probability history $h\subseteq \{T>t\}$ ending at $s$, the conditional score mean is ${\sum }_{a\in B(s)}{\pi }_{\theta }(a\mid s){\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(a\mid s)={\sum }_{a\in B(s)}{\mathrm{\nabla }}_{\theta }{\pi }_{\theta }(a\mid s)={\mathrm{\nabla }}_{\theta }1=0$. The first equality uses positivity on the support and the derivative of the logarithm; the second uses [F5], [F6], and the normalization in [F2].

To verify that this mean is a conditional-expectation version, note that ${\mathcal{H}}_t$ has finitely many atoms because its state, action, and reward alphabets are finite. The score coordinates are bounded at fixed $\theta$, since there are finitely many state-action pairs. On each positive-probability atom $h$ of ${\mathcal{H}}_t$, define $Y_t$ to have the constant value ${\mathbb{P}}_{\theta }(h{)}^{-1}{\int }_h{\psi }_{t}d{\mathbb{P}}_{\theta }$; set $Y_t=0$ on every null atom. If $h\subseteq \{T>t\}$, step 1.1 and the displayed calculation show that this value is zero; if $h\subseteq \{T\le t\}$, ${\psi }_t=0$ on $h$ by definition, so the value is again zero. Thus $Y_t$ is the zero, integrable, ${\mathcal{H}}_t$-measurable vector. Every event in ${\mathcal{H}}_t$ is a union of finitely many atoms, and summing the atom integral identities gives ${\int }_E{Y}_{t}d{\mathbb{P}}_{\theta }={\int }_E{\psi }_{t}d{\mathbb{P}}_{\theta }$ for every $E\in {\mathcal{H}}_t$, including null atoms. This verifies [F4], so $Y_t$ is a version of ${\mathbb{E}}_{\theta }[{\psi }_t\mid {\mathcal{H}}_t]$ and is zero almost surely.