Finite-response score mean and reward-gradient identities

Given: A fixed prompt $q$, its finite response list and support, the differentiable normalized policy, and a fixed finite-valued reward function $r$.

Finite truncated response spaces, correctness predicates, and verifiers: the support $(b_1,\dots ,b_m)$ is finite and fixed; each supported mass is positive and totally differentiable, the masses sum to one, and $S_{\theta }(q,b_j)={\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(b_j\mid q)$.

Differentiable stationary stochastic policies and score vectors: a supported policy mass is positive and totally differentiable on its open parameter domain, and its score is the gradient of its logarithm.

The chain rule for total derivatives: $D(g\circ f)(a)=Dg(f(a))\circ Df(a)$ and The natural logarithm has derivative 1/x and equals the integral from 1 to x of 1/t: for positive ${\pi }_{\theta }(b_j\mid q)$, the chain rule gives ${\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(b_j\mid q)={\mathrm{\nabla }}_{\theta }{\pi }_{\theta }(b_j\mid q)\mathrm{/}{\pi }_{\theta }(b_j\mid q)$.

Finite sums and finite products, by recursion and Sums and scalar multiples of totally differentiable maps are totally differentiable with the expected derivatives: a sum over $m$ terms is defined by recursion, and induction using the binary sum rule differentiates it term by term.

The total (Fréchet) derivative $Df(a)$ as the linear first-order approximation with $o(\mathrm{\parallel }h{\mathrm{\parallel }}_2)$ remainder and The Jacobian matrix of partial derivatives and the gradient in the scalar-valued case: total differentiability supplies the derivative map, and the gradient is its scalar-valued coordinate vector.

Use one-based coordinate labels $X_a:=X(a-1)$ for vectors in ${\mathbb{R}}^d$, $1\le a\le d$, and ${\mathrm{\partial }}_{{\theta }_a}:={\mathrm{\partial }}_{a-1}$ in the supplier gradient convention. Every score vector lies in ${\mathbb{R}}^d$; write $S_{\theta ,a}(q,b_j):=(S_{\theta }(q,b_j){)}_a$ for its $a$th coordinate. For a finite random vector $X$, write $X_a$ for its $a$th coordinate and define ${\mathbb{E}}_{\theta }[X\mid q]$ as the vector whose $a$th coordinate is the finite scalar sum ${\sum }_{j=1}^m{\pi }_{\theta }(b_j\mid q)X_a(b_j)$ for $a=1,\dots ,d$; vector sums and equalities are coordinatewise.