Differentiable stationary stochastic policies and score vectors

Let a finite Markov decision process have nonempty finite state set $S$ and finite nonempty action set $A(s)$ at each state. Let $U\subseteq {\mathbb{R}}^d$ be open, with $d\ge 1$. A differentiable stationary policy family on $U$ is a family of probabilities ${\pi }_{\theta }(a\mid s)$, indexed by $\theta \in U$, $s\in S$, and $a\in A(s)$, such that ${\sum }_{a\in A(s)}{\pi }_{\theta }(a\mid s)=1$, each map $\theta \mapsto {\pi }_{\theta }(a\mid s)$ is totally differentiable, and for each $s$ there is a fixed nonempty support $B(s)\subseteq A(s)$ with ${\pi }_{\theta }(a\mid s)>0$ for every $a\in B(s)$ and $\theta \in U$, while ${\pi }_{\theta }(a\mid s)=0$ for every $a\mathrm{\notin }B(s)$ and $\theta \in U$. The score vector at a supported state-action pair is

${\psi }_{\theta }(s,a):={\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(a\mid s),a\in B(s),$

where the gradient is the Euclidean vector of partial derivatives. The fixed-support condition makes this logarithm defined throughout $U$ on every action that can be sampled.

A finite state set, finite nonempty state-dependent action sets, and stationary policies given by normalized action probabilities.

Total differentiability is the linear first-order approximation with a remainder of smaller order.

The gradient of a scalar-valued map is the vector of its coordinate partial derivatives.

The total derivative of a composition is the composition of the total derivatives.

The logarithm is differentiable at positive arguments with derivative equal to their reciprocal.