Episodic REINFORCE with a frozen episode policy

Inputs: a finite episodic MDP with nonempty finite state set $S$, finite nonempty action sets $A(s)$, finite reward set $\mathcal{R}$, joint transition-reward law $p$, terminal set $S_{†}\subseteq S$, and initial state $s_0\in S$; a discount $0\le \gamma <1$; a differentiable stationary policy family ${\pi }_{\theta }$ with fixed positive support on an open set $U\subseteq {\mathbb{R}}^d$; an initial parameter ${\theta }_0\in U$; and a step size $\alpha \ge 0$.

Admissible conditions: the episode sampled from $s_0$ under the frozen policy ${\pi }_{{\theta }_0}$ reaches $S_{†}$ after finitely many transitions almost surely, and the proposed update remains in $U$.

Output: the updated policy parameter ${\theta }_1$ after one training round.

Hold $\theta ={\theta }_0$ fixed while sampling one episode $(S_0,A_0,R_1,\dots ,S_T)$ from $s_0$ until its first terminal state; if $s_0\in S_{†}$, take $T=0$. For each $t<T$, compute the return $G_t={\sum }_{k=t}^{T-1}{\gamma }^{k-t}{R}_{k+1}$ and score ${\psi }_t={\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(A_t\mid S_t)$ using $\theta ={\theta }_0$. After the episode is complete, set

${\theta }_1={\theta }_0+\alpha {\sum }_{t=0}^{T-1}{\gamma }^t{G}_t{\psi }_t$

and return ${\theta }_1$. All returns, scores, and the full sum use the same parameter value ${\theta }_0$ that generated the episode; no policy update occurs while that episode is being sampled.

The finite state-dependent action sets and joint reward-transition law.

A differentiable policy family with fixed positive support and score vectors.

The almost-surely terminating episode and discounted return from each time step.

The action-state-reward trajectory law for a finite MDP policy under AC.

The expected discounted trajectory-score update under the stated differentiation and integrability hypotheses.

Under the theorem's assumptions, the sample direction ${\sum }_{t=0}^{T-1}{\gamma }^t{G}_t{\psi }_t$ has expectation ${\mathrm{\nabla }}_{\theta }J({\theta }_0)$, including the theorem's trajectory-differentiation and Axiom of Choice premises. The expected parameter increment is therefore $\alpha {\mathrm{\nabla }}_{\theta }J({\theta }_0)$. This identity does not imply convergence of repeated nonlinear or deep parameter updates.

The discounted episodic policy-gradient identity and its hypotheses.