One-step episodic actor–critic with a bootstrapped critic

Inputs: a finite episodic MDP with nonempty finite state set $S$, finite nonempty action sets $A(s)$, finite reward set $\mathcal{R}$, joint transition-reward law $p$, terminal set $S_{†}\subseteq S$, and initial state $s_0\in S$; a discount $0\le \gamma <1$; a differentiable stationary policy family ${\pi }_{\theta }$ with fixed positive support on an open set $U\subseteq {\mathbb{R}}^d$; a value approximation ${\widehat{V}}_w:S\to \mathbb{R}$ differentiable in $w$ on an open set $W\subseteq {\mathbb{R}}^m$; initial parameters $({\theta }_0,w_0)\in U\times W$; and step sizes $\alpha ,\beta \ge 0$.

Admissible conditions: assume AC for the adaptive episode law; every proposed actor and critic update remains in $U$ and $W$, respectively, and the episode generated by this procedure, including its within-episode parameter updates, reaches $S_{†}$ after finitely many transitions almost surely.

For the sampling instruction, condition on the complete finite transition history and use the current parameters computed from that history. The successive-draw law uses these history-dependent kernels on discrete transition-record spaces, with a fixed stop symbol padding the records after termination.

Output: the final actor and critic parameters $({\theta }_{\mathrm{o}\mathrm{u}\mathrm{t}},w_{\mathrm{o}\mathrm{u}\mathrm{t}})$ after one episode.

Set $(\theta ,w)=({\theta }_0,w_0)$, $S=s_0$, and $I=1$. If $S\in S_{†}$, return $({\theta }_0,w_0)$. Otherwise, while $S\mathrm{\notin }{S}_{†}$, using the current pre-update parameters $(\theta ,w)$:

1. Sample $A\sim {\pi }_{\theta }(\cdot \mid S)$ and then sample $(S^{\mathrm{\prime }},R)$ with joint law $p(S^{\mathrm{\prime }},R\mid S,A)$.
2. Set $C=0$ if $S^{\mathrm{\prime }}\in S_{†}$ and $C={\widehat{V}}_w(S^{\mathrm{\prime }})$ otherwise. Compute the temporal-difference error $\delta =R+\gamma C-{\widehat{V}}_w(S)$.
3. Compute both updates from these pre-update parameters: ${\theta }^{\mathrm{\prime }}=\theta +\alpha I\delta {\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(A\mid S)$ and $w^{\mathrm{\prime }}=w+\beta \delta {\mathrm{\nabla }}_w{\widehat{V}}_w(S)$. Replace $(\theta ,w)$ by $({\theta }^{\mathrm{\prime }},w^{\mathrm{\prime }})$.
4. Set $I\leftarrow \gamma I$ and $S\leftarrow S^{\mathrm{\prime }}$.

When the loop ends, return the current $(\theta ,w)$ as $({\theta }_{\mathrm{o}\mathrm{u}\mathrm{t}},w_{\mathrm{o}\mathrm{u}\mathrm{t}})$.

The critic uses a one-step bootstrapped target; the actor uses the current TD error and the accumulated start-state discount $I={\gamma }^t$. The procedure itself asserts no convergence result.

The finite state-dependent action sets and joint reward-transition law.

The differentiable policy family and action-score vectors.

The terminal convention and discounted episodic TD-error notation.

Under AC, the law of successive draws with history-dependent probability kernels on finite discrete transition records.

The choice premise for the adaptive episode law.

The total derivatives used for policy and value-approximation parameters.

The coordinate gradient convention.

Replacing the full Monte Carlo return by a bootstrapped critic target can change the expected actor direction when the critic is inaccurate. This definition supplies no convergence guarantee for tabular, nonlinear, or deep function approximation.