A state-dependent action-independent baseline preserves the gradient mean

Given: The hypotheses and notation of Discounted episodic policy gradients from trajectory likelihoods, its Axiom of Choice premise, and a fixed finite-valued state function $b$.

The Axiom of Choice: every family of nonempty sets has a choice function; the policy trajectory law and conditional-expectation supplier use this axiom.

Discounted episodic policy gradients from trajectory likelihoods: under its stated trajectory-differentiation and integrability assumptions, $\mathbb{E}[{\sum }_{t<T}{\gamma }^t{G}_t{\psi }_t]=\mathrm{\nabla }J({\theta }_0)$.

A policy score has zero conditional mean before the action: for the pre-action history ${\mathcal{H}}_t$, $\mathbb{E}[{\psi }_t\mid {\mathcal{H}}_t]=0$ almost surely.

Finite Markov decision process with joint reward-transition law and Differentiable stationary stochastic policies and score vectors: the state and action sets are finite, and at fixed ${\theta }_0$ every score vector on the common support is finite; hence the finitely many score vectors are bounded.

Episodic returns, temporal-difference errors, and policy support: $0\le \gamma <1$, episodes terminate almost surely, and $G_t$ is the discounted return from time $t$.

Taking out what is known: an integrable variable measurable with respect to ${\mathcal{H}}_t$ can be taken outside conditional expectation when its product is integrable.

For $\mathrm{\mid }r\mathrm{\mid }<1$, ${\sum }_{k\ge 0}{r}^k=1\mathrm{/}(1-r)$, and for $\mathrm{\mid }r\mathrm{\mid }\ge 1$ the series diverges: for $0\le \gamma <1$, ${\sum }_{t=0}^{\mathrm{\infty }}{\gamma }^t=1\mathrm{/}(1-\gamma )$.

Dominated convergence: almost-sure convergence under a single integrable majorant permits convergence of expectations.