Performance-difference identity for an arbitrary bounded baseline

Fix a policy $\sigma$. For each $t$, let $e_t^{\sigma }$ be the finite transition-mass sum of $g$ at time $t$, and let $h_t^{\sigma }:={\sum }_s{\mu }_t^{\sigma }(s)f(s)$. By [F1] and the definition of ${\delta }_f^g$, the finite sum of ${\delta }_f^g$ at time $t$ is $e_t^{\sigma }+\gamma h_{t+1}^{\sigma }-h_t^{\sigma }$. Summing after multiplication by ${\gamma }^t$ for $t=0,\dots ,N$ telescopes to ${\sum }_{t=0}^N{\gamma }^t{e}_t^{\sigma }-h_0^{\sigma }+{\gamma }^{N+1}{h}_{N+1}^{\sigma }$.

The remainder satisfies $\mathrm{\mid }{\gamma }^{N+1}{h}_{N+1}^{\sigma }\mathrm{\mid }\le {\gamma }^{N+1}\mathrm{\parallel }f{\mathrm{\parallel }}_{\mathrm{\infty }}\to 0$ by [F4] and [F5]. Also $\mathrm{\mid }{\delta }_f^g\mathrm{\mid }\le \mathrm{\parallel }g{\mathrm{\parallel }}_{\mathrm{\infty }}+(1+\gamma )\mathrm{\parallel }f{\mathrm{\parallel }}_{\mathrm{\infty }}$, so [F4] makes its discounted series convergent. Passing to the limit in step 1.1 and using [F2]–[F3] gives $J_g^{\sigma }(\mu )={\sum }_s\mu (s)f(s)+(1-\gamma {)}^{-1}{\sum }_s{d}^{\sigma }(s){\bar{\delta }}_f^{g,\sigma }(s)$.

Apply the identity in step 2.1 to $\sigma ={\pi }^{\mathrm{\prime }}$ and $\sigma =\pi$ and subtract; the initial terms ${\sum }_s\mu (s)f(s)$ cancel. Add and subtract ${\sum }_s{d}^{\pi }(s){\bar{\delta }}_f^{g,{\pi }^{\mathrm{\prime }}}(s)$. The first resulting term is $L_{\pi ,f}^g({\pi }^{\mathrm{\prime }})$ by its definition, and the second is ${\sum }_s(d^{{\pi }^{\mathrm{\prime }}}(s)-d^{\pi }(s)){\bar{\delta }}_f^{g,{\pi }^{\mathrm{\prime }}}(s)$. Dividing by $1-\gamma >0$ gives the claimed identity.