Finite-episode importance sampling has the target-policy mean under support

Fix a finite terminal trajectory $\omega =(s_0,a_0,r_1,s_1,\dots ,a_{m-1},r_m,s_m)$ with $s_0=s$ and $s_m\in S_{†}$. By [F2], $P_{\pi ,s}(\omega )={\prod }_{k=0}^{m-1}\pi (a_k\mid s_k)p(s_{k+1},r_{k+1}\mid s_k,a_k),P_{\mu ,s}(\omega )={\prod }_{k=0}^{m-1}\mu (a_k\mid s_k)p(s_{k+1},r_{k+1}\mid s_k,a_k)\mathrm{.}$ If $P_{\pi ,s}(\omega )>0$, every transition-reward factor and every target action factor is positive. Target support then makes every corresponding behavior action factor positive, so $P_{\mu ,s}(\omega )>0$. On each behavior-positive trajectory, canceling the shared transition-reward factors gives $P_{\pi ,s}(\omega )={\mathrm{\Lambda }}_{0:m-1}(\omega )P_{\mu ,s}(\omega )$; if its target probability is zero, the ratio is zero.

For each episode length $m$, there are finitely many terminal trajectories because the state, action, and reward alphabets are finite. These terminal-path events are disjoint over all lengths and cover probability one under both policies by the termination hypothesis. Enumerate them by increasing length and, within each length, any finite enumeration. For any finite initial collection $C$ of these paths, step 1.1 gives ${\sum }_{\omega \in C}{P}_{\mu ,s}(\omega )\mathrm{\Lambda }(\omega )\mathrm{\mid }G(\omega )\mathrm{\mid }={\sum }_{\omega \in C}{P}_{\pi ,s}(\omega )\mathrm{\mid }G(\omega )\mathrm{\mid }\mathrm{.}$ As the collections increase to all terminal paths, [F6] passes both nonnegative sums to their expectations. Thus $E_{\mu ,s}[\mathrm{\Lambda }\mathrm{\mid }G\mathrm{\mid }]=E_{\pi ,s}[\mathrm{\mid }G\mathrm{\mid }]<\mathrm{\infty }$, so $\mathrm{\Lambda }G$ is integrable.

Apply the same finite-path identity separately to the nonnegative functions $G^{+}=\max (G,0)$ and $G^{-}=\max (-G,0)$; step 2.1 makes both weighted expectations finite. Subtracting the two equalities by linearity of the integral gives $E_{\mu ,s}[\mathrm{\Lambda }G]=E_{\pi ,s}[G]$. For $G=G_0$, the target expectation is $V^{\pi }(s)$ by [F4].

Let $H$ be one finite behavior history ending at a first visit to $s$ at time $t$, with $P_{\mu ,s_0}(H)>0$. For every finite terminal suffix $\omega$ starting at $s$, the product formula [F2] factors the joint probability as $P_{\mu ,s_0}(H)P_{\mu ,s}(\omega )$. The suffix likelihood ratio contains only action factors from time $t$ onward, so step 1.1 gives the target-to-behavior ratio for that suffix. Since $G_t$ is the bounded discounted suffix return and target termination holds from $s$, step 3.1 gives $E_{\mu ,s}[{\mathrm{\Lambda }}_{0:T-1}{G}_0]=V^{\pi }(s)$. Summing over suffixes therefore yields $E_{\mu ,s_0}[{\mathbf{1}}_H{\mathrm{\Lambda }}_{t:T-1}{G}_t]=P_{\mu ,s_0}(H)V^{\pi }(s)$. Dividing by the positive $P_{\mu ,s_0}(H)$ proves the conditional first-visit assertion.