Trajectory and per-decision importance sampling are unbiased under coverage

On any input where the algorithm returns numerical estimates, each propensity ratio is finite: when $\mu (A_t^i\mid S_t^i)>0$, it is a quotient of finite probabilities by a positive probability; when both probabilities are zero, the algorithm sets the ratio to zero. If $\mu (A_t^i\mid S_t^i)=0<\pi (A_t^i\mid S_t^i)$, the algorithm returns unsupported instead. Thus every cumulative weight is finite. Each return and every finite sum in OIS and PDIS is finite because the reward set, horizon, and dataset are finite. The WIS numerator and denominator are finite, and the procedure either divides by a positive finite denominator or returns zero.

An observed pair with $\mu (A_t^i\mid S_t^i)=0$ lies on a behavior-null trajectory atom by [F1]. Since the trajectory space, horizon, and sample size are finite, the event that any observed pair has zero behavior propensity is a finite union of null events. The algorithm therefore returns numerical estimates almost surely under the behavior sample law. Extending the three estimator expressions by any fixed finite values on the unsupported null event does not change their expectations.

Fix a logged trajectory index $i$. By [F1], its law is $P_H^{\mu }$, and [F3] applied to $G_i$ gives ${\mathbb{E}}_{\mu }[W_{i,H-1}{G}_i]={\mathbb{E}}_{\pi }[G_H]=J_H(\pi ;s_0)$. Linearity [F4] gives ${\mathbb{E}}_{\mu }[{\widehat{J}}_{\mathrm{O}\mathrm{I}\mathrm{S}}]=n^{-1}{\sum }_{i=1}^n{J}_H(\pi ;s_0)=J_H(\pi ;s_0)$.

Fix $i$ and $0\le t<H$. The reward $R_{t+1}^i$ is a function of the prefix through $(R_{t+1},S_{t+1})$, so [F3] gives ${\mathbb{E}}_{\mu }[W_{i,t}{R}_{t+1}^i]={\mathbb{E}}_{\pi }[R_{t+1}]$. Multiplying each equality by the fixed ${\gamma }^t$ and summing the finite range $0\le t<H$, [F4] yields ${\mathbb{E}}_{\mu }[{\sum }_{t=0}^{H-1}{\gamma }^t{W}_{i,t}{R}_{t+1}^i]=J_H(\pi ;s_0)$.

Each of the $n$ PDIS trajectory scores in [F2] has expectation $J_H(\pi ;s_0)$ by step 3.2. Their finite sample average has the same expectation by [F4], proving ${\mathbb{E}}_{\mu }[{\widehat{J}}_{\mathrm{P}\mathrm{D}\mathrm{I}\mathrm{S}}]=J_H(\pi ;s_0)$. Step 3.1 proves the ordinary-estimator claim.