Finite-horizon policy ratios are trajectory Radon-Nikodym derivatives

For a complete trajectory atom $\tau =(s_0,a_0,r_1,\dots ,a_{H-1},r_H,s_H)$, [F1] gives $P_H^{\sigma }(\{\tau \})={\prod }_{k=0}^{H-1}\sigma (a_k\mid s_k)p(s_{k+1},r_{k+1}\mid s_k,a_k)$. If this atom has positive target mass, coverage makes its behavior mass positive. The initial state and every transition/reward factor are common, so on every behavior-positive atom the quotient $P_H^{\pi }(\{\tau \})\mathrm{/}{P}_H^{\mu }(\{\tau \})$ is exactly ${\prod }_{k=0}^{H-1}\pi (a_k\mid s_k)\mathrm{/}\mu (a_k\mid s_k)$; if the target mass is zero, this quotient is zero.

Fix $t<H$ and project the full trajectory law onto ${\mathrm{\Omega }}_{t+1}$. If a prefix set is behavior-null, its inverse image in ${\mathrm{\Omega }}_H$ is behavior-null; full coverage makes that inverse image target-null, so $P_{t+1}^{\pi }\ll P_{t+1}^{\mu }$. The prefix space is finite. Let ${\lambda }_{t+1}$ be counting measure on it; $P_{t+1}^{\mu }\ll {\lambda }_{t+1}$, and the whole prefix space is a common finite exhaustion for ${\lambda }_{t+1}$, $P_{t+1}^{\mu }$, and $P_{t+1}^{\pi }$. By [A1], [F4]–[F6] give the RN density classes and their chain rule. On each behavior-positive prefix, [F1]–[F2] give the atom-mass quotient by cancelling the common initial and transition/reward factors, leaving ${\prod }_{k=0}^t\pi (A_k\mid S_k)\mathrm{/}\mu (A_k\mid S_k)$. Thus $W_{0:t}=d{P}_{t+1}^{\pi }\mathrm{/}d{P}_{t+1}^{\mu }$ almost everywhere.

Fix $t<H$ and a real $g$ integrable under $P_{t+1}^{\pi }$. Step 2.1 supplies the prefix density $W_{0:t}$. Applying [F7] to $g^{+}$ and $g^{-}$ gives ${\mathbb{E}}_{P_H^{\mu }}[W_{0:t}\mathrm{\mid }g\mathrm{\mid }]={\mathbb{E}}_{P_{t+1}^{\pi }}\mathrm{\mid }g\mathrm{\mid }<\mathrm{\infty }$ and equality of the two weighted integrals for each part. Subtracting the finite positive and negative integrals by [F8] proves the prefix expectation identity. Taking $t=H-1$ and $g=G$ gives the full-horizon assertion.