A finite MDP policy generates action-state-reward trajectories

The blocks $\{s\}\times A(s)$ are finite by [F3] and pairwise disjoint because distinct blocks have different first coordinates. Since $S$ is finite, [F3] makes $D={\bigcup }_{s\in S}\{s\}\times A(s)$ finite. Its product with the finite reward and state sets is finite by [F3], so $Z$ is finite. For each $s\in S$ and $B\subseteq Z$, define $K_{\pi }(s,B)={\sum }_{a\in A(s)}\pi (a\mid s){\sum }_{s^{\mathrm{\prime }}\in S}{\sum }_{r\in \mathcal{R}}{\mathbf{1}}_B(s,a,r,s^{\mathrm{\prime }})p(s^{\mathrm{\prime }},r\mid s,a)\mathrm{.}$ Every term is nonnegative, and [F1] gives $K_{\pi }(s,Z)={\sum }_{a\in A(s)}\pi (a\mid s)=1$. If $(B_j)$ is a disjoint sequence of subsets of finite $Z$, only finitely many $B_j$ are nonempty: enumerate $Z$ and assign each nonempty $B_j$ its least element in that enumeration; disjointness makes these assigned elements distinct. Therefore finite additivity of the displayed finite sums gives countable additivity. Since $Z$ and $S$ have discrete sigma-algebras, $B\mapsto K_{\pi }(s,B)$ is a probability measure and $s\mapsto K_{\pi }(s,B)$ is measurable; hence $K_{\pi }$ is a probability kernel by [F4].

Set $E_0=S$ and $E_n=Z$ for $n\ge 1$, with initial measure ${\delta }_{s_0}$. For each $n\ge 0$, let $K_n$ from $E_0\times \cdots \times E_n$ to $Z$ be $K_n(h,B)=K_{\pi }(s_n,B)$, where $s_n=s_0$ when $n=0$ and $s_n$ is the final next-state coordinate of the last trajectory record in $h$ when $n\ge 1$. That coordinate projection is measurable by [F5], so each $K_n$ is measurable in $h$; its sections are probability measures by step 1.1. Thus the hypotheses of [F7] hold.

Apply [F7] to obtain the probability law with these finite-prefix kernels. For a feasible atom $(s_0,a_0,r_1,s_1,\dots ,a_{m-1},r_m,s_m)$, its iterated prefix mass is the product over $n<m$ of the atom masses $K_{\pi }(s_n,\{(s_n,a_n,r_{n+1},s_{n+1})\})$. The definition of $K_{\pi }$ makes each such mass $\pi (a_n\mid s_n)p(s_{n+1},r_{n+1}\mid s_n,a_n)$, giving the displayed formula. For $m=0$, the initial point mass is $1$.

Fix a finite state-reward path $(s_0,r_1,s_1,\dots ,r_m,s_m)$. Summing the action-path formula from step 3.1 over its finitely many action sequences and regrouping the finite sums by time gives the displayed state-reward marginal formula by [F9]. These are the finite-prefix probabilities in [F8]. The initial state is the same point mass, so uniqueness [F10] identifies the state-reward marginal with the stationary-policy path law.