Finite MDP with bounded real rewards

Let $S$ be a finite nonempty state set and let $A(s)$ be a finite nonempty action set for each $s\in S$, as in the finite-MDP convention. Put $D=\{(s,a):s\in S,a\in A(s)\}$ with its discrete sigma-algebra. Equip $S$ with its power-set sigma-algebra, $[0,1]$ with its Borel sigma-algebra, and $S\times [0,1]$ with the product sigma-algebra. A finite MDP with bounded real rewards specifies a probability kernel $Q$ from $D$ to $S\times [0,1]$; equivalently, each admissible $(s,a)$ has a probability measure $Q(\cdot \mid s,a)$ on that product measurable space. The reward-coordinate projection $(s^{\mathrm{\prime }},\rho )\mapsto \rho$ is measurable because the preimage of each Borel set $B\subseteq [0,1]$ is the product rectangle $S\times B$. Define the next-state law by $P(s^{\mathrm{\prime }}\mid s,a)=Q(\{s^{\mathrm{\prime }}\}\times [0,1]\mid s,a)$ and the mean reward by $r(s,a)={\int }_{S\times [0,1]}\rho Q(d{s}^{\mathrm{\prime }},d\rho \mid s,a)\in [0,1]$. At each time, conditional on the observed history and the selected action, the next-state/reward pair has this law. Assume the Axiom of Choice for the path-space construction under policies.

Finite-MDP supplier: supplies the finite state/action and joint one-step-law conventions; its reward alphabet is finite, while this definition explicitly permits every Borel reward law on $[0,1]$.

Borel sigma-algebra: specifies the Borel events on the reward interval.

Product sigma-algebra: specifies measurable events on $S\times [0,1]$.

Probability measure: $Q(\cdot \mid s,a)$ is a probability measure for each admissible pair.

Probability kernel: the family $Q$ has measurable event sections over the finite pair set $D$.

Expectation as an integral: defines the mean reward integral.

UCRL2 model and reward scope: Introduction, finite state/action, reward, and transition model (PDF lines 34–41).

Ionescu–Tulcea path-law construction: unique path law from a specified initial probability law and sequential probability kernels under Choice.

Axiom of Choice: assumed by the path-law construction.

The Axiom of Choice is assumed exactly for the path-space construction under policies. The one-step kernel and its finite state marginal require no choice.

For a measurable history-dependent policy and an initial state distribution $\mu$ on $S$, start the construction with $S_0\sim \mu$ (use the point mass ${\delta }_x$ when starting at a specified state $x$). Alternate the policy action kernels with the observation kernels $Q(\cdot \mid s,a)$. The cited Ionescu–Tulcea theorem then supplies a unique interaction law for this initial distribution and policy. The MDP kernel alone does not specify an initial distribution.