The optimal action value is attained by a greedy stationary policy

The unique fixed point $V^{\ast }$ exists by [F6]. Let $B=\mathrm{\parallel }{V}^{\ast }{\mathrm{\parallel }}_{\mathrm{\infty }}$, which is finite because $S$ is finite. Fix $s\in S$ and enumerate $A(s)=\{a_0,\dots ,a_n\}$ using [F1] and [F14]. For $u_j=r(s,a_j)+\gamma {\sum }_{x}P(x\mid s,a_j)V^{\ast }(x)$, [F15] gives an attained list maximum; because the list enumerates all actions, its maximum is exactly the maximum in [F4], independent of the enumeration by uniqueness in [F15]. For each $j$, expanding $r$ and $P$ by [F1] gives $u_j={\sum }_{x,\rho }p(x,\rho \mid s,a_j)[\rho +\gamma V^{\ast }(x)]$. The triangle inequality, $\mathrm{\mid }\rho \mathrm{\mid }\le M$, and the probability-mass identity give $\mathrm{\mid }{u}_j\mathrm{\mid }\le M+\gamma B$. Since the maximum is one of the list entries, $\mathrm{\mid }(T^{\ast }{V}^{\ast })(s)\mathrm{\mid }\le M+\gamma B$. As $s$ was arbitrary, $B=\mathrm{\parallel }{T}^{\ast }{V}^{\ast }{\mathrm{\parallel }}_{\mathrm{\infty }}\le M+\gamma B$. Since $1-\gamma >0$, $B\le M\mathrm{/}(1-\gamma )$.

Fix any admissible continuation from $(s,a)$. For each $n\ge 1$, expand $R_{n+1}+\gamma V^{\ast }(S_{n+1})$ as the finite sum ${\sum }_{s^{\mathrm{\prime }},\rho }(\rho +\gamma V^{\ast }(s^{\mathrm{\prime }})){\mathbf{1}}_{\{S_{n+1}=s^{\mathrm{\prime }},R_{n+1}=\rho \}}$. These variables are bounded by [F1] and step 1.1, so they are integrable. Conditional linearity and the transition-reward identity in the definition give $E[R_{n+1}+\gamma V^{\ast }(S_{n+1})\mid {\mathcal{F}}_n]={\sum }_{s^{\mathrm{\prime }},\rho }p(s^{\mathrm{\prime }},\rho \mid S_n,A_n)[\rho +\gamma V^{\ast }(s^{\mathrm{\prime }})]$. The right side is the one-step action value of the measurable current action $A_n$, so it is at most $(T^{\ast }{V}^{\ast })(S_n)=V^{\ast }(S_n)$ by [F4] and the fixed-point identity.

The attained list maximum argument in step 1.1 makes the maximizing-action set nonempty at every state. By [A1], choose one such action at every state and call the resulting deterministic stationary policy ${\pi }^{\ast }$. Define its stationary probability-family embedding by ${\widehat{\pi }}^{\ast }(b\mid s)={\mathbf{1}}_{\{b={\pi }^{\ast }(s)\}}$ for $b\in A(s)$. These numbers are nonnegative and sum to one because exactly one legal action equals ${\pi }^{\ast }(s)$, so ${\widehat{\pi }}^{\ast }$ is a stationary Markov policy in [F5]–[F8]. Its policy Bellman sum has only the chosen maximizing action as a nonzero term. The definitions [F4]–[F5] and $T^{\ast }{V}^{\ast }=V^{\ast }$ give $T^{{\widehat{\pi }}^{\ast }}{V}^{\ast }=V^{\ast }$. By [F6] this is the unique fixed point of $T^{{\widehat{\pi }}^{\ast }}$, while [F8] gives $T^{{\widehat{\pi }}^{\ast }}{V}^{{\widehat{\pi }}^{\ast }}=V^{{\widehat{\pi }}^{\ast }}$; hence $V^{{\widehat{\pi }}^{\ast }}=V^{\ast }$.

Apply order and expectation preservation from [F3] to the conditional inequality in step 2.1, multiply by the nonnegative scalar ${\gamma }^n$, and sum for $n=1,\dots ,N$. The intermediate state-value terms cancel, giving ${\sum }_{n=1}^N{\gamma }^{n}E[R_{n+1}]+{\gamma }^{N+1}E[V^{\ast }(S_{N+1})]\le \gamma E[V^{\ast }(S_1)]$. The initial transition law gives $E[R_1+\gamma V^{\ast }(S_1)]={\sum }_{s^{\mathrm{\prime }},\rho }p(s^{\mathrm{\prime }},\rho \mid s,a)[\rho +\gamma V^{\ast }(s^{\mathrm{\prime }})]=:Q^{†}(s,a)$. Since $\mathrm{\mid }E[V^{\ast }(S_{N+1})]\mathrm{\mid }\le B$ by step 1.1 and [F13], it follows that $E[{\sum }_{n=0}^N{\gamma }^n{R}_{n+1}]\le Q^{†}(s,a)+{\gamma }^{N+1}B$.

Write $G_N={\sum }_{n=0}^N{\gamma }^n{R}_{n+1}$. Each $G_N$ is measurable because it is a finite sum of measurable rewards, and [F11] gives pointwise convergence to $G$ and the bound $\mathrm{\mid }{G}_N\mathrm{\mid }\le M\mathrm{/}(1-\gamma )$. Thus $G$ is measurable by [F17]; the constant bound is integrable by [F13], so [F12] gives $E[G_N]\to E[G]$. Also ${\gamma }^{N+1}B\to 0$ by [F16]. Passing to the limit in the upper bound of step 3.1 shows $E[G]\le Q^{†}(s,a)$. The continuation was arbitrary, so $Q_{\mathrm{o}\mathrm{p}\mathrm{t}}(s,a)\le Q^{†}(s,a)$.

Construct a continuation with initial pair $(s,a)$ and thereafter action ${\pi }^{\ast }(S_{n+1})$ following each transition. Take $E_0=D$ and $E_{n+1}=\mathcal{R}\times D$ with their discrete sigma-algebras, initial law concentrated at $(s,a)$, and history kernels $K_n(h,B)={\sum }_{s^{\mathrm{\prime }},\rho }p(s^{\mathrm{\prime }},\rho \mid s_n,a_n){\mathbf{1}}_B(\rho ,(s^{\mathrm{\prime }},{\pi }^{\ast }(s^{\mathrm{\prime }})))$ for every measurable $B\subseteq E_{n+1}$, where $(s_n,a_n)$ is the last pair in $h$. For disjoint sets $B_j\subseteq E_{n+1}$, their preimages $\{(s^{\mathrm{\prime }},\rho ):(\rho ,(s^{\mathrm{\prime }},{\pi }^{\ast }(s^{\mathrm{\prime }})))\in B_j\}$ are disjoint subsets of the finite set $S\times \mathcal{R}$, so only finitely many are nonempty and the finite mass formula is countably additive; its total mass is one by [F1]. It is measurable in $h$ because it is a finite sum of functions of the current-pair coordinate projection, which is measurable by [F18]. Thus each $K_n$ is a probability kernel by [F19], and [F9] gives a path law with these prefix products. Set ${\mathcal{F}}_n=\sigma (X_0,Y_1,\dots ,Y_n)$, where $X_0=(S_0,A_0)$ and $Y_j=(R_j,(S_j,A_j))$. The coordinate projections make current pairs measurable at time $n$ and next observations measurable at time $n+1$ by [F18]. The finite history-coordinate set is finite by [F20], so each atom of ${\mathcal{F}}_n$ fixes a finite history. On a positive-mass atom ending at $(s_n,a_n)$, the prefix product in [F9] shows that extending it by reward $\rho$, state $s^{\mathrm{\prime }}$, and next action ${\pi }^{\ast }(s^{\mathrm{\prime }})$ has conditional probability $p(s^{\mathrm{\prime }},\rho \mid s_n,a_n)$; all other next-action extensions have conditional probability zero. Thus the conditional probability of $\{R_{n+1}=\rho ,S_{n+1}=s^{\mathrm{\prime }}\}$ given ${\mathcal{F}}_n$ is $p(s^{\mathrm{\prime }},\rho \mid S_n,A_n)$ on every positive-mass atom, which proves the required almost-sure conditional identity. Hence this process is admissible. By the product form of the finite-prefix probabilities, after the first transition its conditional continuation from $S_1=s^{\mathrm{\prime }}$ has the stationary-policy prefix laws in [F10] for ${\widehat{\pi }}^{\ast }$: the next action equals ${\pi }^{\ast }$ at its current state with probability one, so [F7] identifies its conditional expected discounted tail as $V^{{\widehat{\pi }}^{\ast }}(s^{\mathrm{\prime }})$. The expectation convergence proved for every admissible continuation in step 4.1 passes this prefix identity to the infinite return. Therefore $E[G]={\sum }_{s^{\mathrm{\prime }},\rho }p(s^{\mathrm{\prime }},\rho \mid s,a)[\rho +\gamma V^{{\widehat{\pi }}^{\ast }}(s^{\mathrm{\prime }})]=Q^{†}(s,a)$ by step 2.2. This admissible continuation attains the upper bound, so $Q_{\mathrm{o}\mathrm{p}\mathrm{t}}(s,a)=Q^{†}(s,a)$ and the supremum is a finite maximum.

Since $(s,a)$ was arbitrary, the formula and finite attainment claim hold on all of $D$.