The belief state is a sufficient statistic for finite POMDP control

Fix $h_t$ and $a$ as in the statement. By [F2], $b_t(s)=\mathrm{Pr}(S_t=s\mid h_t)$. Conditioning first on $S_t$ and using the POMDP's specified one-step law [F1] gives $\mathrm{Pr}(S_{t+1}=s^{\mathrm{\prime }},O_{t+1}=o\mid h_t,A_t=a)={\sum }_{s\in S}{b}_t(s)T(s^{\mathrm{\prime }}\mid s,a)O(o\mid s^{\mathrm{\prime }})$. The action selection does not alter $b_t$ after conditioning on $h_t$, because its randomization depends only on that history and is conditionally independent of the hidden state.

Summing the joint probability in step 1.1 over $s^{\mathrm{\prime }}$ yields $q(o\mid b_t,a)$ by [F2]. If this is positive, dividing the joint mass by it gives $\mathrm{Pr}(S_{t+1}=s^{\mathrm{\prime }}\mid h_t,A_t=a,O_{t+1}=o)=\tau (b_t,a,o)(s^{\mathrm{\prime }})$; if it is zero, the observation event has probability zero and no conditional quotient is used. By [F3], these positive branches form the probability kernel $K$.

Conditioning the immediate reward on the current hidden state and chosen action gives $\mathbb{E}[R_t\mid h_t,A_t=a]={\sum }_{s\in S}{b}_t(s)r(s,a)=\bar{r}(b_t,a)$ by [F1] and [F2]. Thus the conditional immediate reward and next-belief law depend on the history only through $b_t$ and $a$, which proves the stated controlled belief-state transition and reward description.

We prove the finite-horizon recursion by induction on $n$. With no decisions remaining, the reward sum is empty, so $V_0(b)=0$. Suppose the optimal $n$-step continuation value is the function $V_n$ of the current belief. If action $a$ is taken at belief $b$, its immediate expected reward is $\bar{r}(b,a)$ by step 2.2, and for each positive-probability observation $o$ the remaining belief is $\tau (b,a,o)$ by step 2.1. Conditional expectation over the finite observation set therefore gives the $n+1$ step action value $Q_{n+1}(b,a):=\bar{r}(b,a)+\gamma {\sum }_{o:q>0}q(o\mid b,a)V_n(\tau (b,a,o))$. The set $\{Q_{n+1}(b,a):a\in A\}$ is a nonempty finite image of $A$ in $\mathbb{R}$, so [F4] gives an attained maximum. Taking that maximum gives the displayed $V_{n+1}(b)$. The expression depends only on $b$, completing the induction and proving history sufficiency for every finite horizon.

If the action at a history is randomized with probabilities ${\lambda }_a$, its expected action value is ${\sum }_{a\in A}{\lambda }_a{Q}_{n+1}(b,a)$, where $Q_{n+1}$ is the finite action value from step 3.1, ${\lambda }_a\ge 0$, and ${\sum }_a{\lambda }_a=1$. Since each $Q_{n+1}(b,a)\le {\max }_{c\in A}{Q}_{n+1}(b,c)$, [F5] gives ${\sum }_a{\lambda }_a{Q}_{n+1}(b,a)\le {\max }_{c\in A}{Q}_{n+1}(b,c)$. Randomization cannot improve the finite-horizon value.