Known-state/action MDP with prescribed absorbing values

Let $M$ be a bounded-reward finite discounted MDP with discount $\gamma$, let $K\subseteq D$ be a set of known state-action pairs, and let $q:D\to [0,V_{\max }]$ with $V_{\max }\le (1-\gamma {)}^{-1}$. The known-state/action MDP $M_K(q)$ agrees with $M$ on every pair in $K$. For each pair $(s,a)\mathrm{\notin }K$, add a distinct absorbing state ${\mathrm{\perp }}_{s,a}$ with one admissible action. The unknown pair transitions deterministically to ${\mathrm{\perp }}_{s,a}$ and gives deterministic reward $(1-\gamma )q(s,a)$; the sink action self-loops with the same deterministic reward $(1-\gamma )q(s,a)$. These rewards lie in $[0,1]$. The sink state's value is $q(s,a)$, so the unknown pair's Bellman backup is exactly $(1-\gamma )q(s,a)+\gamma q(s,a)=q(s,a)$, independent of the value function on the original states.

The finite discounted MDP and its admissible state-action pairs.

The normalized discounted-value cap.

Strehl, Li, and Littman, Reinforcement Learning in Finite MDPs: PAC Analysis.