State-weighted Bellman error and projected Bellman error

Let $S$ be a nonempty finite state set of a finite Markov decision process, let $0\le \gamma <1$, let $\pi$ be a stationary Markov policy, and let $\mu :S\to [0,1]$ satisfy ${\sum }_{s\in S}\mu (s)=1$. The function $\mu$ is a fixed state-weighting distribution; it need not be stationary for $\pi$. Let $d$ be a positive integer, let $\varphi :S\to {\mathbb{R}}^d$ be a feature map, and for $w\in {\mathbb{R}}^d$ put $v_w(s)=\varphi (s{)}^{\mathsf{T}}w$.

For real functions $f,g:S\to \mathbb{R}$, write $f{\sim }_{\mu }g$ when ${\sum }_{s\in S}\mu (s)(f(s)-g(s){)}^2=0$. Let $[f{]}_{\mu }$ denote the equivalence class of $f$ and define $⟨[f{]}_{\mu },[g{]}_{\mu }{⟩}_{\mu }={\sum }_{s\in S}\mu (s)f(s)g(s)$ and $\mathrm{\parallel }[f{]}_{\mu }{\mathrm{\parallel }}_{\mu }^2=⟨[f{]}_{\mu },[f{]}_{\mu }{⟩}_{\mu }$. Let $H_{\mu }$ be the resulting finite-dimensional real inner-product space, and let $V_{\mu }$ be the span in $H_{\mu }$ of the equivalence classes of the $d$ coordinate feature functions.

For $h\in H_{\mu }$, let ${\mathrm{\Pi }}_{\mu }h$ be the unique vector in $V_{\mu }$ whose difference from $h$ is orthogonal to every vector in $V_{\mu }$. The Bellman-error vector for $v_w$ is ${\delta }_w=[T^{\pi }{v}_w-v_w{]}_{\mu }$. Define the mean square Bellman error and mean square projected Bellman error by ${\mathrm{BE}}_{\mu }(w)=\mathrm{\parallel }{\delta }_w{\mathrm{\parallel }}_{\mu }^2$ and ${\mathrm{PBE}}_{\mu }(w)=\mathrm{\parallel }{\mathrm{\Pi }}_{\mu }{\delta }_w{\mathrm{\parallel }}_{\mu }^2$.

A finite state/action model with a joint one-step reward-transition law and stationary Markov policies.

The finite discounted policy Bellman operator.

Finite indexed sums.

The finite weighted quotient and feature span admit the stated unique projection.

Sutton and Barto, Reinforcement Learning: An Introduction, 2nd ed., §§9.2 and 11.4.

Tsitsiklis and Van Roy, “An Analysis of Temporal-Difference Learning with Function Approximation,” §III.A.