Discounted value, action-value, and advantage functions

Let a finite discounted CMDP, a stationary policy $\pi$, and a bounded transition signal $g$ be given. For $s\in S$, let ${\delta }_s$ be the initial distribution assigning probability one to $s$. The state value is $V_g^{\pi }(s):=J_g^{\pi }({\delta }_s)$. For $a\in A(s)$, define the action value and advantage by

$Q_g^{\pi }(s,a):={\sum }_{\rho \in \mathcal{R}}{\sum }_{s^{\mathrm{\prime }}\in S}p(s^{\mathrm{\prime }},\rho \mid s,a)(g(s,a,\rho ,s^{\mathrm{\prime }})+\gamma V_g^{\pi }(s^{\mathrm{\prime }})),A_g^{\pi }(s,a):=Q_g^{\pi }(s,a)-V_g^{\pi }(s)\mathrm{.}$

Taking $g=r$ gives reward values and advantages; taking $g=c_i$ gives the value and advantage for the $i$th cost. These quantities are finite because the signal is bounded, $0\le \gamma <1$, and the transition law is normalized.

The finite CMDP and bounded transition signals.

Expected discounted-cost feasibility constraint: The well-defined expected discounted sum $J_g^{\pi }(\mu )$.

Bounded discounted signal sums are absolutely convergent and finite.

Achiam et al., Constrained Policy Optimization, §§3 and 5.3.