Expected discounted-cost feasibility constraint

Let a finite discounted CMDP have initial distribution $\mu$, discount $\gamma \in [0,1)$, $m$ bounded transition signals $c_i$, thresholds $b_i$, and finite-time state masses ${\mu }_t^{\pi }$ under a stationary policy $\pi$, as defined in the CMDP model. For any bounded transition signal $g$ on $(s,a,\rho ,s^{\mathrm{\prime }})$, define its expected discounted sum under $\pi$ by

$J_g^{\pi }(\mu ):={\sum }_{t=0}^{\mathrm{\infty }}{\gamma }^t{\sum }_{s\in S}{\sum }_{a\in A(s)}{\sum }_{\rho \in \mathcal{R}}{\sum }_{s^{\mathrm{\prime }}\in S}{\mu }_t^{\pi }(s)\pi (a\mid s)p(s^{\mathrm{\prime }},\rho \mid s,a)g(s,a,\rho ,s^{\mathrm{\prime }})\mathrm{.}$

The expected discounted reward is $J^{\pi }:=J_r^{\pi }(\mu )$ for the immediate reward signal $r(s,a,\rho ,s^{\mathrm{\prime }})=\rho$, and the $i$th expected discounted cost is $J_{c_i}^{\pi }(\mu )$. The policy $\pi$ is feasible for the discounted-cost constraints exactly when $J_{c_i}^{\pi }(\mu )\le b_i$ for every $i\in \{1,\dots ,m\}$. Feasibility constrains these expectations; it does not assert a per-trajectory bound.

A finite CMDP fixes the finite joint transition/reward model, initial distribution, discount, bounded cost signals, and thresholds.

Absolute convergence and tail bound for bounded discounted rewards: Every bounded real signal sequence has an absolutely convergent discounted sum for $0\le \gamma <1$, bounded by its signal bound divided by $1-\gamma$.

The finite-time masses are probability distributions, so each bounded finite-time transition-signal expectation is bounded and the displayed discounted series is well defined. ↗

Altman, Constrained Markov Decision Processes, Chapter 2 §2.2.

Achiam et al., Constrained Policy Optimization, §3.