Normalized discounted state occupancy

Let a finite discounted CMDP and a stationary policy $\pi$ be given. The column transition matrix $K_{\pi }$ has entry

$K_{\pi }(s^{\mathrm{\prime }},s):={\sum }_{a\in A(s)}{\sum }_{\rho \in \mathcal{R}}\pi (a\mid s)p(s^{\mathrm{\prime }},\rho \mid s,a),$

so that its column indexed by $s$ gives the next-state distribution from current state $s$. With the state-mass column vectors ${\mu }_t^{\pi }$ from the CMDP definition, the normalized discounted state occupancy is

$d^{\pi }(s):=(1-\gamma ){\sum }_{t=0}^{\mathrm{\infty }}{\gamma }^t{\mu }_t^{\pi }(s)\mathrm{.}$

This normalization is part of the definition; the occupancy is a probability distribution by the separate well-definedness result below.

The finite CMDP specifies the joint next-state/reward masses and the recursively defined state-mass vectors.

Normalized discounted occupancy is a probability distribution ↗: The recurrence yields probability vectors, and multiplying the geometric occupancy sum by $1-\gamma$ gives total mass one.

Achiam et al., Constrained Policy Optimization, supplementary material, Appendix 10.1.1.

Altman, Constrained Markov Decision Processes, Chapter 3 §3.1.