Practical CPO updates do not inherit exact safety guarantees automatically

An expected discounted-cost constraint controls an expectation. It bounds the probability of a specified pathwise violation only when the cost signal is linked to that event; by itself it does not guarantee pathwise safety or safe exploration. The performance theorem concerns exact policy returns. Its total-variation branch is choice-free, while its average-KL branch assumes the Axiom of Choice. The practical CPO update instead uses sample estimates, a linearized reward/cost objective, a second-order KL approximation, conjugate-gradient solves, and a backtracking line search against sample-estimated constraints. Optional nonnegative cost shaping changes the constrained signal but supplies no error bound for the sample estimates or numerical solves. Consequently, the ideal theorem does not by itself certify the sampled iterates' true costs or imply convergence of the implementation; this remark asserts no such guarantee.

The exact expected-return bounds and their choice-free TV/AC-conditional KL branches.

The sample-based update, quadratic approximation, CG solves, line search, and optional cost shaping.

Achiam et al., Constrained Policy Optimization, §§6.1–6.3 and §9.