Alphabeta MathConstrained MDPs and policy safety
About this page
Pipeline-generated
How statement and proof provenance work
The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.
- Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
- AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
- AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.
These labels describe origin, not correctness: citations and verification chips remain separate evidence.
1 · Prerequisites
Abstract Algebra
- Binary Operations, Monoids, Groups and Subgroups
- Cosets, Index and Lagrange's Theorem
- Divisibility, Euclidean Domains, Principal Ideal Domains and Unique Factorisation
- Ideals, Quotient Rings and the Isomorphism Theorems for Rings
- Normal Subgroups and Quotient Groups
- Polynomial Rings, the Division Algorithm and Roots
- Rings, Subrings, Integral Domains and Fields
- Simple Field Extensions and the Construction of the Complex Numbers
Linear Algebra
Machine Learning Algorithms
Mathematical Statistics
Measure Theory
Probability
Real Analysis
- Absolute and Conditional Convergence; Rearrangement; Products
- Construction of the Real Numbers via Cauchy Sequences
- Construction of the Real Numbers via Dedekind Cuts
- Continuity, IVT, EVT, and Uniform Continuity
- Convexity
- Countability and Uncountability
- Darboux, L'Hôpital, and Taylor's Theorem
- Foundations of the Real Numbers for Analysis
- Limits of Real Functions
- limsup, liminf, and Subsequential Limits
- Mixed Partials, Taylor Formulae, and Extrema
- Monotone Functions, Discontinuities, and Continuity Sets
- Monotone Sequences, Bolzano-Weierstrass, and Cauchy Completeness
- Power Series and Real-Analytic Functions
- Properties of the Integral and the Working FTC
- Rⁿ as a Normed Space; Vector-Valued Functions
- Roots, Rational Powers, and Classical Inequalities
- Sequences and Limits
- Sequences and Series of Functions; Uniform Convergence
- Series: Convergence and the Nonnegative Tests
- Suprema and Infima
- The Derivative and the Mean Value Theorems
- The Exponential Function
- The Integral Logarithm and the Equivalence of Its Characterisations
- The Logarithm and General Powers
- The Riemann Integral: Definition and Integrability
- The Total Derivative in ℝᵐ → ℝⁿ
- Topology of ℝ
Set Theory
2 · Logical flowchart
3 · Items
Lemma
Finite-action average total variation is bounded by average KL
Definition
Finite discounted constrained Markov decision process
Definition
Normalized discounted state occupancy
Definition
Expected discounted-cost feasibility constraint
Lemma
Normalized discounted occupancy is a probability distribution
Definition
Discounted value, action-value, and advantage functions
Lemma
Discounted occupancy change is bounded by average policy TV
Lemma
Performance-difference identity for an arbitrary bounded baseline
Theorem
Policy performance bound by average divergence
Algorithm
Sample-based constrained policy optimization update
Lemma
The sample CPO update has a well-defined proposal and finite backtracking search
Remark
Practical CPO updates do not inherit exact safety guarantees automatically
Sources
- Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel, Constrained Policy Optimization, PMLR 70 (2017)
- Eitan Altman, Constrained Markov Decision Processes
- Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel, Constrained Policy Optimization, supplementary material
- Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel, Constrained Policy Optimization, PMLR 70 (2017), main paper
- Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel, Constrained Policy Optimization, PMLR 70 (2017), supplementary material
- Achiam et al., Constrained Policy Optimization, Sections 6.1–6.3