Alphabeta MathFinite MDPs and Bellman operators
About this page
Pipeline-generated
How statement and proof provenance work
The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.
- Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
- AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
- AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.
These labels describe origin, not correctness: citations and verification chips remain separate evidence.
✓ 2 results · all verified · 0 also independently AI-judged
Every result on this page is machine-checked by a proof checker and read in full and owner-audited; the judge is an additional, independent cross-model AI review of the proofs. The 2 not AI-judged were verified by owner audit (typically over a confirmed judge false positive), not failures.
1 · Prerequisites
Abstract Algebra
- Binary Operations, Monoids, Groups and Subgroups
- Cosets, Index and Lagrange's Theorem
- Divisibility, Euclidean Domains, Principal Ideal Domains and Unique Factorisation
- Ideals, Quotient Rings and the Isomorphism Theorems for Rings
- Normal Subgroups and Quotient Groups
- Polynomial Rings, the Division Algorithm and Roots
- Rings, Subrings, Integral Domains and Fields
- Simple Field Extensions and the Construction of the Complex Numbers
Combinatorics
Linear Algebra
Measure Theory
- Measurable Functions and Simple Approximation
- Measures and Their Basic Properties
- Outer Measure and the Caratheodory Extension Theorem
- Product Measures and the Fubini Tonelli Theorems
- Sigma Algebras and Borel Sets
- Signed and Complex Measures Hahn and Jordan
- The Lebesgue Integral and the Convergence Theorems
- The Lᵖ Spaces Holder Minkowski and Riesz Fischer
- The Radon Nikodym Theorem and Lebesgue Decomposition
Probability
Real Analysis
- Absolute and Conditional Convergence; Rearrangement; Products
- Construction of the Real Numbers via Cauchy Sequences
- Construction of the Real Numbers via Dedekind Cuts
- Continuity, IVT, EVT, and Uniform Continuity
- Countability and Uncountability
- Foundations of the Real Numbers for Analysis
- Limits of Real Functions
- limsup, liminf, and Subsequential Limits
- Monotone Functions, Discontinuities, and Continuity Sets
- Monotone Sequences, Bolzano-Weierstrass, and Cauchy Completeness
- Power Series and Real-Analytic Functions
- Properties of the Integral and the Working FTC
- Roots, Rational Powers, and Classical Inequalities
- Sequences and Limits
- Sequences and Series of Functions; Uniform Convergence
- Series: Convergence and the Nonnegative Tests
- Suprema and Infima
- The Derivative and the Mean Value Theorems
- The Exponential Function
- The Logarithm and General Powers
- The Riemann Integral: Definition and Integrability
- Topology of ℝ
Set Theory
2 · Logical flowchart
3 · Items
Definition
Finite Markov decision process with joint reward-transition law
Lemma
Absolute convergence and tail bound for bounded discounted rewards
Lemma
Completeness of the sup metric on finite-state value functions
Theorem
A finite MDP and stationary policy generate a reward-state Markov chain
Definition
Policy Bellman operator
Definition
Bellman optimality operator
Definition
Discounted returns and policy values in a finite MDP
Theorem
Finite-MDP Bellman operators are gamma contractions with unique fixed points
Algorithm
Exact value iteration for the Bellman optimality operator
Lemma
Expected discounted policy value is a fixed point of its Bellman operator
Theorem
Value iteration converges at the Bellman contraction rate
Algorithm
Exact finite policy iteration with retained greedy actions
Theorem
Greedy policy improvement and finite policy-iteration termination
Sources
- Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd ed.
- Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra, Planning and Acting in Partially Observable Stochastic Domains, Brown CS-96-08
- Sergey Mozgovoy, Lecture Notes on Metric Spaces and Functional Analysis