TheoremStatement: AI-adaptedProof: AI-generatedPipeline-generatedjudge pass (gpt-6-luna)

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

Value iteration converges at the Bellman contraction rate

Statement

Let $V^{*}$ be the unique fixed point of $T^{*}$ for a finite MDP with $0 \leq γ < 1$ . For every initial $V_{0} \in R^{S}$ and every $m \in N$ , the exact value-iteration iterates $V_{m + 1} = T^{*} V_{m}$ satisfy

$∥ V_{m} - V^{*} ∥_{\infty} \leq γ^{m} ∥ V_{0} - V^{*} ∥_{\infty},$

with $γ^{0} = 1$ .

Statement references

Sutton and Barto, Reinforcement Learning: An Introduction.

Facts & Assumptions

Given: A finite MDP, $0 \leq γ < 1$ , an initial function $V_{0} \in R^{S}$ , and $m \in N$ .

[F1]

Exact value iteration for the Bellman optimality operator: the exact iterates satisfy $V_{j + 1} = T^{*} V_{j}$ .

[F2]

Finite-MDP Bellman operators are gamma contractions with unique fixed points: $T^{*}$ has a unique fixed point $V^{*}$ and $∥ T^{*} U - T^{*} W ∥_{\infty} \leq γ ∥ U - W ∥_{\infty}$ .

[L1]

Integer powers $a^{m}$ : $γ^{0} = 1$ and $γ^{j + 1} = γ γ^{j}$ for every $j \in N$ .

[L2]

The principle of mathematical induction: a claim true at $0$ and preserved from $j$ to $j + 1$ holds for every natural $j$ .

Proof

1.1givenF1L1algebrabase

At $m = 0$ , $V_{m} = V_{0}$ and $γ^{0} = 1$ by [L1], so the asserted inequality is equality.

1.2ih

Induction hypothesis: assume $∥ V_{m} - V^{*} ∥_{\infty} \leq γ^{m} ∥ V_{0} - V^{*} ∥_{\infty}$ for an arbitrary $m \in N$ .

2.1step 1.2F1F2L1algebra

By [F1], [F2], and $T^{*} V^{*} = V^{*}$ , $∥ V_{m + 1} - V^{*} ∥_{\infty} = ∥ T^{*} V_{m} - T^{*} V^{*} ∥_{\infty} \leq γ ∥ V_{m} - V^{*} ∥_{\infty} \leq γ^{m + 1} ∥ V_{0} - V^{*} ∥_{\infty},$ where the last equality of factors uses [L1].

3.1step 1.1step 2.1L2inductiondischarge-induction: step 2.1∎

Step 1.1 is the base case and step 2.1 is the induction step, so [L2] gives the inequality for every $m \in N$ .

Depends on

Used by

A one-state value-iteration model attains the Bellman error bound Example

Dependency tree · two levels

33 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd ed. (standard reference, not scraped)