ExampleConstruction: Literature-sourcedVerification: AI-generatedPipeline-generatedjudge pass (gpt-6-luna)

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

At discount one-half, identical observable laws hide different Bellman-error minima

Example

Assume Choice for the infinite trajectory laws and set $γ = 1 / 2$ . In MRP 1, the states are $A, B$ ; $A \to B$ has reward $0$ deterministically, while $B \to A$ has reward $1$ and $B \to B$ has reward $- 1$ , each with probability $1 / 2$ . Its stationary weighting is $(1 / 3, 2 / 3)$ . In MRP 2, the states are $A, B, B^{'}$ ; $A \to B$ and $A \to B^{'}$ each have reward $0$ and probability $1 / 2$ , $B \to A$ has reward $1$ deterministically, and $B^{'} \to B$ and $B^{'} \to B^{'}$ each have reward $- 1$ and probability $1 / 2$ . Its stationary weighting is $(1 / 3, 1 / 3, 1 / 3)$ . Both processes use feature $ψ (A) = (1, 0)$ and $ψ (B) = ψ (B^{'}) = (0, 1)$ , and they induce the same stationary feature/reward sequence law. Their state-weighted mean square Bellman errors are

\begin{aligned} {BE}_{1} (a, b) & = \frac{1}{3} {(\frac{b}{2} - a)}^{2} + \frac{2}{3} {(\frac{a + b}{4} - b)}^{2}, \\ {BE}_{2} (a, b) & = \frac{1}{3} [{(\frac{b}{2} - a)}^{2} + {(1 + \frac{a}{2} - b)}^{2} + {(- 1 - \frac{b}{2})}^{2}] . \end{aligned}

The unique minimizers are $(0, 0)$ for MRP 1 and $(- 2 / 7, 1 / 7)$ for MRP 2. Their minimum Bellman errors are $0$ and $25 / 42$ , respectively, while ${BE}_{2} (0, 0) = 2 / 3$ .

Example references

The axiom used to form the infinite trajectory laws.

The state-weighted mean square Bellman-error objective.

The general matched-observation MRP theorem and exact minimizing parameters.

Sutton and Barto, Reinforcement Learning: An Introduction, 2nd ed., §11.6.

Hutter, Yang-Zhao, and Majeed, “Conditions on Features for Temporal Difference-Like Methods to Converge,” §4.

Facts & Assumptions

Given: The two finite transition/reward models, stationary weightings, common feature map, and $γ = 1 / 2$ in the Example.

[A1]

The Axiom of Choice: every family of nonempty sets has a choice function; the theorem used in [F2] assumes this axiom to construct the infinite path laws.

[F1]

State-weighted Bellman error and projected Bellman error: the mean square Bellman error is the weighted sum of the squares of the conditional expected one-step TD errors.

[F2]

Feature-reward data do not determine the Bellman error or, at positive discount, its minimizer: for general $0 \leq γ < 1$ , these models have identical observable laws, MRP 1 has unique minimizer $(0, 0)$ , and MRP 2 has unique minimizer $(- γ / (1 + γ + γ^{2}), γ / (2 (1 + γ + γ^{2})))$ with minimum $(γ + 2)^{2} / (6 (1 + γ + γ^{2}))$ .

Verification

1.1givenF2algebra

At $γ = 1 / 2$ , $D = 1 + γ + γ^{2} = 7 / 4$ . Substitution in [F2] gives MRP 2’s minimizing parameter $(- 2 / 7, 1 / 7)$ and minimum $(5 / 2)^{2} / (6 \cdot 7 / 4) = 25 / 42$ . The same substitution gives MRP 1’s minimizing parameter $(0, 0)$ and minimum $0$ .

2.1givenA1F1F2algebra∎

At $w = (0, 0)$ the MRP 2 residuals are $0, 1, - 1$ , so [F1] gives ${BE}_{2} (0, 0) = (0^{2} + 1^{2} + (- 1)^{2}) / 3 = 2 / 3$ . The MRP pair and common observation law are the $γ = 1 / 2$ instance of [F2], whose path-law claim uses [A1], completing the example.

Depends on

Used by

Nothing in the library uses this result yet.

Dependency tree · two levels

10 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd ed. (standard reference, not scraped)
Marcus Hutter, Samuel Yang-Zhao, and Sultan J. Majeed, Conditions on Features for Temporal Difference-Like Methods to Converge (standard reference, not scraped)