ExampleConstruction: AI-generatedVerification: AI-generatedPipeline-generatedprecheck pass

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

A finite bad-step count need not be a permanent stopping time

Example

Consider a one-state MDP with $γ = 0$ and two actions $a, b$ whose deterministic rewards are $1, 0$ , respectively. For an integer $n \geq 1$ , let a deterministic history-dependent controller choose $b$ at times $1, \dots, n$ and $n + 2$ , and choose $a$ at every other time. For $ϵ = 1 / 2$ , it has exactly $n + 1$ bad steps, including one after a good step.

Example references

The one-state finite MDP and deterministic rewards.

Bad steps are counted over the whole run, with no prefix condition.

Optimal value and continuation values.

Strehl, Li, and Littman, Reinforcement Learning in Finite MDPs: PAC Analysis.

Facts & Assumptions

Given: The MDP, controller, time indices, and $ϵ$ in the Example.

[F1]

Well-defined discounted values and optimal Bellman fixed points: with $γ = 0$ , the continuation value is the expected immediate reward and the optimal state value is $1$ .

[F2]

PAC-MDP sample complexity of exploration: a time is $ϵ$ -bad exactly when its continuation value is strictly below $V^{*} (s) - ϵ$ .

Proof

1.1givenF1algebra

Since $γ = 0$ , every reward after the next reward has coefficient zero. Thus the controller continuation value at time $t$ is exactly the deterministic reward of its action at $t$ , while the optimal value is $V^{*} = 1$ .

2.1givenF1F2step 1.1algebra∎

At exactly the indices $1, \dots, n, n + 2$ , the chosen action is $b$ , so its continuation value is $0 < 1 - 1 / 2$ and each of those times is bad by [F2]. At every other time it chooses $a$ and has value $1$ , so that time is not bad. There are $n + 1$ bad indices, and $n + 2$ is bad after the good index $n + 1$ .

Depends on

Used by

Nothing in the library uses this result yet.

Dependency tree · two levels

25 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Alexander L. Strehl, Lihong Li, and Michael L. Littman, Reinforcement Learning in Finite MDPs: PAC Analysis (standard reference, not scraped)