ExampleConstruction: AI-generatedVerification: AI-generatedPipeline-generatedjudge pass (gpt-6-luna)

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

Full action support permits reweighting while missing support does not

Example

Assume AC and take horizon $H = 1$ , discount $γ = \frac{1}{2}$ , states $s_{0}, z$ , actions $a, b$ at $s_{0}$ , and action $c$ at $z$ . Every action moves to $z$ ; action $a$ gives reward $0$ , action $b$ gives reward $1$ , and action $c$ gives reward $0$ . The target policy chooses $b$ at $s_{0}$ , and both policies choose $c$ at $z$ . With behavior probabilities $μ (a ∣ s_{0}) = μ (b ∣ s_{0}) = \frac{1}{2}$ , full trajectory coverage holds: the single-trajectory ordinary importance-sampling score is $2$ when action $b$ is observed and $0$ when action $a$ is observed, and its expectation is the target value $1$ . If instead $μ (a ∣ s_{0}) = 1$ , the target action has no behavior support. Two processes that agree on the logged action $a$ and its reward $0$ , but give action $b$ rewards $1$ and $- 1$ , respectively, have identical logged-data laws and different target values; the behavior data cannot identify which value is correct.

Example references

Finite-horizon trajectory laws and the target-value convention.

Ordinary trajectory importance sampling and its weight formula.

A proof that logged data can fail to identify value without coverage.

Sutton and Barto, Reinforcement Learning: An Introduction, 2nd ed.

Jiang and Li, Doubly Robust Off-policy Value Evaluation for Reinforcement Learning

Facts & Assumptions

Given: AC, the horizon-one MDP and policies in the Example, one behavior trajectory at a time, and the corresponding target value.

[F1]

Finite-horizon offline reinforcement-learning dataset and coverage: $P_{H}^{σ}$ is the finite-horizon path law and the target value is the expectation of the discounted return.

[F2]

Ordinary, per-decision, and weighted trajectory importance sampling: for $H = 1$ , the ordinary score is the observed return multiplied by $π (A_{0} ∣ s_{0}) / μ (A_{0} ∣ s_{0})$ .

[F3]

Without trajectory coverage, logged data need not identify policy value: two behavior-indistinguishable finite MDPs can have different target values when the target action is absent from behavior support.

Verification

1.1givenF1F2algebra

Under $μ (b ∣ s_{0}) = \frac{1}{2}$ , the ratio for an observed $b$ is $1 / (\frac{1}{2}) = 2$ ; its return is $1$ . The ratio for an observed $a$ is $0 / (\frac{1}{2}) = 0$ ; its return is $0$ . Thus the OIS score is $2$ with probability $\frac{1}{2}$ and $0$ with probability $\frac{1}{2}$ , so its expectation is $1$ , equal to the target value. Since $H = 1$ , PDIS has the same score.

2.1givenF1F3algebra∎

Under $μ (a ∣ s_{0}) = 1$ , the target trajectory taking $b$ has target probability $1$ and behavior probability $0$ , so the full-coverage premise fails and its likelihood ratio is undefined. The two reward choices $r (b) = 1$ and $r (b) = - 1$ yield the same deterministic logged trajectory but target values $1$ and $- 1$ .

Depends on

Used by

Nothing in the library uses this result yet.

Dependency tree · two levels

12 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd ed. (standard reference, not scraped)
Nan Jiang and Lihong Li, Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (standard reference, not scraped)