Alphabeta MathHow statement and proof provenance work
The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.
- Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
- AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
- AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.
These labels describe origin, not correctness: citations and verification chips remain separate evidence.
Off-policy estimates do not guarantee offline policy safety
Limits of offline evaluation and policy optimization
Under the finite-horizon data assumptions above, full trajectory-law coverage, true known behavior propensities, and integrable weighted returns, the ordinary trajectory and per-decision importance-sampling estimators are unbiased for the expected return of the fixed target policy. In this finite trajectory setting their finite numerical outputs have finite variance, but they do not provide a variance bound uniform over behavior policies, a confidence bound, or a guarantee that the target policy outperforms the behavior policy. For each fixed model, horizon, and sample size, the trajectory dataset space is finite; after any fixed finite assignment on the behavior-null unsupported event, each estimator has finitely many finite values, so its second moment and variance are finite. Product weights can vary exponentially with horizon; self-normalization and clipping can change finite-sample expectations. With missing coverage, two environments can agree on every logged trajectory and assign different values to an unobserved action. A conservative policy constraint explicitly restricts candidate policies, for example by limiting their divergence from the behavior policy; a pessimistic value adjustment lowers predicted values for poorly supported actions. These choices can reduce unsupported action selection but can also limit policy improvement, and they are not by themselves safety guarantees. Model exploitation occurs when optimization selects state-action pairs where an approximate dynamics or value model overpredicts return; optimizing such errors can worsen real performance. A finite-sample safety or performance guarantee needs its own assumptions and error bounds.
Limits of offline evaluation and policy optimization references
The fixed-data, target-policy, behavior-policy, and full-coverage conventions.
OIS, PDIS, and the self-normalized WIS procedure.
The finite-horizon doubly robust residual estimator.
A finite example showing that missing behavior support can prevent value identification.
Under finite-horizon coverage and the stated behavior-data assumptions, OIS and PDIS are unbiased.
Sutton and Barto, Reinforcement Learning: An Introduction, 2nd ed.
Jiang and Li, Doubly Robust Off-policy Value Evaluation for Reinforcement Learning
Depends on
- Finite-horizon doubly robust off-policy value evaluation
- Ordinary, per-decision, and weighted trajectory importance sampling
- The Axiom of Choice
- Finite-horizon offline reinforcement-learning dataset and coverage
- Without trajectory coverage, logged data need not identify policy value
- Trajectory and per-decision importance sampling are unbiased under coverage
Used by
Nothing in the library uses this result yet.
Dependency tree · two levels
15 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.
Sources
- Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd ed. (standard reference, not scraped)
- Nan Jiang and Lihong Li, Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (standard reference, not scraped)
- Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu, Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (standard reference, not scraped)