AlgorithmAlgorithm: Literature-sourcedProof: Not applicablePipeline-generatedjudge pass (gpt-6-luna)

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

Finite-horizon doubly robust off-policy value evaluation

Definition

Assume AC for the behavior-policy trajectory law. Input a finite-horizon logged dataset with $n \geq 1$ behavior trajectories, a target policy $π$ , finite real functions ${\hat{Q}}_{t} (s, a)$ for $0 \leq t < H$ , and finite nonnegative action-weight multipliers ${\hat{ρ}}_{t} (s, a)$ for $0 \leq t < H$ . Hold the functions and multipliers fixed when taking expectation over the evaluation dataset; they may be conditioned on an independent training sample. Define ${\hat{V}}_{H} (s) = 0$ for every $s$ , and for $t = H - 1, \dots, 0$ define

${\hat{V}}_{t} (s) = \sum_{a \in A (s)} π (a ∣ s) {\hat{Q}}_{t} (s, a) .$

For each logged trajectory $i$ , set ${\hat{W}}_{i, - 1} = 1$ and ${\hat{W}}_{i, t} = {\hat{W}}_{i, t - 1} {\hat{ρ}}_{t} (S_{t}^{i}, A_{t}^{i})$ for $0 \leq t < H$ . Compute

$D_{i} = {\hat{V}}_{0} (s_{0}) + \sum_{t = 0}^{H - 1} γ^{t} {\hat{W}}_{i, t} (R_{t + 1}^{i} + γ {\hat{V}}_{t + 1} (S_{t + 1}^{i}) - {\hat{Q}}_{t} (S_{t}^{i}, A_{t}^{i})),$

and return ${\hat{J}}_{D R} = n^{- 1} \sum_{i = 1}^{n} D_{i}$ . When supplied as propensity ratios, the multipliers have the form ${\hat{ρ}}_{t} (s, a) = π (a ∣ s) / {\tilde{μ}}_{t} (a ∣ s)$ , with positive denominators on target-positive actions. In the exact-ratio case they equal the target-to-behavior action-likelihood factors $π (a ∣ s) / μ (a ∣ s)$ ; the procedure also permits finite fitted multipliers for the correct-value-model case. The fixed terminal value ${\hat{V}}_{H} = 0$ is part of the procedure.

Definition references

The finite-horizon evaluation dataset, target value, and behavior-data law.

Exact behavior propensities yield the finite-prefix target-to-behavior density.

Jiang and Li, Doubly Robust Off-policy Value Evaluation for Reinforcement Learning

Levine, Kumar, Tucker, and Fu, Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Depends on

Used by

Dependency tree · two levels

15 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Nan Jiang and Lihong Li, Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (standard reference, not scraped)
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu, Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (standard reference, not scraped)