TheoremStatement: Literature-sourcedProof: AI-generatedPipeline-generatedjudge pass (gpt-6-luna)

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

Doubly robust evaluation is unbiased under either exact component

Statement

Assume AC and the finite-horizon dataset conditions with fixed target policy $π$ , behavior policy $μ$ , horizon $H$ , discount $0 \leq γ \leq 1$ , and initial state $s_{0}$ . Hold the finite functions ${\hat{Q}}_{t}$ and multipliers ${\hat{ρ}}_{t}$ fixed when taking expectation over the evaluation trajectories. Set ${\hat{V}}_{H} (s) = 0$ and ${\hat{V}}_{t} (s) = \sum_{a \in A (s)} π (a ∣ s) {\hat{Q}}_{t} (s, a)$ . Let

$D (τ) = {\hat{V}}_{0} (s_{0}) + \sum_{t = 0}^{H - 1} γ^{t} {\hat{W}}_{0 : t} (τ) (R_{t + 1} + γ {\hat{V}}_{t + 1} (S_{t + 1}) - {\hat{Q}}_{t} (S_{t}, A_{t})), {\hat{W}}_{0 : t} = \prod_{k = 0}^{t} {\hat{ρ}}_{k} (S_{k}, A_{k}) .$

For a dataset of $n$ independent evaluation trajectories, let ${\hat{J}}_{D R} = n^{- 1} \sum_{i = 1}^{n} D (τ_{i})$ . Then $E [{\hat{J}}_{D R}] = J_{H} (π; s_{0})$ under either of these separate conditions:

The full target trajectory law is absolutely continuous with respect to the behavior trajectory law, and every multiplier is the correct action ratio ${\hat{ρ}}_{t} (s, a) = π (a ∣ s) / μ (a ∣ s)$ on every behavior-positive state-action pair reachable before time $H$ ; the ratio is $0$ when $π (a ∣ s) = 0 < μ (a ∣ s)$ . No correctness assumption is imposed on the functions ${\hat{Q}}_{t}$ .
For every $0 \leq t < H$ , state $s$ , and action $a$ , the value model is exact:

${\hat{Q}}_{t} (s, a) = Q_{t}^{π} (s, a) : = \sum_{s^{'} \in S} \sum_{r \in R} p (s^{'}, r ∣ s, a) (r + γ V_{t + 1}^{π} (s^{'})), V_{H}^{π} (s) = 0, V_{t}^{π} (s) = \sum_{a \in A (s)} π (a ∣ s) Q_{t}^{π} (s, a),$

so ${\hat{V}}_{t} = V_{t}^{π}$ at every step, including the zero terminal value. This branch requires no trajectory coverage and no correctness assumption on the fixed finite multipliers. If neither exact condition holds, this theorem asserts no unbiasedness.

Statement references

The finite-horizon doubly robust score and its zero terminal-value convention.

The finite-horizon path law, target value, and behavior-data sampling convention.

Exact prefix density change under correct action ratios.

The shared transition/reward law conditional on each finite history.

AC is used for path-law, RN-density, and conditional-expectation existence.

Jiang and Li, Doubly Robust Off-policy Value Evaluation for Reinforcement Learning

Levine, Kumar, Tucker, and Fu, Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Facts & Assumptions

Given: AC, a finite MDP, finite horizon $H$ , $0 \leq γ \leq 1$ , fixed initial state $s_{0}$ , stationary target and behavior policies, independent behavior trajectories, fixed finite Q functions and multipliers, and the OPE estimator in the Statement.

[A1]

The Axiom of Choice: AC is required by the finite path-law, RN-density, and conditional-expectation existence suppliers cited below.

[F1]

Finite-horizon offline reinforcement-learning dataset and coverage: the same finite joint reward/next-state mass is used under both policies, the dataset trajectories have behavior law $P_{H}^{μ}$ , and $J_{H} (π; s_{0}) = E_{P_{H}^{π}} G_{H}$ .

[F2]

Finite-horizon doubly robust off-policy value evaluation: ${\hat{V}}_{H} = 0$ , ${\hat{V}}_{t} (s) = \sum_{a} π (a ∣ s) {\hat{Q}}_{t} (s, a)$ , and the estimator is the average of the stated finite weighted residual scores.

[F3]

Finite-horizon policy ratios are trajectory Radon-Nikodym derivatives: under full coverage, exact products of action ratios are the RN densities on every prefix through the next reward and state.

[F4]

A finite MDP policy generates action-state-reward trajectories: conditional on a finite history ending at state $s$ and action $a$ , the next reward/state atom has probability $p (s^{'}, r ∣ s, a)$ ; this follows by dividing the successive prefix atom masses when the conditioning history has positive behavior probability.

[F5]

Conditional expectation as an ae class and Conditional expectation exists by radon nikodym: under AC, conditional expectations exist as almost-sure classes with versions satisfying the event-integral definition Conditional expectation given a sigma algebra.

[F6]

Taking out what is known: if $X$ is integrable and $Z$ is bounded and measurable with respect to the conditioning sigma-algebra, then $E [Z X ∣ G] = Z E [X ∣ G]$ almost surely.

[F7]

Basic algebra and order properties of conditional expectation: conditional expectation is linear and preserves total expectation.

[F8]

Integrable real and complex functions, and their integrals and The Lebesgue integral is linear on $L^{1} (μ)$ : finite-MDP, finite-horizon terms are integrable, and finite real sums may be integrated termwise.

Proof

Given: The hypotheses and conventions in the Statement and [A1], [F1]–[F8].

1.1givenF1F2F3construct

First assume condition 1. For every $t$ , the residual $C_{t} = R_{t + 1} + γ {\hat{V}}_{t + 1} (S_{t + 1}) - {\hat{Q}}_{t} (S_{t}, A_{t})$ is a function of the prefix through $(R_{t + 1}, S_{t + 1})$ , so [F3] gives $E_{μ} [{\hat{W}}_{0 : t} C_{t}] = E_{π} [C_{t}]$ . The baseline term is the same under both laws because $S_{0} = s_{0}$ is fixed.

1.2givenF1F2algebra

Now assume condition 2. For each $s$ , let $U_{t}^{π} (s)$ be the expected discounted return over steps $t, \dots, H - 1$ under target policy $π$ , starting at state $s$ at time $t$ , and set $U_{H}^{π} (s) = 0$ . The zero-step return gives $U_{H}^{π} = V_{H}^{π}$ . If $U_{t + 1}^{π} = V_{t + 1}^{π}$ , then conditioning on the finite target action sum and the joint next-state/reward mass gives $U_{t}^{π} (s) = \sum_{a} π (a ∣ s) \sum_{s^{'}, r} p (s^{'}, r ∣ s, a) (r + γ V_{t + 1}^{π} (s^{'})) = \sum_{a} π (a ∣ s) Q_{t}^{π} (s, a) = V_{t}^{π} (s)$ . Backward induction proves $U_{t}^{π} = V_{t}^{π}$ for all $t$ , hence $V_{0}^{π} (s_{0}) = J_{H} (π; s_{0})$ . Condition 2 and the algorithm's definition of ${\hat{V}}_{t}$ then give ${\hat{V}}_{t} = V_{t}^{π}$ for all $t$ .

1.3givenA1F4F5F6F7F8constructalgebra

Fix $t$ and let $H_{t}$ be the finite sigma-algebra generated by the history through $(S_{t}, A_{t})$ . For each positive-probability history atom $h$ ending at $(s, a)$ , the path-law formula in [F4] gives $P_{μ} (h, (R_{t + 1}, S_{t + 1}) = (r, s^{'})) = P_{μ} (h) p (s^{'}, r ∣ s, a)$ ; dividing by $P_{μ} (h) > 0$ proves that the conditional next reward/state mass is $p (s^{'}, r ∣ s, a)$ . Therefore $E_{μ} [C_{t} ∣ H_{t}] = \sum_{s^{'}, r} p (s^{'}, r ∣ s, a) (r + γ V_{t + 1}^{π} (s^{'}) - Q_{t}^{π} (s, a)) = 0$ . The equality follows because $Q_{t}^{π} (s, a)$ is defined by the same finite sum. Each ${\hat{W}}_{0 : t}$ is a finite $H_{t}$ -measurable function, since the trajectory space and horizon are finite and the multipliers are fixed. Applying [F6] and then expectation preservation [F7] yields $E_{μ} [{\hat{W}}_{0 : t} C_{t}] = 0$ .

2.1givenF1F2F3F8step 1.1algebra

For each state $s$ , $\sum_{a} π (a ∣ s) {\hat{Q}}_{t} (s, a) = {\hat{V}}_{t} (s)$ , so averaging over the target-law distribution of $S_{t}$ gives $E_{π} [{\hat{Q}}_{t} (S_{t}, A_{t})] = E_{π} [{\hat{V}}_{t} (S_{t})]$ . Substituting the density identities from step 1.1 into the expected score, the ${\hat{V}}_{t}$ terms telescope: $E_{μ} D = {\hat{V}}_{0} (s_{0}) + \sum_{t = 0}^{H - 1} γ^{t} E_{π} [R_{t + 1}] + \sum_{t = 0}^{H - 1} γ^{t + 1} E_{π} [{\hat{V}}_{t + 1} (S_{t + 1})] - \sum_{t = 0}^{H - 1} γ^{t} E_{π} [{\hat{V}}_{t} (S_{t})] = J_{H} (π; s_{0})$ because $S_{0} = s_{0}$ and ${\hat{V}}_{H} = 0$ . This proves condition 1 without requiring a correct Q model.

2.2givenF2step 1.2step 1.3algebra

Every residual term in $D$ has zero expectation by step 1.3, while the baseline is ${\hat{V}}_{0} (s_{0}) = V_{0}^{π} (s_{0}) = J_{H} (π; s_{0})$ by step 1.2. Hence $E_{μ} D = J_{H} (π; s_{0})$ . No coverage or ratio-correctness premise was used in this branch.

3.1step 2.1step 2.2F1F8algebra∎

Condition 1 implies that one trajectory score has expectation $J_{H} (π; s_{0})$ by step 2.1, and condition 2 implies the same by step 2.2. Every evaluation trajectory has this same behavior law by [F1], so linearity of the integral [F8] makes their finite sample average have expectation $J_{H} (π; s_{0})$ . Thus either exact condition is sufficient.

Depends on

Used by

Correct ratios and a correct value model give separate DR identities Example

Dependency tree · two levels

48 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Nan Jiang and Lihong Li, Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (standard reference, not scraped)
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu, Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (standard reference, not scraped)