ExampleConstruction: AI-generatedVerification: AI-generatedPipeline-generatedjudge pass (gpt-6-luna)

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

Correct ratios and a correct value model give separate DR identities

Example

Assume AC and take a one-step MDP with states $s_{0}, z$ , actions $a, b$ at $s_{0}$ , action $c$ at $z$ , horizon $H = 1$ , and discount $γ = \frac{1}{2}$ . Every action moves to $z$ ; action $a$ gives reward $0$ , action $b$ gives reward $1$ , and action $c$ gives reward $0$ . The target policy chooses $b$ at $s_{0}$ and $c$ at $z$ , so its value from $s_{0}$ is $1$ .

In the first evaluation case, behavior chooses $a$ and $b$ each with probability $\frac{1}{2}$ at $s_{0}$ , the action ratios are exact, and the fitted Q values are ${\hat{Q}}_{0} (s_{0}, a) = {\hat{Q}}_{0} (s_{0}, b) = {\hat{Q}}_{0} (z, c) = 0$ . The DR score is $2$ on a logged $b$ trajectory and $0$ on a logged $a$ trajectory, so its expectation is $1$ although the Q model is wrong at $b$ .

In the second evaluation case, behavior chooses $a$ at $s_{0}$ with probability $1$ , so the target action has no behavior support. The fitted model is exact: ${\hat{Q}}_{0} (s_{0}, a) = 0$ , ${\hat{Q}}_{0} (s_{0}, b) = 1$ , and ${\hat{Q}}_{0} (z, c) = 0$ , with ${\hat{V}}_{1} = 0$ and ${\hat{V}}_{0} (s_{0}) = 1$ . Use the finite multipliers ${\hat{ρ}}_{0} (s_{0}, a) = {\hat{ρ}}_{0} (s_{0}, b) = {\hat{ρ}}_{0} (z, c) = 1$ . Every behavior trajectory has zero residual, so its DR score is $1$ , equal to the target value despite missing support and incorrect action ratios.

In the diagram, vertices are states; an edge labeled $a : r$ means action $a$ produces reward $r$ and moves deterministically to the arrow endpoint. The proof uses the transition/reward values stated in prose, not the drawing.

diagram failed to render:
\begin{tikzpicture}[>=stealth,thick]
\node[draw,circle] (s) at (0,0) {$s_0$};
\node[draw,circle] (z) at (3,0) {$z$};
\draw[->,bend left=18] (s) to node[above] {$a:0$} (z);
\draw[->,bend right=18] (s) to node[below] {$b:1$} (z);
\draw[->,loop right] (z) to node[right] {$c:0$} (z);
\end{tikzpicture}

Example references

The finite-horizon return and behavior-data law.

The DR score and finite-horizon terminal convention.

The separate exact-ratio and exact-model unbiasedness branches.

Jiang and Li, Doubly Robust Off-policy Value Evaluation for Reinforcement Learning

Levine, Kumar, Tucker, and Fu, Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Facts & Assumptions

Given: AC, the one-step MDP, both behavior policies, the target policy, the stated Q functions and multipliers, and $H = 1$ , $γ = \frac{1}{2}$ .

[F1]

Finite-horizon offline reinforcement-learning dataset and coverage: the target value is the expected finite-horizon return, and behavior trajectories use the behavior policy with the same transition/reward law.

[F2]

Finite-horizon doubly robust off-policy value evaluation: with $H = 1$ , $V_{1} = 0$ , the DR score is ${\hat{V}}_{0} (s_{0}) + {\hat{ρ}}_{0} (S_{0}, A_{0}) (R_{1} - {\hat{Q}}_{0} (S_{0}, A_{0}))$ .

[F3]

Doubly robust evaluation is unbiased under either exact component: exact ratios with trajectory coverage or a correct finite-horizon value model with the zero terminal convention suffices for unbiasedness.

Verification

1.1givenF1F2algebra

In the first case, ${\hat{V}}_{0} (s_{0}) = \frac{1}{2} {\hat{Q}}_{0} (s_{0}, a) + \frac{1}{2} {\hat{Q}}_{0} (s_{0}, b) = 0$ , ${\hat{ρ}}_{0} (s_{0}, a) = 0$ , and ${\hat{ρ}}_{0} (s_{0}, b) = 2$ . A logged $a$ has score $0 + 0 (0 - 0) = 0$ ; a logged $b$ has score $0 + 2 (1 - 0) = 2$ . Since each occurs with probability $\frac{1}{2}$ , the mean DR score is $1$ , the target value, although ${\hat{Q}}_{0} (s_{0}, b) = 0 \neq 1$ .

1.2givenF1F2F3algebra

In the second case, the exact finite-horizon model has $Q_{0}^{π} (s_{0}, a) = 0$ , $Q_{0}^{π} (s_{0}, b) = 1$ , and $Q_{0}^{π} (z, c) = 0$ ; its target-policy value is ${\hat{V}}_{0} (s_{0}) = 1$ and its terminal value is $0$ . The behavior policy always logs $a$ , so each observed residual is $0 + \frac{1}{2} \cdot 0 - 0 = 0$ , regardless of the supplied multiplier $1$ . Every DR score is therefore $1$ , and its expectation is the target value despite the absent target action in behavior data.

2.1step 1.1step 1.2F2F3algebra∎

The first calculation verifies the exact-ratio branch with a misspecified Q function; the second verifies the exact-model branch with missing support and incorrect multipliers. Both use the terminal convention ${\hat{V}}_{1} = 0$ from [F2].

Depends on

Used by

Nothing in the library uses this result yet.

Dependency tree · two levels

18 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Nan Jiang and Lihong Li, Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (standard reference, not scraped)
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu, Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (standard reference, not scraped)