Pipeline-generated

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

✓ 3 results · all verified · 3 also independently AI-judged

Every result on this page is machine-checked by a proof checker and read in full and owner-audited; the judge is an additional, independent cross-model AI review of the proofs; all 3 also cleared it.

Model-based planning, Dyna, and model exploitation — Examples

1 · Prerequisites

2 · Logical flowchart

3 · Definitions, theorems and proofs

None yet.

4 · Examples, counterexamples and false statements

ExampleConstruction: AI-generatedVerification: AI-generatedprecheck pendingjudge pass (gpt-6-luna)Open item page →

An inaccurate model can give the exact return on behavior support

Example

Let $S = {s, g}$ , with one dummy action $d$ at $g$ and actions $a, b$ at $s$ . At $g$ , action $d$ has reward $0$ and returns to $g$ . At $s$ , action $a$ has reward $1$ and moves to $g$ , while action $b$ returns to $s$ with true reward $0$ and learned reward $1$ . The true and learned models agree on $(s, a)$ and $(g, d)$ . Let the supported set be $B = {(s, a), (g, d)}$ , let the initial state be $s$ , and let $H \geq 2$ . A policy that chooses $a$ at $s$ and $d$ at $g$ has identical true and learned $H$ -step returns, although the learned model is wrong on the unsupported pair $(s, b)$ .

The TikZ diagram shows the two model transition/reward tables: a label $u : r$ gives action $u$ and its immediate reward on the arrow, and an arrow returning to its source denotes a self-transition. The learned reward for $b$ is marked with a hat.

\begin{center} \begin{tikzpicture}[>=stealth, node distance=32mm] \node[circle,draw] (s) { $s$ }; \node[circle,draw,right of=s] (g) { $g$ }; \draw[->] (s) to[bend left=12] node[above] { $a : 1$ } (g); \draw[->] (g) to[loop right] node[right] { $d : 0$ } (g); \draw[->] (s) to[loop left] node[left] { $b : 0 (\hat{b} : 1)$ } (s); \end{tikzpicture} \end{center}

Example references

The true and learned reward and transition tables share a finite state-action domain, and errors can be restricted to a pair set.

A fixed policy's finite-horizon return difference is bounded by reachable-pair reward and transition errors.

Facts & Assumptions

Given: The two-state models, supported set, initial law, horizon, and policy in the Example.

[F1]

Finite learned transition and reward models: the true and learned reward means lie in $[0, 1]$ , and each transition row is a probability distribution.

[F2]

Finite-horizon simulation bound for one fixed policy: for a common initial law and fixed Markov policy, uniform errors on the union of reachable pairs give the stated finite-horizon return bound.

Verification

Given: The finite models and policy in the Example.

1.1givenF1F2algebra

The true and learned occupancies under the policy place mass only on $(s, a)$ at time $0$ and on $(g, d)$ at each later time. These pairs are exactly the listed supported pairs, and the models agree on both, so the reachable reward and transition errors are zero. Theorem [F2] applies with $ϵ_{r} = ϵ_{p} = 0$ .

2.1step 1.1F2algebra∎

Substitution into [F2] gives $∣ J_{H} - {\hat{J}}_{H} ∣ \leq 0$ . Since an absolute value is nonnegative, $J_{H} = {\hat{J}}_{H}$ . Directly, each return is $1$ from the reward at the first step and $0$ thereafter. Thus model errors outside the fixed policy's support do not affect this return.

ExampleConstruction: AI-generatedVerification: AI-generatedprecheck pendingjudge pass (gpt-6-luna)Open item page →

A small transition error accumulates over longer rollouts

Example

Let the states be $u$ (safe) and $v$ (failure), each with one action. Both true and learned rewards are $1$ at $u$ and $0$ at $v$ . The true model stays at $u$ with probability $1$ , while the learned model moves from $u$ to $v$ with probability $ϵ$ and stays at $u$ with probability $1 - ϵ$ , where $0 < ϵ < 1$ ; both models keep $v$ absorbing. Starting at $u$ and following the only policy for horizon $H$ , the true return is $H$ and the learned return is $\sum_{t = 0}^{H - 1} (1 - ϵ)^{t}$ .

The diagram compares the true transition model on the left with the learned transition model on the right. Nodes are states, and arrows carry transition probabilities; each state label gives its reward. The dashed arrow is present only in the learned model.

\begin{center} \begin{tikzpicture}[>=stealth, node distance=25mm] \node[circle,draw] (u1) { $u, r = 1$ }; \node[circle,draw,right of=u1] (v1) { $v, r = 0$ }; \draw[->] (u1) to[loop above] node[above] { $1$ } (u1); \draw[->] (v1) to[loop right] node[right] { $1$ } (v1); \node at (1.25,-0.8) {true}; \begin{scope}[xshift=7cm] \node[circle,draw] (u2) at (0,0) { $u, r = 1$ }; \node[circle,draw] (v2) at (2.8,0) { $v, r = 0$ }; \draw[->] (u2) to[loop above] node[above] { $1 - ϵ$ } (u2); \draw[->,dashed] (u2) to[bend left=12] node[above] { $ϵ$ } (v2); \draw[->] (v2) to[loop right] node[right] { $1$ } (v2); \node at (1.4,-0.8) {learned}; \end{scope} \end{tikzpicture} \end{center}

Example references

The learned transition error is total variation on each state-action pair.

The fixed-policy return discrepancy is controlled by horizon and reachable transition error.

Facts & Assumptions

Given: The two-state models, initial state, policy, error parameter, and horizon in the Example.

[F1]

Finite learned transition and reward models: each model has probability transition rows and rewards in $[0, 1]$ ; transition error is total variation.

[F2]

Finite-horizon simulation bound for one fixed policy: a fixed policy with zero reward error and transition error at most $ϵ$ on its reachable pairs has return discrepancy at most $(\binom{H}{2}) ϵ$ from a common initial state.

Verification

Given: The finite model pair and policy in the Example.

1.1givenF1algebra

Under the true model, the state remains $u$ , so each of the $H$ rewards is $1$ and $J_{H} = H$ . Under the learned model, the probability of occupying $u$ at time $t$ is $(1 - ϵ)^{t}$ , by induction from the transition row, and the only positive reward is received at $u$ . Hence ${\hat{J}}_{H} = \sum_{t = 0}^{H - 1} (1 - ϵ)^{t}$ .

2.1givenF1F2step 1.1algebra∎

The reward error is zero. At $u$ , the total variation between the true row $(1, 0)$ and learned row $(1 - ϵ, ϵ)$ is $\frac{1}{2} (ϵ + ϵ) = ϵ$ ; at $v$ the rows agree. Every reachable pair therefore satisfies the hypotheses of [F2], and $H - {\hat{J}}_{H} \leq (\binom{H}{2}) ϵ$ . The exact difference from step 1.1 is $\sum_{t = 0}^{H - 1} [1 - (1 - ϵ)^{t}]$ , which is nondecreasing in $H$ and is positive for $H \geq 2$ . Thus even a fixed, small one-step transition error can produce a larger return discrepancy on longer rollouts, within the finite-horizon bound.

ExampleConstruction: AI-generatedVerification: AI-generatedprecheck pendingjudge pass (gpt-6-luna)Open item page →

A one-step optimizer selects a falsely rewarding unsupported action

Example

There is one state $s$ , two actions $a_{s a f e}$ and $a_{o f f}$ , and both actions return deterministically to $s$ . Their true rewards are $1 / 2$ and $0$ , respectively; their learned rewards are $1 / 2$ and $1$ . Let $H = 1$ , let $B = {(s, a_{s a f e})}$ , and let the finite policy class contain the two policies that choose one action at time $0$ . Supply MPC with the ordered action listing $(a_{s a f e}, a_{o f f})$ at $s$ . Finite-horizon model-predictive control chooses $a_{o f f}$ , whose predicted return is $1$ and true return is $0$ , while the true best return in the class is $1 / 2$ .

The diagram uses a vertex for the state and a loop for each deterministic self-transition; each loop label gives the action and its true/learned immediate rewards.

\begin{center} \begin{tikzpicture}[>=stealth] \node[circle,draw] (s) at (0,0) { $s$ }; \draw[->] (s) to[loop above] node[above] { $a_{s a f e} : 1 / 2, \hat{r} = 1 / 2$ } (s); \draw[->] (s) to[loop below] node[below] { $a_{o f f} : 0, \hat{r} = 1$ } (s); \end{tikzpicture} \end{center}

Example references

Off-support exploitation requires learned occupancy outside the data-supported set, overprediction, and true suboptimality.

Finite-horizon MPC returns a first action maximizing the learned-model finite-horizon value.

Facts & Assumptions

Given: The one-state model pair, data set, horizon, and policy class in the Example.

[F1]

Model exploitation under support shift: a policy is model-optimal if it attains the learned return maximum in the finite policy class; off-support exploitation additionally requires positive learned occupancy on an unsupported pair, overprediction, and true suboptimality.

[F2]

Finite-horizon model-predictive control: at horizon one, the returned action maximizes the learned immediate reward, with the complete supplied action listing used only to break ties.

Verification

Given: The finite model pair and class in the Example.

1.1givenF2algebra

The two horizon-one learned returns are $1 / 2$ and $1$ , so $a_{o f f}$ is the unique learned optimizer and [F2] returns it. Its learned occupancy on $(s, a_{o f f})$ is $1$ , although this pair is outside $B$ .

2.1givenF1step 1.1algebra∎

The true returns of $a_{s a f e}$ and $a_{o f f}$ are $1 / 2$ and $0$ . Thus the learned optimizer overpredicts its own return ( $1 > 0$ ) and is truly suboptimal ( $0 < 1 / 2$ ). All three clauses of [F1] hold, so this is off-support model exploitation as defined.