CounterexampleConstruction: AI-generatedVerification: AI-generatedPipeline-generatedjudge pass (gpt-6-luna)

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

Omitting the start-time discount changes the policy gradient

Statement refuted

For the fixed start-state discounted objective $J (θ) = v^{π_{θ}} (s_{0})$ and returns $G_{t} = \sum_{k = t}^{T - 1} γ^{k - t} R_{k + 1}$ , under episode termination, common-support differentiability, finite expected return/score products, and justified differentiation through the trajectory law, the identity

$\nabla J (θ) = E_{π_{θ}} [\sum_{t = 0}^{T - 1} \nabla_{θ} \log π_{θ} (A_{t} ∣ S_{t}) G_{t}]$

always holds.

Statement references

Finite MDP data and stationary policies.

Fixed positive policy support and score vectors.

Facts & Assumptions

Given: States $s_{0}, s_{1}, z$ , start $s_{0}$ , terminal set ${z}$ , discount $γ = 1 / 2$ , one action at $s_{0}$ leading to $s_{1}$ with reward $0$ , and actions $0, 1$ at $s_{1}$ leading to $z$ with reward equal to the action. At $z$ take one zero-reward self-transition action. For $θ \in (0, 1)$ , put $π_{θ} (1 ∣ s_{1}) = θ$ , $π_{θ} (0 ∣ s_{1}) = 1 - θ$ ; the single-action policies at the other states have probability one. The episode is the two-point probability space of terminal paths with masses $θ, 1 - θ$ .

[F1]

Finite Markov decision process with joint reward-transition law: finite normalized transition-reward masses and statewise normalized action probabilities specify a finite MDP and a stationary policy.

[F2]

Differentiable stationary stochastic policies and score vectors: the score on positive fixed support is the gradient of the logarithm of the policy mass.

[F3]

The natural logarithm has derivative 1/x and equals the integral from 1 to x of 1/t: $\log^{'} (x) = 1 / x$ for $x > 0$ .

Counterexample

Given: The finite model and two-point episode law above, with $G_{t} = \sum_{k = t}^{T - 1} γ^{k - t} R_{k + 1}$ .

1.1givenF1F2algebra

Every episode has $T = 2$ , $R_{1} = 0$ and $R_{2} = A_{1}$ . The given deterministic transition masses are normalized, and the displayed policy probabilities are normalized and positive on fixed support for every $θ \in (0, 1)$ . Both terminal-path masses are affine in $θ$ , so differentiation through their finite sum is exact with identically zero remainders; all score-return products are finite at every fixed interior parameter. Thus all the asserted regularity and termination conditions hold.

2.1givenF2F3step 1.1algebra∎

The objective is $J (θ) = E_{θ} [R_{1} + γ R_{2}] = θ / 2$ , so $\nabla J (θ) = 1 / 2$ . The time-zero score is zero. At time one the score is $1 / θ$ on action $1$ and $- 1 / (1 - θ)$ on action $0$ , while $G_{1} = A_{1}$ . Hence the claimed right-hand side is $θ (1 / θ) + (1 - θ) 0 = 1 \neq 1 / 2$ . Including the missing factor $γ^{1}$ changes this expectation to $1 / 2$ , as required.

Depends on

Used by

Nothing in the library uses this result yet.

Dependency tree · two levels

24 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.