ExampleConstruction: AI-generatedVerification: AI-generatedPipeline-generatedjudge pass (gpt-6-luna)

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

A state baseline can eliminate this estimator's variance

Example

Use the one-decision MDP with states ${s, d}$ , terminal state $d$ , actions ${0, 1}$ at $s$ , one action at $d$ , and deterministic transitions from $(s, 1)$ to $(d, 1)$ , from $(s, 0)$ to $(d, 0)$ , and from $d$ to $(d, 0)$ . All unlisted joint masses are zero. Choose discount $0 \leq γ < 1$ . For $θ \in (0, 1)$ , set $π_{θ} (1 ∣ s) = θ$ and $π_{θ} (0 ∣ s) = 1 - θ$ , and start at $s$ . At $θ = 1 / 2$ , the true objective is $J (θ) = θ$ with gradient $1$ . Let $Z = G_{0} \nabla_{θ} \log π_{θ} (A_{0} ∣ s)$ be the unscaled REINFORCE sample direction, and use the action-independent state baseline $b (s) = 1 / 2$ . Then $Z$ has variance $1$ , while $Z_{b} = (G_{0} - b (s)) \nabla_{θ} \log π_{θ} (A_{0} ∣ s)$ is identically $1$ and has variance $0$ ; both have expectation $1$ .

Example references

The frozen-policy episodic REINFORCE update.

The episodic state-baseline update.

A state-dependent action-independent baseline preserves the expected gradient.

Facts & Assumptions

Given: The stated finite one-decision MDP, $0 \leq γ < 1$ , $θ = 1 / 2$ , the REINFORCE estimator, and the fixed baseline $b (s) = 1 / 2$ .

[F1]

Finite Markov decision process with joint reward-transition law: each state-action pair has the specified normalized joint reward-transition law.

[F2]

Differentiable stationary stochastic policies and score vectors: the policy has fixed positive support and score $\nabla_{θ} \log π_{θ} (a ∣ s)$ on the support.

[F3]

Episodic returns, temporal-difference errors, and policy support: the one-step discounted return is $G_{0} = R_{1}$ .

[F4]

Episodic REINFORCE with a frozen episode policy and Episodic REINFORCE with a fixed state baseline: the unscaled sample directions are $G_{0} ψ_{0}$ and $(G_{0} - b (S_{0})) ψ_{0}$ respectively; each parameter increment multiplies its direction by the chosen step size.

[F5]

A state-dependent action-independent baseline preserves the gradient mean: the expected baseline-adjusted episodic update equals the policy gradient under its stated integrability and trajectory assumptions.

[F6]

The natural logarithm has derivative 1/x and equals the integral from 1 to x of 1/t: for $x > 0$ , $\log^{'} (x) = 1 / x$ .

[F7]

The total (Fréchet) derivative $D f (a)$ as the linear first-order approximation with $o (∥ h ∥_{2})$ remainder: the derivative of the identity map $θ \mapsto θ$ is the identity linear map.

Verification

Given: The Example data and facts above.

1.1givenF1F2F3F6F7algebra

At $θ = 1 / 2$ , the two actions have probability $1 / 2$ and scores $2$ and $- 2$ by [F2] and [F6]. The specified deterministic reward-transition law [F1] and return convention [F3] give $G_{0} = A_{0}$ and $J (θ) = θ$ , so the true gradient is $1$ by [F7].

2.1givenstep 1.1algebra

Without a baseline, $Z = 2$ when $A_{0} = 1$ and $Z = 0$ when $A_{0} = 0$ . Therefore $E [Z] = 1$ and $Var (Z) = \frac{1}{2} (2 - 1)^{2} + \frac{1}{2} (0 - 1)^{2} = 1$ .

2.2givenstep 1.1F4algebra

With $b (s) = 1 / 2$ , the baseline-adjusted value is $(1 - 1 / 2) 2 = 1$ when $A_{0} = 1$ and $(0 - 1 / 2) (- 2) = 1$ when $A_{0} = 0$ . Thus $Z_{b} = 1$ surely, so $E [Z_{b}] = 1$ and $Var (Z_{b}) = 0$ .

3.1step 1.1step 2.1step 2.2F5algebra∎

The two estimators have the same expectation, equal to the true gradient in step 1.1, while their variances are $1$ and $0$ by steps 2.1 and 2.2. This verifies variance reduction for this baseline and this MDP only.

Remarks

[A1] Axiom of Choice: every family of nonempty sets has a choice function; the referenced episodic algorithms and baseline lemma use the policy path law under AC.

Remarks references

The Axiom of Choice.

Depends on

Used by

Nothing in the library uses this result yet.

Dependency tree · two levels

43 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.