TheoremStatement: Literature-sourcedProof: AI-adaptedPipeline-generatedprecheck pass

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

The group-centered score estimator has factor one minus one over G

Statement

Fix a prompt $q$ , a parameter $θ$ in the open domain of its finite response policy, a finite real reward function $r$ on the policy support, and an integer $G \geq 2$ . Let $Y_{1}, \dots, Y_{G}$ be independent responses with common law $π_{θ} (\cdot ∣ q)$ , let $R_{i} : = r (Y_{i})$ , let $S_{i} : = \nabla_{θ} \log π_{θ} (Y_{i} ∣ q)$ , and set $\overline{R} : = G^{- 1} \sum_{j = 1}^{G} R_{j}$ and $J (θ) : = E_{θ} [r (Y) ∣ q]$ . Then $E_{θ} [\frac{1}{G} \sum_{i = 1}^{G} (R_{i} - \overline{R}) S_{i} ∣ q] = (1 - \frac{1}{G}) \nabla_{θ} J (θ) .$ If instead ${\overline{R}}_{- i} : = (G - 1)^{- 1} \sum_{j \neq i} R_{j}$ , then $E_{θ} [\frac{1}{G} \sum_{i = 1}^{G} (R_{i} - {\overline{R}}_{- i}) S_{i} ∣ q] = \nabla_{θ} J (θ) .$

Statement references

The finite response policy, support, score vector, and expectation convention.

The finite score has zero mean and its reward-weighted mean is the gradient of expected reward.

Facts & Assumptions

Given: The finite policy at $q$ , its fixed positive support, a finite reward $r$ , and independent draws $Y_{1}, \dots, Y_{G}$ from that same policy.

[F1]

Finite truncated response spaces, correctness predicates, and verifiers: $Y_{q}$ is finite, $π_{θ}$ is normalized and positive on its fixed support, $S_{θ} (q, y) = \nabla_{θ} \log π_{θ} (y ∣ q)$ there, and independent draws have the product joint mass.

[F2]

Finite-response score mean and reward-gradient identities: $E_{θ} [S_{θ} (q, Y) ∣ q] = 0$ and $E_{θ} [r (Y) S_{θ} (q, Y) ∣ q] = \nabla_{θ} J (θ)$ .

[F3]

Probability measures and probability spaces, Expectation of a real random variable on a finite probability space, Linearity, monotonicity, and the modulus bound for expectation, and Finite sums and finite products, by recursion: the finite group sample law is a probability measure, scalar expectations are finite weighted sums, and finite sums are recursively defined. Each variable is bounded and integrable because it is a real-valued function on the finite group response space, so the expectation linearity rule applies. Field supplies distributivity for the finite expansions below.

[F4]

Independent random elements and Independent sigma-algebras and independent events: singleton response events from distinct independent draws have product probabilities; together with [F1]'s finite response law this gives the product pair masses used below.

[F5]

Use the one-based coordinate labels $X^{(a)} : = X (a - 1)$ and $\partial_{θ_{a}} : = \partial_{a - 1}$ , $1 \leq a \leq d$ , as in the score-identity supplier. Every score vector lies in $R^{d}$ ; write $S_{i}^{(a)} : = (S_{i})_{a}$ for its $a$ th coordinate. Each vector expectation in this proof is defined coordinatewise by the finite scalar expectation in that coordinate; vector sums and equalities are coordinatewise. Accordingly, [F3] is applied to scalar coordinates, and the finite-sum factorization below is shown in each coordinate.

Proof

Given: The Statement hypotheses and the scalar finite-expectation convention in [F3], extended coordinatewise as specified in [F5].

1.1givenF2F3F5algebra

For each coordinate $a = 1, \dots, d$ , [F2] gives $E [R_{i} S_{i}^{(a)} ∣ q] = \partial_{θ_{a}} J (θ)$ and $E [S_{i}^{(a)} ∣ q] = 0$ for every $i$ . Scalar linearity from [F3] yields $E [G^{- 1} \sum_{i} R_{i} S_{i}^{(a)} ∣ q] = \partial_{θ_{a}} J (θ)$ . Since this holds in every coordinate, [F5] gives $E [G^{- 1} \sum_{i} R_{i} S_{i} ∣ q] = \nabla_{θ} J (θ)$ .

1.2F1F2F3F4F5algebrainduction

For $i \neq j$ and each coordinate $a$ , expand the finite expectation over response pairs $(Y_{i}, Y_{j}) = (b_{q, k}, b_{q, ℓ})$ . By [F1] and [F4], their joint mass is $π_{θ} (b_{q, k} ∣ q) π_{θ} (b_{q, ℓ} ∣ q)$ . The recursive finite-sum clauses in [F3] and field distributivity in [F3] factor the pair sum: induction over the $ℓ$ -list first extracts the fixed $π_{θ} (b_{q, k} ∣ q) S_{θ, a} (q, b_{q, k})$ , and induction over the $k$ -list then gives $E [R_{j} S_{i}^{(a)} ∣ q] = (\sum_{k = 1}^{n_{q}} π_{θ} (b_{q, k} ∣ q) S_{θ, a} (q, b_{q, k})) (\sum_{ℓ = 1}^{n_{q}} π_{θ} (b_{q, ℓ} ∣ q) r (b_{q, ℓ})) = 0$ by [F2]. This proves the zero cross expectation in every coordinate under [F5].

2.1givenF2F3F5step 1.1step 1.2algebra

Expand $\overline{R} = G^{- 1} \sum_{j} R_{j}$ , so $G^{- 1} \sum_{i} \overline{R} S_{i} = G^{- 2} \sum_{i, j} R_{j} S_{i}$ . For each coordinate $a$ , scalar linearity from [F3], the $G$ diagonal expectations $\partial_{θ_{a}} J (θ)$ from [F2], and the zero cross expectations from step 1.2 give $E [G^{- 1} \sum_{i} \overline{R} S_{i}^{(a)} ∣ q] = G^{- 2} G \partial_{θ_{a}} J (θ) = G^{- 1} \partial_{θ_{a}} J (θ)$ . By [F5] this is the coordinatewise vector identity. Subtracting it from step 1.1 proves the first identity.

3.1givenF3F5step 1.1step 1.2algebra∎

For each $i$ and coordinate $a$ , every term in ${\overline{R}}_{- i} S_{i}^{(a)}$ has the form $R_{j} S_{i}^{(a)}$ with $j \neq i$ , so scalar linearity and step 1.2 give $E [{\overline{R}}_{- i} S_{i}^{(a)} ∣ q] = 0$ . This holds for every coordinate, so [F5] gives the zero vector expectation. Subtracting these zero means from step 1.1 proves the leave-one-out identity coordinatewise.

Remarks

This identity covers unnormalized same-policy centering and the leave-one-out center. It does not cover division by a random group standard deviation, clipped probability ratios, or samples reused from a frozen old policy; none of those modifications inherits an unbiasedness claim from this calculation alone.

Depends on

Used by

Reasoning-training evidence and evaluation targets Remark

Dependency tree · two levels

39 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, original arXiv v1 (standard reference, not scraped)
Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, second-edition draft (standard reference, not scraped)