ExampleConstruction: AI-generatedVerification: AI-generatedPipeline-generated

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

A prompt-only reward offset leaves Bradley–Terry comparisons and policy unchanged

Example

For a fixed prompt $x$ , let $Y_{x}$ be finite and nonempty, let $q_{x}$ be a probability vector with $q_{x} (y) > 0$ for every $y \in Y_{x}$ , let $β > 0$ , and let $r_{x} : Y_{x} \to R$ . For any constant $c_{x} \in R$ , set $r_{x}^{'} (y) = r_{x} (y) + c_{x}$ for every $y \in Y_{x}$ . Then every Bradley–Terry comparison probability at $x$ and the unique Gibbs-optimal policy on $Y_{x}$ are unchanged.

Example references

Finite Bradley–Terry preference model: comparison probabilities depend on reward differences for a fixed prompt.

Finite KL-regularized policy improvement: defines the finite objective being optimized.

Finite-action Gibbs policy theorem: supplies the normalized policy formula.

Rafailov et al., §5.1 and Theorem 1

Purdue PrefOpt course notes, §2 and §3e

Facts & Assumptions

Given: A finite response set $Y_{x}$ , full-support reference $q_{x}$ , positive $β$ , reward $r_{x}$ , and constant $c_{x}$ .

[F1]

Finite Bradley–Terry preference model: $\Pr (a ≻ b ∣ x) = σ (r_{x} (a) - r_{x} (b))$ for distinct responses.

[F2]

Finite KL-regularized policy improvement: the finite objective is $F_{q_{x}, r_{x}, β} (p) = \sum_{y} p_{y} r_{x} (y) - β D (p ∥ q_{x})$ .

[F3]

The finite-action Gibbs policy uniquely maximizes reward minus KL: the Gibbs policy has unnormalized weights $q_{x} (y) e^{r_{x} (y) / β}$ and partition sum $Z_{x}$ .

Verification

1.1F1givenalgebra

For any distinct $a, b \in Y_{x}$ , $r_{x}^{'} (a) - r_{x}^{'} (b) = (r_{x} (a) + c_{x}) - (r_{x} (b) + c_{x}) = r_{x} (a) - r_{x} (b)$ .

2.1F2F3givenalgebra∎

The objective being optimized is the one in [F2], and [F3] gives its normalized weights. The shifted unnormalized weight is $q_{x} (y) e^{r_{x}^{'} (y) / β} = e^{c_{x} / β} q_{x} (y) e^{r_{x} (y) / β}$ , so $Z_{x}^{'} = e^{c_{x} / β} Z_{x}$ . Dividing each shifted weight by $Z_{x}^{'}$ cancels the common factor and gives $p_{x}^{*^{'}} (y) = p_{x}^{*} (y)$ for every $y$ .

Depends on

Used by

Nothing in the library uses this result yet.

Dependency tree · two levels

8 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Rafael Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, arXiv:2305.18290v3 (2024) (standard reference, not scraped)
Purdue course lecture notes, Preference Optimization (PrefOpt) (standard reference, not scraped)