Pipeline-generated

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

Preference learning and direct preference optimization — Examples

1 · Prerequisites

2 · Logical flowchart

3 · Definitions, theorems and proofs

None yet.

4 · Examples, counterexamples and false statements

ExampleConstruction: AI-generatedVerification: AI-generatedjudge pass (gpt-6-luna)Open item page →

A two-response Gibbs policy with exact probabilities

Example

Take $Y = {a, b}$ , reference probabilities $q_{a} = q_{b} = 1 / 2$ , any $β > 0$ , and rewards $r (a) = 0$ and $r (b) = β \log 3$ . The partition sum is $Z = 2$ , the unique maximizing policy is $p^{*} (a) = 1 / 4$ , $p^{*} (b) = 3 / 4$ , and the maximum objective value is $F_{q, r, β} (p^{*}) = β \log 2$ .

Example references

Finite KL-regularized policy improvement: defines the objective and partition inputs.

Finite-action Gibbs policy theorem: gives the maximizing policy and objective-gap identity.

Natural logarithm: ensures $e^{\log 3} = 3$ .

Facts & Assumptions

Given: $Y = {a, b}$ , $q = (1 / 2, 1 / 2)$ , $β > 0$ , and $r (a) = 0$ , $r (b) = β \log 3$ .

[F1]

Finite KL-regularized policy improvement: $F_{q, r, β} (p) = \sum_{y} p_{y} r (y) - β D (p ∥ q)$ on the finite probability simplex.

[F2]

The finite-action Gibbs policy uniquely maximizes reward minus KL: $Z = \sum_{y} q_{y} e^{r (y) / β}$ , $p_{y}^{*} = q_{y} e^{r (y) / β} / Z$ , and $F (p^{*}) - F (p) = β D (p ∥ p^{*})$ , with equality only at $p = p^{*}$ .

[F3]

The natural logarithm as the inverse of the exponential function: $e^{\log t} = t$ for every $t > 0$ .

Verification

1.1F2F3givenalgebra

By [F3], $e^{r (a) / β} = 1$ and $e^{r (b) / β} = e^{\log 3} = 3$ . Hence $Z = (1 / 2) 1 + (1 / 2) 3 = 2$ , and [F2] gives $p^{*} (a) = 1 / 4$ and $p^{*} (b) = 3 / 4$ .

2.1F1F2givenstep 1.1algebra∎

By [F1], the objective being maximized is $F_{q, r, β}$ . Its gap identity in [F2] gives $F (p^{*}) - F (p) = β D (p ∥ p^{*}) \geq 0$ for every probability vector $p$ , with equality only at $p = p^{*}$ . Substituting $Z = 2$ into $F (p^{*}) = β \log Z$ gives the maximum value $β \log 2$ .

CounterexampleConstruction: AI-generatedVerification: AI-generatedjudge pass (gpt-6-luna)Open item page →

Disconnected comparison support does not identify Gibbs policy mass

Statement refuted

For one prompt with finite nonempty response set $Y$ , reference probability vector $q = (q_{y})_{y \in Y}$ with $q_{y} > 0$ and $\sum_{y} q_{y} = 1$ , coefficient $β > 0$ , and undirected graph $G = (Y, E)$ with $E \subseteq {{u, v} : u, v \in Y, u \neq v}$ and at least two connected components, the exact Bradley–Terry probabilities on every observed edge uniquely determine the Gibbs-optimal policy for every reward $r : Y \to R$ . For an edge ${u, v}$ the two orientations have probabilities $σ (r (u) - r (v))$ and $σ (r (v) - r (u))$ , where $σ (t) = (1 + e^{- t})^{- 1}$ . An edge is one observed pair, and two vertices lie in the same connected component exactly when a finite path of observed edges joins them.

Statement references

Finite Bradley–Terry preference model: gives the comparison probability from a reward difference.

Finite-action Gibbs policy theorem: gives the unique policy for each reward and reference.

Natural logarithm: gives $e^{\log 3} = 3$ in the witness.

Facts & Assumptions

Given: The finite response set $Y = {a, b, c, d}$ , uniform reference $q_{y} = 1 / 4$ , coefficient $β = 1$ , and observed edge set $E = {{a, b}, {c, d}}$ .

[F1]

Finite Bradley–Terry preference model: for a distinct pair, $\Pr (u ≻ v) = σ (r (u) - r (v))$ with $σ (t) = (1 + e^{- t})^{- 1}$ .

[F2]

The finite-action Gibbs policy uniquely maximizes reward minus KL: the unique regularized policy is $p_{y}^{*} = q_{y} e^{r (y) / β} / Z$ with $Z = \sum_{y} q_{y} e^{r (y) / β}$ .

[F3]

The natural logarithm as the inverse of the exponential function: $e^{\log t} = t$ for $t > 0$ .

Counterexample

1.1F1givenalgebra

Let $r_{0} (a) = r_{0} (b) = r_{0} (c) = r_{0} (d) = 0$ and let $r_{1} (a) = r_{1} (b) = 0$ , $r_{1} (c) = r_{1} (d) = \log 3$ . On each observed edge the two endpoint rewards agree under both functions, so [F1] gives the same comparison probability $σ (0) = 1 / 2$ .

2.1F2F3givenstep 1.1algebra

Under $r_{0}$ , [F2] has four unnormalized weights $1 / 4$ , so $Z_{0} = 1$ and $p_{0}^{*} = (1 / 4, 1 / 4, 1 / 4, 1 / 4)$ . Under $r_{1}$ , [F3] gives unnormalized weights $(1 / 4, 1 / 4, 3 / 4, 3 / 4)$ , hence $Z_{1} = 2$ and $p_{1}^{*} = (1 / 8, 1 / 8, 3 / 8, 3 / 8)$ .

3.1givenstep 1.1step 2.1algebra∎

The observed edge probabilities are identical by step 1.1, while the two Gibbs policies differ by step 2.1. Therefore the exact comparison probabilities on this disconnected graph do not uniquely determine the policy, refuting the Statement.

Comparison graph

The vertices below are responses, and an undirected edge records one observed pair. The two connected components are ${a, b}$ and ${c, d}$ ; there are no observed cross-component comparisons. This diagram only displays the stated edge set; the verification above uses the explicit set $E$ and formulas.

ExampleConstruction: AI-generatedVerification: AI-generatedOpen item page →

A prompt-only reward offset leaves Bradley–Terry comparisons and policy unchanged

Example

For a fixed prompt $x$ , let $Y_{x}$ be finite and nonempty, let $q_{x}$ be a probability vector with $q_{x} (y) > 0$ for every $y \in Y_{x}$ , let $β > 0$ , and let $r_{x} : Y_{x} \to R$ . For any constant $c_{x} \in R$ , set $r_{x}^{'} (y) = r_{x} (y) + c_{x}$ for every $y \in Y_{x}$ . Then every Bradley–Terry comparison probability at $x$ and the unique Gibbs-optimal policy on $Y_{x}$ are unchanged.

Example references

Finite Bradley–Terry preference model: comparison probabilities depend on reward differences for a fixed prompt.

Finite KL-regularized policy improvement: defines the finite objective being optimized.

Finite-action Gibbs policy theorem: supplies the normalized policy formula.

Rafailov et al., §5.1 and Theorem 1

Purdue PrefOpt course notes, §2 and §3e

Facts & Assumptions

Given: A finite response set $Y_{x}$ , full-support reference $q_{x}$ , positive $β$ , reward $r_{x}$ , and constant $c_{x}$ .

[F1]

Finite Bradley–Terry preference model: $\Pr (a ≻ b ∣ x) = σ (r_{x} (a) - r_{x} (b))$ for distinct responses.

[F2]

Finite KL-regularized policy improvement: the finite objective is $F_{q_{x}, r_{x}, β} (p) = \sum_{y} p_{y} r_{x} (y) - β D (p ∥ q_{x})$ .

[F3]

The finite-action Gibbs policy uniquely maximizes reward minus KL: the Gibbs policy has unnormalized weights $q_{x} (y) e^{r_{x} (y) / β}$ and partition sum $Z_{x}$ .

Verification

1.1F1givenalgebra

For any distinct $a, b \in Y_{x}$ , $r_{x}^{'} (a) - r_{x}^{'} (b) = (r_{x} (a) + c_{x}) - (r_{x} (b) + c_{x}) = r_{x} (a) - r_{x} (b)$ .

2.1F2F3givenalgebra∎

The objective being optimized is the one in [F2], and [F3] gives its normalized weights. The shifted unnormalized weight is $q_{x} (y) e^{r_{x}^{'} (y) / β} = e^{c_{x} / β} q_{x} (y) e^{r_{x} (y) / β}$ , so $Z_{x}^{'} = e^{c_{x} / β} Z_{x}$ . Dividing each shifted weight by $Z_{x}^{'}$ cancels the common factor and gives $p_{x}^{*^{'}} (y) = p_{x}^{*} (y)$ for every $y$ .

CounterexampleConstruction: AI-generatedVerification: AI-generatedjudge pass (gpt-6-luna)Open item page →

Zero reference support makes the DPO reward ratio undefined

Statement refuted

The finite reward-policy identity $r (y) = β \log (p_{y}^{*} / q_{y}) + β \log Z$ , where $Z = \sum_{y \in Y} q_{y} e^{r (y) / β}$ and $p^{*}$ maximizes the extended objective $F_{q, r, β}$ , remains defined and valid at every response even when the reference probability vector $q$ has zero coordinates.

Statement references

Finite KL-regularized policy improvement: defines the extended finite objective and its support convention.

DPO reward-policy reparameterization: states the logarithmic identity under the full-support reference hypothesis.

Natural logarithm: defines $\log$ only for positive arguments.

Facts & Assumptions

Given: $Y = {a, b}$ , $q_{a} = 1$ , $q_{b} = 0$ , $β = 1$ , $r (a) = 0$ , and $r (b) = 1$ .

[F1]

Finite KL-regularized policy improvement: if $p_{y} > 0$ at a coordinate where $q_{y} = 0$ , then $D (p ∥ q) = + \infty$ ; if $p$ is supported on $q$ , the finite KL sum uses only positive coordinates.

[F2]

Bradley–Terry reward-policy reparameterization and finite DPO gradient: the proved reward-policy formula assumes the reference vector is strictly positive on every response.

[F3]

The natural logarithm as the inverse of the exponential function: the real logarithm has domain $(0, \infty)$ .

Counterexample

1.1F1givenalgebra

If a probability vector $p$ on $Y$ has $p_{b} > 0$ , [F1] gives $D (p ∥ q) = + \infty$ and hence $F_{q, r, 1} (p) = - \infty$ , since the reward expectation is finite. If $p_{b} = 0$ , then $p = δ_{a}$ , $D (p ∥ q) = 0$ , and $F_{q, r, 1} (p) = 0$ . Thus $p^{*} = δ_{a}$ is the unique maximizer.

2.1F2F3givenstep 1.1algebra∎

At response $b$ , $p_{b}^{*} = q_{b} = 0$ , so $p_{b}^{*} / q_{b}$ is the undefined ratio $0 / 0$ and is not in the domain of the logarithm in [F3]. The stated identity therefore cannot be defined at every response. This example violates the full-support hypothesis of [F2], showing that the identity cannot be extended to zero reference support as stated.

Preference learning and direct preference optimization — Examples

2 · Logical flowchart

3 · Definitions, theorems and proofs

4 · Examples, counterexamples and false statements

A two-response Gibbs policy with exact probabilities

Example

Example references

Facts & Assumptions

Verification

Disconnected comparison support does not identify Gibbs policy mass

Statement refuted

Statement references

Facts & Assumptions

Counterexample

Comparison graph

A prompt-only reward offset leaves Bradley–Terry comparisons and policy unchanged

Example

Example references

Facts & Assumptions

Verification

Zero reference support makes the DPO reward ratio undefined

Statement refuted

Statement references

Facts & Assumptions

Counterexample

Sources