CounterexampleConstruction: AI-generatedVerification: AI-generatedPipeline-generatedjudge pass (gpt-6-luna)

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

Zero reference support makes the DPO reward ratio undefined

Statement refuted

The finite reward-policy identity $r (y) = β \log (p_{y}^{*} / q_{y}) + β \log Z$ , where $Z = \sum_{y \in Y} q_{y} e^{r (y) / β}$ and $p^{*}$ maximizes the extended objective $F_{q, r, β}$ , remains defined and valid at every response even when the reference probability vector $q$ has zero coordinates.

Statement references

Finite KL-regularized policy improvement: defines the extended finite objective and its support convention.

DPO reward-policy reparameterization: states the logarithmic identity under the full-support reference hypothesis.

Natural logarithm: defines $\log$ only for positive arguments.

Facts & Assumptions

Given: $Y = {a, b}$ , $q_{a} = 1$ , $q_{b} = 0$ , $β = 1$ , $r (a) = 0$ , and $r (b) = 1$ .

[F1]

Finite KL-regularized policy improvement: if $p_{y} > 0$ at a coordinate where $q_{y} = 0$ , then $D (p ∥ q) = + \infty$ ; if $p$ is supported on $q$ , the finite KL sum uses only positive coordinates.

[F2]

Bradley–Terry reward-policy reparameterization and finite DPO gradient: the proved reward-policy formula assumes the reference vector is strictly positive on every response.

[F3]

The natural logarithm as the inverse of the exponential function: the real logarithm has domain $(0, \infty)$ .

Counterexample

1.1F1givenalgebra

If a probability vector $p$ on $Y$ has $p_{b} > 0$ , [F1] gives $D (p ∥ q) = + \infty$ and hence $F_{q, r, 1} (p) = - \infty$ , since the reward expectation is finite. If $p_{b} = 0$ , then $p = δ_{a}$ , $D (p ∥ q) = 0$ , and $F_{q, r, 1} (p) = 0$ . Thus $p^{*} = δ_{a}$ is the unique maximizer.

2.1F2F3givenstep 1.1algebra∎

At response $b$ , $p_{b}^{*} = q_{b} = 0$ , so $p_{b}^{*} / q_{b}$ is the undefined ratio $0 / 0$ and is not in the domain of the logarithm in [F3]. The stated identity therefore cannot be defined at every response. This example violates the full-support hypothesis of [F2], showing that the identity cannot be extended to zero reference support as stated.

Depends on

Used by

Nothing in the library uses this result yet.

Dependency tree · two levels

10 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Rafael Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, arXiv:2305.18290v3 (2024) (standard reference, not scraped)
Purdue course lecture notes, Preference Optimization (PrefOpt) (standard reference, not scraped)