CounterexampleConstruction: AI-generatedVerification: AI-generatedPipeline-generatedjudge pass (gpt-6-luna)

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

Disconnected comparison support does not identify Gibbs policy mass

Statement refuted

For one prompt with finite nonempty response set $Y$ , reference probability vector $q = (q_{y})_{y \in Y}$ with $q_{y} > 0$ and $\sum_{y} q_{y} = 1$ , coefficient $β > 0$ , and undirected graph $G = (Y, E)$ with $E \subseteq {{u, v} : u, v \in Y, u \neq v}$ and at least two connected components, the exact Bradley–Terry probabilities on every observed edge uniquely determine the Gibbs-optimal policy for every reward $r : Y \to R$ . For an edge ${u, v}$ the two orientations have probabilities $σ (r (u) - r (v))$ and $σ (r (v) - r (u))$ , where $σ (t) = (1 + e^{- t})^{- 1}$ . An edge is one observed pair, and two vertices lie in the same connected component exactly when a finite path of observed edges joins them.

Statement references

Finite Bradley–Terry preference model: gives the comparison probability from a reward difference.

Finite-action Gibbs policy theorem: gives the unique policy for each reward and reference.

Natural logarithm: gives $e^{\log 3} = 3$ in the witness.

Facts & Assumptions

Given: The finite response set $Y = {a, b, c, d}$ , uniform reference $q_{y} = 1 / 4$ , coefficient $β = 1$ , and observed edge set $E = {{a, b}, {c, d}}$ .

[F1]

Finite Bradley–Terry preference model: for a distinct pair, $\Pr (u ≻ v) = σ (r (u) - r (v))$ with $σ (t) = (1 + e^{- t})^{- 1}$ .

[F2]

The finite-action Gibbs policy uniquely maximizes reward minus KL: the unique regularized policy is $p_{y}^{*} = q_{y} e^{r (y) / β} / Z$ with $Z = \sum_{y} q_{y} e^{r (y) / β}$ .

[F3]

The natural logarithm as the inverse of the exponential function: $e^{\log t} = t$ for $t > 0$ .

Counterexample

1.1F1givenalgebra

Let $r_{0} (a) = r_{0} (b) = r_{0} (c) = r_{0} (d) = 0$ and let $r_{1} (a) = r_{1} (b) = 0$ , $r_{1} (c) = r_{1} (d) = \log 3$ . On each observed edge the two endpoint rewards agree under both functions, so [F1] gives the same comparison probability $σ (0) = 1 / 2$ .

2.1F2F3givenstep 1.1algebra

Under $r_{0}$ , [F2] has four unnormalized weights $1 / 4$ , so $Z_{0} = 1$ and $p_{0}^{*} = (1 / 4, 1 / 4, 1 / 4, 1 / 4)$ . Under $r_{1}$ , [F3] gives unnormalized weights $(1 / 4, 1 / 4, 3 / 4, 3 / 4)$ , hence $Z_{1} = 2$ and $p_{1}^{*} = (1 / 8, 1 / 8, 3 / 8, 3 / 8)$ .

3.1givenstep 1.1step 2.1algebra∎

The observed edge probabilities are identical by step 1.1, while the two Gibbs policies differ by step 2.1. Therefore the exact comparison probabilities on this disconnected graph do not uniquely determine the policy, refuting the Statement.

Comparison graph

The vertices below are responses, and an undirected edge records one observed pair. The two connected components are ${a, b}$ and ${c, d}$ ; there are no observed cross-component comparisons. This diagram only displays the stated edge set; the verification above uses the explicit set $E$ and formulas.

diagram failed to render:
\begin{tikzpicture}[every node/.style={circle,draw,minimum size=8mm}]
  \node (a) at (0,0) {$a$};
  \node (b) at (1.5,0) {$b$};
  \node (c) at (4,0) {$c$};
  \node (d) at (5.5,0) {$d$};
  \draw (a) -- (b);
  \draw (c) -- (d);
\end{tikzpicture}

Depends on

Used by

Nothing in the library uses this result yet.

Dependency tree · two levels

9 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Rafael Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, arXiv:2305.18290v3 (2024) (standard reference, not scraped)
Purdue course lecture notes, Preference Optimization (PrefOpt) (standard reference, not scraped)