Preference coverage and Bradley–Terry model limits

The Bradley–Terry equation is a modeling assumption for pairwise comparison noise, not a universal law of human preference. Here, model misspecification means that the actual pairwise conditional probabilities are not generated by any reward in this Bradley–Terry model. The Gibbs and direct preference optimization (DPO) formulas proved here use a finite response set, a full-support reference, and finite real rewards. For a countably infinite response space, the normalizing sum $Z={\sum }_y{q}_y{e}^{r(y)\mathrm{/}\beta }$ would have to be positive and finite, and the reward and KL sums would need convergence conditions that make them well-defined; no such extension is claimed.

For a fixed prompt, let $G=(Y,E)$ be the graph whose vertices are the finite responses and whose edges are the observed comparisons, with each edge joining the two responses compared. A connected component is a maximal subset of vertices in which every two vertices can be joined by a path of edges in $E$. If $G$ has connected components $C_1,\dots ,C_k$, adding a constant $c_j$ to every reward in $C_j$ leaves every observed edge difference unchanged because both endpoints of each edge lie in one component. Thus the Bradley–Terry probabilities on observed edges stay the same. But the Gibbs mass of component $C_j$ is $p^{\ast }(C_j)=\frac{{\sum }_{y\in C_j}{q}_y{e}^{r(y)\mathrm{/}\beta }}{{\sum }_{\mathrm{\ell }=1}^k{\sum }_{y\in C_{\mathrm{\ell }}}{q}_y{e}^{r(y)\mathrm{/}\beta }},$ so after the component shift its numerator and corresponding denominator contribution are multiplied by $e^{c_j\mathrm{/}\beta }$. With full-support $q$, distinct component shifts can therefore change the policy's relative mass across components without changing any observed comparison probability. Pairwise data on a disconnected graph alone do not identify that policy mass.

Here, identifying a policy from comparisons means uniquely determining it from the exact probabilities on the observed edges. The finite DPO loss and finite update procedure do not by themselves show that minimizing empirical loss recovers the true policy, that the parameterized family can represent that policy, that the learned policy performs well on new prompts or comparisons, or that the updates converge to a minimizer. Such claims require additional assumptions on the comparison sampling process, model class, and optimization behavior; none is supplied here.