Bradley–Terry reward-policy reparameterization and finite DPO gradient

For each prompt $x$ under consideration, let $Y_x$ be a finite response set with at least two elements, let $q_x$ be a probability vector with $q_x(y)>0$ for every $y\in Y_x$, let $r_x:Y_x\to \mathbb{R}$, and let $\beta >0$. Define $Z_x={\sum }_{y\in Y_x}{q}_x(y)e^{r_x(y)\mathrm{/}\beta }$ and $p_x^{\ast }(y)=q_x(y)e^{r_x(y)\mathrm{/}\beta }\mathrm{/}{Z}_x$. For every distinct $a,b\in Y_x$, $r_x(a)-r_x(b)=\beta \log \frac{p_x^{\ast }(a)}{q_x(a)}-\beta \log \frac{p_x^{\ast }(b)}{q_x(b)},$ so the Bradley–Terry probability is $\mathrm{Pr}(a\succ b\mid x)=\sigma \text{\hspace{0.17em}⁣}(\beta \log \frac{p_x^{\ast }(a)}{q_x(a)}-\beta \log \frac{p_x^{\ast }(b)}{q_x(b)})\mathrm{.}$ Conversely, every full-support probability vector ${\pi }_x$ on $Y_x$ is the unique Gibbs policy for reward ${\widehat{r}}_x(y)=\beta \log ({\pi }_x(y)\mathrm{/}{q}_x(y))$; adding any prompt-only constant $c_x$ to this reward leaves both the Gibbs policy and the Bradley–Terry probabilities unchanged.

Let $D=((x_i,y_i^w,y_i^l){)}_{i=1}^n$ be a finite ordered dataset with $n\ge 1$, $y_i^w,y_i^l\in Y_{x_i}$, and $y_i^w\mathrm{\ne }{y}_i^l$. Let $\mathrm{\Theta }\subseteq {\mathbb{R}}^d$ be open and let each conditional policy ${\pi }_{\theta }(\cdot \mid x_i)$ be a probability vector with ${\pi }_{\theta }(y\mid x_i)>0$ for every $y\in Y_{x_i}$, whose coordinates are continuously differentiable in $\theta \in \mathrm{\Theta }$. Put $u_i(\theta )=\beta (\log \frac{{\pi }_{\theta }(y_i^w\mid x_i)}{q_{x_i}(y_i^w)}-\log \frac{{\pi }_{\theta }(y_i^l\mid x_i)}{q_{x_i}(y_i^l)}),{\mathcal{L}}_D(\theta )=\frac{1}{n}{\sum }_{i=1}^n\log (1+e^{-u_i(\theta )})\mathrm{.}$ Then ${\mathcal{L}}_D$ is the finite direct preference optimization (DPO) loss: the average negative log likelihood for the Bradley–Terry model with implicit reward ${\widehat{r}}_{\theta }(x_i,y)=\beta \log \text{\hspace{0.17em}⁣}({\pi }_{\theta }(y\mid x_i)\mathrm{/}{q}_{x_i}(y))$. Its Euclidean gradient is ${\mathrm{\nabla }}_{\theta }{\mathcal{L}}_D(\theta )=-\frac{\beta }{n}{\sum }_{i=1}^n\sigma (-u_i(\theta ))({\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(y_i^w\mid x_i)-{\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(y_i^l\mid x_i))\mathrm{.}$ Here ${\mathrm{\nabla }}_{\theta }$ is the column vector of Euclidean partial derivatives on the open set $\mathrm{\Theta }$.

Finite Bradley–Terry preference model: defines $\mathrm{Pr}(a\succ b\mid x)=\sigma (r_x(a)-r_x(b))$ and the sigmoid.

Finite KL-regularized policy improvement: defines the finite KL objective and its reward/reference conventions.

Finite-action Gibbs policy theorem: gives the unique optimizer $p_x^{\ast }$ and its normalization.

Chain rule for total derivatives: differentiates the finite composition of the loss and the conditional log-probability coordinates.

Natural logarithm derivative: supplies $(\log t{)}^{\mathrm{\prime }}=1\mathrm{/}t$ for $t>0$.

Exponential derivative: supplies $(e^t{)}^{\mathrm{\prime }}=e^t$ for real $t$.

Rafailov et al., §4 Eqs. (6)–(7), Appendix A.2 and A.4

Purdue PrefOpt course notes, §§3e–4a