The generalized Bayes loss update uniquely minimizes loss plus KL

Let $g(\theta )=e^{-w{\mathrm{\ell }}_x(\theta )}\mathrm{/}Z$. By [F1]–[F2], $q$ is the probability measure with density $g$ relative to $\pi$. Since ${\mathrm{\ell }}_x$ is real-valued, [F7] gives $g>0$ everywhere.

For $t>0$, put $u(t)=\log t-1+t^{-1}$. By [F10], $u^{\mathrm{\prime }}(t)=(t-1)\mathrm{/}{t}^2$. If $0<t<1$, the mean value theorem on $[t,1]$ gives $u(t)>u(1)=0$; if $t>1$, it on $[1,t]$ gives $u(t)>u(1)=0$. Thus $\log t\ge 1-t^{-1}$. Multiplying by $t>0$ gives $t\log t\ge t-1$, with equality only at $t=1$. At $t=0$ the inequality also holds under $0\log 0:=0$.

The update is admissible: $q\ll \pi$ by [F1], and [F4], [F9] give $D(q\mathrm{\parallel }\pi )=\int \log gdq=-w\int {\mathrm{\ell }}_{x}dq-\log Z,$ which is finite by the loss-integrability hypothesis and $0<Z<\mathrm{\infty }$. Thus $q\in \mathcal{V}$.

Fix $\nu \in \mathcal{V}$. If $q(A)=0$, then for every integer $m\ge 1$, [F8] and [F1] give $0\le m^{-1}\pi (A\cap \{g\ge m^{-1}\})\le {\int }_{A}gd\pi =q(A)=0.$ Let $A_1=A\cap \{g\ge 1\}$ and, for $m\ge 2$, let $A_m=(A\cap \{g\ge m^{-1}\})\setminus (A\cap \{g\ge (m-1{)}^{-1}\})$. These sets are disjoint, each is $\pi$-null by the displayed bound, and their union is $A$ because $g>0$ everywhere. Countable additivity [F15] therefore gives $\pi (A)=0$. Hence $\pi \ll q$, and $\nu \ll q$ follows from $\nu \ll \pi$ and [F5].

The measures $\nu ,q,\pi$ are finite probabilities. The constant exhaustion $X_m=\mathrm{\Theta }$ is therefore a common finite exhaustion, so [A1] and [F6] apply to $\nu \ll q\ll \pi$. With $r=d\nu \mathrm{/}d\pi$ and $h=d\nu \mathrm{/}dq$, the chain rule gives $r=hg$ $\pi$-almost everywhere.

The Radon–Nikodym formula [F3] gives $\nu (\{r=0\})={\int }_{\{r=0\}}rd\pi =0$ and likewise $\nu (\{h=0\})=0$, so $r,h>0$ $\nu$-almost everywhere. Since $D(\nu \mathrm{\parallel }\pi )<\mathrm{\infty }$ and the negative part of the log-density integral is finite by [F4], $\log r$ is $\nu$-integrable; $\log g$ is $\nu$-integrable by $\int \mathrm{\mid }{\mathrm{\ell }}_x\mathrm{\mid }d\nu <\mathrm{\infty }$. Using $r=hg$ from step 3.1, [F9], and $\log g=-w{\mathrm{\ell }}_x-\log Z$, we obtain $D(\nu \mathrm{\parallel }q)=\int \log hd\nu =D(\nu \mathrm{\parallel }\pi )+w\int {\mathrm{\ell }}_{x}d\nu +\log Z\mathrm{.}$ The right-hand side is finite, so $D(\nu \mathrm{\parallel }q)<\mathrm{\infty }$.

For $h=d\nu \mathrm{/}dq$, the density form of [F4] gives $D(\nu \mathrm{\parallel }q)=\int \log hd\nu =\int h\log hdq,$ with $0\log 0:=0$. The integral is finite by step 4.1.

Apply step 1.2 to $h$ and integrate under $q$. Since $\int hdq=\nu (\mathrm{\Theta })=1$ and $q(\mathrm{\Theta })=1$, step 5.1 yields $D(\nu \mathrm{\parallel }q)=\int h\log hdq\ge \int (h-1)dq=0.$ If equality holds, the nonnegative function $h\log h-h+1$ has integral zero, so [F13] makes it zero $q$-almost everywhere. Step 1.2 then gives $h=1$ $q$-almost everywhere, and the Radon–Nikodym formula [F3] implies $\nu =q$. Conversely, if $\nu =q$, its Radon–Nikodym density relative to itself is $h=1$ $q$-almost everywhere, so [F4] and [F9] give $D(q\mathrm{\parallel }q)=\int 1\log 1dq=0$. Thus equality holds exactly when $\nu =q$.

By step 4.1 and [F12], $J_w(\nu )=D(\nu \mathrm{\parallel }q)-\log Z\ge -\log Z\mathrm{.}$ Step 2.1 gives $J_w(q)=-\log Z$, and step 6.1 shows equality is possible only for $\nu =q$. Therefore $q$ is the unique minimizer and $J_w(\nu )-J_w(q)=D(\nu \mathrm{\parallel }q)$.