Variational inference and the evidence lower bound

Define $Q(A)={\int }_{A}qd\mu$ and $\mathrm{\Pi }(A)={\int }_A\pi d\mu$. These are probability measures by [F1] and [F7]. If $\mathrm{\Pi }(A)=0$, [F4] applied to $\pi {\mathbf{1}}_A$ gives $\pi =0$ $\mu$-almost everywhere on $A$. Then [F1] gives $q=0$ almost everywhere on $A$, so $Q(A)=0$ and $Q\ll \mathrm{\Pi }$. Define $r=q\mathrm{/}\pi$ on $\{\pi >0\}$ and $r=0$ on $\{\pi =0\}$. For every measurable $A$, ${\int }_{A}rd\mathrm{\Pi }={\int }_{A\cap \{\pi >0\}}qd\mu =Q(A)$, so $r=dQ\mathrm{/}d\mathrm{\Pi }$.

On $\{q>0\}$, $\pi =\gamma \mathrm{/}Z>0$ and $\log r=\log q-\log \gamma +\log Z$. The assumed absolute integrability and $\int qd\mu =1$ make all terms integrable. Hence $\int \log rdQ=\int q\log (q\mathrm{/}\pi )d\mu =\log Z-\mathrm{ELBO}(q)$. This algebraic identity uses the explicit density ratio and no choice axiom.

By [A1] and [F5], the integral $\int \log rdQ$ is $D(Q\mathrm{\parallel }\mathrm{\Pi })$ under the library convention. Rearranging step 2.1 gives $\log Z=\mathrm{ELBO}(q)+D(Q\mathrm{\parallel }\mathrm{\Pi })$.

The measures $Q$ and $\mathrm{\Pi }$ are probabilities, so [F6] gives $D(Q\mathrm{\parallel }\mathrm{\Pi })\ge 0$. Step 3.1 then yields $\mathrm{ELBO}(q)\le \log Z$.

Let $s=\sqrt{\beta }>0$ and $\varphi (t)=(2\pi {)}^{-1\mathrm{/}2}{e}^{-t^2\mathrm{/}2}$. Direct algebra gives ${\gamma }_{\rho }(x,y)=\varphi (x)g_{\rho x,\beta }(y)=\varphi (y)g_{\rho y,\beta }(x)$. The target is nonnegative and measurable, and [F8] supplies the sigma-finite Lebesgue product spaces; Tonelli [F6] and [F3] show that it integrates to one and has both marginals $\varphi$. By [F5], both marginal means are zero and both marginal variances are one. Since $\mathrm{\mid }xy\mathrm{\mid }\le (x^2+y^2)\mathrm{/}2$, the cross moment is absolutely integrable. Fubini [F6], density integration [F7], and [F5] give $E[XY]=\int x\varphi (x)(\rho x)dx=\rho$, because the conditional density $g_{\rho x,\beta }$ has mean $\rho x$. Thus the covariance is $\left(\begin{array}{cc}1& \rho \\ \rho & 1\end{array}\right)$.

For $m_j\in \mathbb{R}$ and $v_j>0$, [F4] makes $g_{m_j,v_j}$ probability densities with means $m_j$ and variances $v_j$. Tonelli [F6] gives $\int q=1$. The product density $q$ and target ${\gamma }_{\rho }$ are positive everywhere, so the support condition in [F1] holds. Their logarithms are quadratic; the second moments and $\mathrm{\mid }xy\mathrm{\mid }\le (x^2+y^2)\mathrm{/}2$ make the absolute logarithms integrable under $q$, so $q$ is admissible. Fubini [F6], density integration [F7], and [F5] give $E_q[X^2]=m_1^2+v_1$, $E_q[Y^2]=m_2^2+v_2$, and $E_q[XY]=m_1{m}_2$.

From the displayed density, $\log {\gamma }_{\rho }=-\log (2\pi s)-(x^2-2\rho xy+y^2)\mathrm{/}(2\beta )$. Also $-E_{g_{m_j,v_j}}\log g_{m_j,v_j}=\frac{1}{2}\log (2\pi v_j)+\frac{1}{2}$, by the moments in step 2.1. Substituting these expectations into [F2] gives $\mathrm{ELBO}(q)=C_{\rho }-\frac{m_1^2-2\rho m_1{m}_2+m_2^2+v_1+v_2}{2\beta }+\frac{1}{2}\log v_1+\frac{1}{2}\log v_2$, where $C_{\rho }$ is independent of $m_1,m_2,v_1,v_2$. The mean quadratic equals $(m_1-\rho m_2{)}^2+\beta m_2^2$, so its unique minimizing, hence ELBO-maximizing, means are $m_1=m_2=0$.

For each variance, the part depending on $v$ is $g(v)=-v\mathrm{/}(2\beta )+(1\mathrm{/}2)\log v$. Its derivative is $g^{\mathrm{\prime }}(v)=(\beta -v)\mathrm{/}(2\beta v)$, positive for $0<v<\beta$ and negative for $v>\beta$. Thus the unique maximizer is $v=\beta$. Since $0<\mathrm{\mid }\rho \mathrm{\mid }<1$, $0<\beta <1$, proving strict variance underestimation.

Put $\alpha =q_j{\mu }_j$ and $\nu =v{\mu }_{-j}$. They are probabilities and the law with density $q$ is their product by [F1]–[F3]. A density is positive and finite almost surely under its own probability, since its zero set has mass zero and its infinite set has reference measure zero. Thus $A(x)=\log q_j(x)$ and $B(y)=\log v(y)$ are finite almost surely, and [F8] gives $\log q=A+B$ almost surely. Choose a finite integer $K$ for which $p:=\nu (\mathrm{\mid }B\mathrm{\mid }\le K)>0$; such a $K$ exists because those events increase to a full-measure set. On that event $\mathrm{\mid }A\mathrm{\mid }\le \mathrm{\mid }A+B\mathrm{\mid }+K$, so product integration [F2]–[F3] yields $p\int \mathrm{\mid }A\mathrm{\mid }d\alpha \le \int \mathrm{\mid }A+B\mathrm{\mid }d(\alpha \otimes \nu )+Kp<\mathrm{\infty }$. Interchanging $A$ and $B$ and choosing an analogous positive-probability bounded event proves $\int \mathrm{\mid }B\mathrm{\mid }d\nu <\mathrm{\infty }$. Hence $H=\int Bd\nu$ is finite, including $H=0$ for the unit complementary factor.

By [F2] and $Z<\mathrm{\infty }$, the marginal $G(x):=\int \gamma (x,y)d{\mu }_{-j}(y)$ is finite for ${\mu }_j$-almost every $x$. Fix such an $x$. Under $\nu$, define $g(y)=\gamma (x,y)\mathrm{/}v(y)$ on $\{0<v<\mathrm{\infty }\}$; arbitrary values on its $\nu$-null complement do not matter. Then $\int gd\nu ={\int }_{\{0<v<\mathrm{\infty }\}}\gamma (x,y)d{\mu }_{-j}(y)\le G(x)$ by [F3]. In particular $g$ is finite almost surely. Write $L=\log g$, allowing $L=-\mathrm{\infty }$ at $g=0$. Its positive part is integrable since ${\log }^{+}g\le g$, and the defined expected-log convention and integrability of $B$ give $\int Ld\nu =h_j(x)-H$ in the extended sense; in particular $h_j(x)$ cannot equal $+\mathrm{\infty }$.

Let $L_n=\max (L,-n)$. Its negative part is bounded and its positive part is integrable by step 2.1, so $L_n$ is integrable. Also $e^{L_n}=\max (g,e^{-n})\le g+1$ is integrable. Apply [F4] with the convex exponential [F5] to obtain $\exp (\int L_{n}d\nu )\le \int e^{L_n}d\nu$. The negative parts increase to $L^{-}$, so [F6] gives $\int L_{n}d\nu \downarrow \int Ld\nu$, possibly $-\mathrm{\infty }$. By [F7], the right side tends to $\int gd\nu$, with domination $g+1$; continuity of the exponential and $e^{-\mathrm{\infty }}=0$ therefore give $\exp (h_j(x)-H)\le \int gd\nu \le G(x)$.

The inequality in step 3.1 holds for ${\mu }_j$-almost every $x$. Multiply it by $e^H$ and integrate using [F2] to obtain $c_j\le e^H\int Gd{\mu }_j=e^{H}Z<\mathrm{\infty }$. Since $h_j$ is finite $\alpha$-almost surely, $e^{h_j}>0$ on an $\alpha$-full measurable set. If $c_j=0$, [F9] would make this set ${\mu }_j$-null, hence $\alpha$-null by [F3], contradicting $\alpha ({\mathrm{\Omega }}_j)=1$. Thus $c_j>0$. This does not prove admissibility of the normalized replacement, which remains a separate requirement.

Fix any admissible replacement $r_j$ and form $u=r_j{q}_{-j}$. Tonelli [F5] gives $\int ud\mu =1$, and the product rectangle formula [F4] gives ${\mathbb{P}}_u:=u\mu ={⨂}_{k=1}^m(u_k{\mu }_k)$ is the product probability. Each coordinate density $u_k$ is positive and finite almost surely under its factor law; put $X_k=\log u_k(Z_k)$ and $S=\log u(Z)={\sum }_{k=1}^m{X}_k$. By [F1], ${\mathbb{E}}_u\mathrm{\mid }S\mathrm{\mid }=\int u\mathrm{\mid }\log u\mathrm{\mid }d\mu <\mathrm{\infty }$. Fix $j$ and put $R={\sum }_{k\mathrm{\ne }j}{X}_k$, which is finite almost surely under the product of the other factor laws. Since $\{\mathrm{\mid }R\mathrm{\mid }\le n\}$ increases to a full-measure set, choose an integer $K$ for which $p:={\mathbb{P}}_u(\mathrm{\mid }R\mathrm{\mid }\le K)>0$. Product integration from [F5] and [F6] then gives $p\int u_j\mathrm{\mid }\log u_j\mathrm{\mid }d{\mu }_j={\int }_{\{\mathrm{\mid }R\mathrm{\mid }\le K\}}\mathrm{\mid }{X}_j\mathrm{\mid }d{\mathbb{P}}_u\le \int \mathrm{\mid }S\mathrm{\mid }d{\mathbb{P}}_u+Kp<\mathrm{\infty }$, using $\mathrm{\mid }{X}_j\mathrm{\mid }\le \mathrm{\mid }S\mathrm{\mid }+\mathrm{\mid }R\mathrm{\mid }$ on that event. Thus every factor log is integrable; the same argument applies to the current factor and to $q_j^{\ast }$ because their products are admissible.

For each admissible replacement $u_j$, [F1] makes $u\log \gamma$ absolutely integrable. Fubini [F7] applies also to $u\mathrm{\mid }\log \gamma \mathrm{\mid }$, so $\int u_j\mathrm{\mid }{h}_j\mathrm{\mid }d{\mu }_j<\mathrm{\infty }$ and exchanging the $j$ and $-j$ integrals gives $\int u\log \gamma d\mu =\int u_j{h}_{j}d{\mu }_j$. By step 1.1, all factor log terms are integrable, so $\log u=\log u_j+\log q_{-j}$ almost surely and $\mathrm{ELBO}(u_j{q}_{-j})=\int u_j{h}_{j}d{\mu }_j-\int u_j\log u_{j}d{\mu }_j-\int q_{-j}\log q_{-j}d{\mu }_{-j}$. The final term is constant over coordinate replacements.

Write $q_j^{\ast }=e^{h_j}\mathrm{/}{c}_j$. If $q_j^{\ast }=0$, then $h_j=-\mathrm{\infty }$ almost everywhere on that set. For any admissible $r_j$, step 2.1 makes $\int r_j\mathrm{\mid }{h}_j\mathrm{\mid }d{\mu }_j<\mathrm{\infty }$, hence $r_j=0$ almost everywhere there. Thus $r_j{\mu }_j\ll q_j^{\ast }{\mu }_j$. On $\{q_j^{\ast }>0\}$ the ratio $r_j\mathrm{/}{q}_j^{\ast }$ is measurable, and for every measurable $A$, ${\int }_A(r_j\mathrm{/}{q}_j^{\ast })d(q_j^{\ast }{\mu }_j)={\int }_{A\cap \{q_j^{\ast }>0\}}{r}_{j}d{\mu }_j={\int }_A{r}_{j}d{\mu }_j$, so it is the Radon–Nikodym density.

By [A1] and [F8], $D(r_j{\mu }_j\mathrm{\parallel }{q}_j^{\ast }{\mu }_j)=\int r_j\log (r_j\mathrm{/}{q}_j^{\ast })d{\mu }_j=\int r_j\log r_{j}d{\mu }_j-\int r_j{h}_{j}d{\mu }_j+\log c_j$. All terms are finite by step 1.1 and step 2.1. Also $\int q_j^{\ast }{h}_j-\int q_j^{\ast }\log q_j^{\ast }=\log c_j$. Subtracting the ELBO expressions in step 2.1 therefore gives $\mathrm{ELBO}(q_j^{\ast }{q}_{-j})-\mathrm{ELBO}(r_j{q}_{-j})=D(r_j{\mu }_j\mathrm{\parallel }{q}_j^{\ast }{\mu }_j)\ge 0$ by [F9]. Hence $q_j^{\ast }$ is an attained exact coordinate maximizer.

The current factor $q_j$ is itself an admissible replacement. Taking $r_j=q_j$ in step 3.2 proves that the update does not decrease ELBO.

By [F3], the procedure has at most $N$ sweeps and at most $m$ coordinate-update blocks in each sweep. An unavailable result returns immediately, so every run is either a prefix ending at that branch or a run through all $N$ sweeps. Since $N$ and $m$ are finite, under [A2] it returns after at most $Nm$ calls. The call-count complexity is therefore $O(Nm)$; no cost for evaluating an individual call is asserted.

Fix any $c\in \mathbb{R}$. By the onto clause of [F5], choose $a>0$ with $\log a=c$. Define ${\gamma }_c=a\gamma$. Homogeneity [F4] gives $Z_c=\int {\gamma }_{c}d\mu =aZ$, so ${\pi }_c={\gamma }_c\mathrm{/}{Z}_c=\pi$ and both $Q$ and $\mathrm{\Pi }$ remain unchanged. Since ${\pi }_c=\pi$, the support condition in [F1] is unchanged. On the q-support, $\mathrm{\mid }\log {\gamma }_c\mathrm{\mid }\le \mathrm{\mid }\log \gamma \mathrm{\mid }+\mathrm{\mid }c\mathrm{\mid }$, so the logarithmic integrability condition remains finite.

On the q-support the logarithm product law [F5] gives $\log {\gamma }_c=\log \gamma +c$. Using [F2], ${\mathrm{ELBO}}_c(q)=\int q(\log \gamma +c-\log q)d\mu =\mathrm{ELBO}(q)+c$. Since $c$ is arbitrary, the absolute ELBO can be made arbitrarily large while $Q$, $\mathrm{\Pi }$, and any fixed discrepancy between them stay unchanged.

Under [A1], the decomposition [F3] gives $\log Z-\mathrm{ELBO}(q)=D(Q\mathrm{\parallel }\mathrm{\Pi })$. Without control of $\log Z$, a separate calibration theorem, or another posterior-discrepancy bound, the absolute ELBO alone supplies no calibration guarantee.