An optimized mean-field law can lose posterior correlation

Let $\rho =1\mathrm{/}2$, so $\beta =1-{\rho }^2=3\mathrm{/}4$, and write $\varphi (t)=(2\pi {)}^{-1\mathrm{/}2}{e}^{-t^2\mathrm{/}2}$. Countable additivity of the supplied area measure makes $A\mapsto {\lambda }_2(A\times (0,1])$ a Borel measure, with ${\lambda }_1((a,b])=b-a$ by rectangle volume. The explicit intervals $(-n,n]$ exhaust $\mathbb{R}$ and have supplied measure $2n$, so the factor measure is sigma-finite. Its product exists by [F7]. It and the supplied ${\lambda }_2$ agree on half-open boxes by the volume formula and on the finite-mass exhaustion $(-n,n{]}^2$; [F7] and [F8] identify their Borel measures. All displayed densities and polynomial moment integrands are continuous and hence Borel: inverse images of open sets are open, and inverse-image preservation of sigma-algebra operations gives the Borel assertion. Thus [F8] makes them product-measurable. The compact comparison and Gaussian integral statements in [F3] now apply to these supplied measures. Direct algebra gives $\gamma (x,y)=\varphi (x)g_{\rho x,\beta }(y)=\varphi (y)g_{\rho y,\beta }(x)$. Each inner Gaussian density has mass one by [F3], so Tonelli [F4] proves that $\gamma$ has mass one and both marginals $\varphi$. By [F3] and density integration [F6], the marginal means are zero and second moments are one. Since $\mathrm{\mid }xy\mathrm{\mid }\le (x^2+y^2)\mathrm{/}2$, the cross moment is absolutely integrable. Fubini [F5], [F6] and the mean of $g_{\rho x,\beta }$ in [F3] give $E[XY]=\int x\varphi (x)\rho xdx=\rho$. Thus the target has marginal variances $1$ and covariance $1\mathrm{/}2$. With likelihood one the evidence is $1$, so [F1] identifies $\gamma$ as the exact target.

For $m_j\in \mathbb{R}$ and $v_j>0$, [F3] supplies the normalized Lebesgue densities $g_{m_j,v_j}$ and their means $m_j$ and variances $v_j$. Tonelli [F4] gives total mass one for $q$. The logarithms of $\gamma$ and $q$ are quadratic; the finite second moments and $\mathrm{\mid }xy\mathrm{\mid }\le (x^2+y^2)\mathrm{/}2$ make their absolute values integrable under $q$. Both densities are everywhere positive, so $q$ is admissible by [F2]. Fubini [F5] and [F6] give $E_q[X^2]=m_1^2+v_1$, $E_q[Y^2]=m_2^2+v_2$, and $E_q[XY]=m_1{m}_2$.

One has $\log \gamma =-\log (2\pi \sqrt{\beta })-(x^2-xy+y^2)\mathrm{/}(2\beta )$ and $-E_{g_{m_j,v_j}}\log g_{m_j,v_j}=\frac{1}{2}\log (2\pi v_j)+\frac{1}{2}$, using the mean and variance in step 2.1. Substituting these expressions and the second moments from step 2.1 into [F2] yields $\mathrm{ELBO}(q)=C-\frac{m_1^2-m_1{m}_2+m_2^2+v_1+v_2}{2\beta }+\frac{1}{2}\log v_1+\frac{1}{2}\log v_2$, where $C$ is independent of $(m_1,m_2,v_1,v_2)$. The mean quadratic equals $(m_1-\rho m_2{)}^2+\beta m_2^2$, so its unique minimum and the ELBO-maximizing means are $m_1=m_2=0$.

For either variance the variable part is $r(v)=-v\mathrm{/}(2\beta )+\frac{1}{2}\log v$. The identity has derivative $1$ and constants have derivative $0$ by their difference quotients; [F9] then gives that $r$ is differentiable on $(0,\mathrm{\infty })$ with $r^{\mathrm{\prime }}(v)=(\beta -v)\mathrm{/}(2\beta v)$. If $0<v<\beta$, the mean value theorem [F10] gives $c\in (v,\beta )$ with $r(\beta )-r(v)=r^{\mathrm{\prime }}(c)(\beta -v)>0$. If $v>\beta$, it gives $c\in (\beta ,v)$ with $r(v)-r(\beta )=r^{\mathrm{\prime }}(c)(v-\beta )<0$. Thus $v=\beta =3\mathrm{/}4$ is the unique global maximum. The target variance is $1$ and covariance is $1\mathrm{/}2$ by step 1.1, whereas the optimizing independent factors have variance $3\mathrm{/}4$ and covariance $0$ by step 2.1. Thus the unique ELBO optimizer differs from the exact target.