Variational family and evidence lower bound

Let $(\mathrm{\Omega },\mathcal{A},\mu )$ be a measure space. Let $\gamma :\mathrm{\Omega }\to [0,\mathrm{\infty }]$ be measurable and suppose $Z:={\int }_{\mathrm{\Omega }}\gamma d\mu \in (0,\mathrm{\infty })\mathrm{.}$ Define the normalized target density $\pi :=\gamma \mathrm{/}Z$. Then $\int \pi d\mu =1$. For a measurable probability density $q\ge 0$ (that is, ${\int }_{\mathrm{\Omega }}qd\mu =1$), write $q{\ll }_{\mu }\pi$ to mean $q=0$ $\mu$-almost everywhere on $\{\pi =0\}$. This is a support condition on the two densities. It ensures that the induced probability measures $Q:=q\mu$ and $\mathrm{\Pi }:=\pi \mu$ have $Q\ll \mathrm{\Pi }$.

The normalization follows from positive-scalar homogeneity of the nonnegative integral: $\int \pi d\mu =Z^{-1}\int \gamma d\mu =1.$ The induced density set functions are measures by The indefinite integral of a nonnegative measurable function is a measure. Their total masses are one: $Q(\mathrm{\Omega })=1$ by the probability-density hypothesis on $q$, and $\mathrm{\Pi }(\mathrm{\Omega })=1$ by the preceding calculation. To verify the absolute-continuity claim, let $A\in \mathcal{A}$ satisfy $\mathrm{\Pi }(A)=0$. Then ${\int }_A\pi d\mu =0$, so the zero-integral criterion gives $\pi =0$ $\mu$-almost everywhere on $A$. The support condition then gives $q=0$ $\mu$-almost everywhere on $A$, and hence $Q(A)=0$. The homogeneity and zero-integral criteria used here are Monotonicity and nonnegative homogeneity of the nonnegative integral and A nonnegative measurable function has integral $0$ exactly when it vanishes almost everywhere.

Call $q$ admissible when $q{\ll }_{\mu }\pi$ and ${\int }_{\mathrm{\Omega }}q(\mathrm{\mid }\log \gamma \mathrm{\mid }+\mathrm{\mid }\log q\mathrm{\mid })d\mu <\mathrm{\infty }\mathrm{.}$ Use the extended logarithm $\log 0=-\mathrm{\infty }$, agreeing with The natural logarithm as the inverse of the exponential function on positive inputs. Since $\int \gamma d\mu <\mathrm{\infty }$ and $\int qd\mu =1$, both functions are finite $\mu$-almost everywhere: for either $f=\gamma$ or $f=q$, $\mu \{f=\mathrm{\infty }\}\le n^{-1}\int fd\mu$ for every $n\ge 1$. Interpret the weighted logarithmic terms on this measurable conull set and set them equal to zero on its measurable null complement, so the resulting functions remain measurable. In particular, set $q\mathrm{\mid }\log \gamma \mathrm{\mid }$, $q\mathrm{\mid }\log q\mathrm{\mid }$, $q\log \gamma$, and $q\log q$ equal to zero on $\{q=0\}$. On the set where $q>0$, admissibility implies $\gamma >0$ $\mu$-almost everywhere. Define the finite real number $\mathrm{ELBO}(q):={\int }_{\mathrm{\Omega }}q(\log \gamma -\log q)d\mu \mathrm{.}$ A variational family is a nonempty set of admissible densities. A maximizing member need not exist.

For a finite integer $m\ge 1$, let $\mathrm{\Omega }={\prod }_{j=1}^m{\mathrm{\Omega }}_j$, let each $({\mathrm{\Omega }}_j,{\mathcal{A}}_j,{\mu }_j)$ be sigma-finite, and let $\mu ={⨂}_{j=1}^m{\mu }_j$. A mean-field density is a product $q(z)={\prod }_{j=1}^m{q}_j(z_j)$ of probability densities $q_j$; it belongs to the admissible mean-field family only when the product satisfies the support and integrability conditions above. Tonelli gives $\int qd\mu =1$. The product form alone does not guarantee support or finite ELBO.

In a dominated Bayesian model, a posterior interpretation is available only with the model hypotheses and on fibers with finite positive evidence, as in Bayes formula for dominated kernels and Conditional density formula. In particular, Likelihood densities supplies separate likelihood versions but does not assert joint measurability in parameter and data. Supply the single jointly measurable density $\gamma$ explicitly before using Tonelli or a posterior-kernel formula. No quotient is formed at zero or infinite normalizer.