Variational autoencoder model, encoder, and ELBO

Let $(\mathcal{X},{\mathcal{A}}_X,{\mu }_X)$ and $(\mathcal{Z},{\mathcal{A}}_Z,{\mu }_Z)$ be sigma-finite measure spaces; that is, each has a countable measurable cover by sets of finite measure. Let $\mathrm{\Theta }\subseteq {\mathbb{R}}^{d_{\theta }}$ and $\mathrm{\Phi }\subseteq {\mathbb{R}}^{d_{\varphi }}$ be parameter sets of finite dimension. A latent-variable model with parameter $\theta \in \mathrm{\Theta }$ consists of a measurable prior density $r_{\theta }(z)\ge 0$ on $\mathcal{Z}$ and a decoder density $d_{\theta }(x\mid z)\ge 0$ on $\mathcal{X}$ for each $z$, with ${\int }_{\mathcal{Z}}{r}_{\theta }(z)d{\mu }_Z(z)=1,{\int }_{\mathcal{X}}{d}_{\theta }(x\mid z)d{\mu }_X(x)=1.$ The decoder is jointly measurable in $(x,z)$, and the joint model density is $p_{\theta }(x,z)={\gamma }_{\theta }(z;x):=r_{\theta }(z)d_{\theta }(x\mid z)$ relative to ${\mu }_X\otimes {\mu }_Z$. The decoder is also called the generative model: after drawing $z$ from the prior, it assigns a conditional density to the observation $x$.

For a fixed observation $x$, the evidence density and posterior density are $Z_{\theta }(x):=p_{\theta }(x):={\int }_{\mathcal{Z}}{\gamma }_{\theta }(z;x)d{\mu }_Z(z),{\pi }_{\theta }(z\mid x):=\frac{{\gamma }_{\theta }(z;x)}{Z_{\theta }(x)}\mathrm{.}$ Use this posterior only when $0<Z_{\theta }(x)<\mathrm{\infty }$. A variational encoder is a jointly measurable family of functions $q_{\varphi }(z\mid x)\ge 0$ on $\mathcal{Z}$, parameterized by $\varphi \in \mathrm{\Phi }$ and normalized by ${\int }_{\mathcal{Z}}{q}_{\varphi }(z\mid x)d{\mu }_Z(z)=1$ for every $(\varphi ,x)$; the same parameter $\varphi$ is used for each observation. For an admissible triple $(\theta ,\varphi ,x)$ require $0<Z_{\theta }(x)<\mathrm{\infty }$ and $q_{\varphi }(z\mid x)=0{\mu }_Z\text{-almost everywhere on }\{z:{\pi }_{\theta }(z\mid x)=0\},{\int }_{\mathcal{Z}}{q}_{\varphi }(z\mid x)(\mathrm{\mid }\log {\gamma }_{\theta }(z;x)\mathrm{\mid }+\mathrm{\mid }\log q_{\varphi }(z\mid x)\mathrm{\mid })d{\mu }_Z(z)<\mathrm{\infty }\mathrm{.}$ Terms multiplied by $q_{\varphi }=0$ are defined to be zero. The evidence lower bound is ${\mathrm{ELBO}}_{\theta ,\varphi }(x):={\int }_{\mathcal{Z}}{q}_{\varphi }(z\mid x)(\log {\gamma }_{\theta }(z;x)-\log q_{\varphi }(z\mid x))d{\mu }_Z(z)\mathrm{.}$

A pathwise representation of the encoder at $x$ is a probability space $(\mathcal{E},\mathcal{F},\rho )$ whose law $\rho$ is independent of $\theta$ and $\varphi$, together with a measurable map $g_{\varphi }(ϵ,x)$ such that the law of $g_{\varphi }(ϵ,x)$ for $ϵ\sim \rho$ is the measure with density $q_{\varphi }(\cdot \mid x)$ relative to ${\mu }_Z$. This representation defines how to generate encoder samples; it does not by itself assert that differentiation may pass through an expectation.

The generic admissibility conditions and ELBO convention.

The posterior-kernel formula on positive finite evidence fibres.

Kingma and Welling, Auto-Encoding Variational Bayes, arXiv v1.

Blei, Kucukelbir, and McAuliffe, Variational Inference: A Review for Statisticians.

Zheng, Hu, and Deng, 10-708 Lecture 13: Variational Inference II.