Exact-density normalizing flows

Let $d\ge 1$, let $U,V\subseteq {\mathbb{R}}^d$ be nonempty open sets, and let $T:U\to V$ be a bijection such that both $T$ and $T^{-1}$ are continuously differentiable. Here $C^1$ means totally differentiable at every point with continuous derivative matrix. Such a map is a global $C^1$ diffeomorphism. A finite composition $T=T_K\circ \cdots \circ T_1$ of global $C^1$ diffeomorphisms between open subsets of ${\mathbb{R}}^d$ is a normalizing flow.

A base density on $U$ is a Lebesgue-measurable function $p_U:U\to [0,\mathrm{\infty }]$ with ${\int }_U{p}_{U}d{\lambda }_d=1$. It defines the probability measure ${\mu }_U(A)={\int }_A{p}_{U}d{\lambda }_d$ for Lebesgue-measurable $A\subseteq U$.

For the model-measure construction, assume the Axiom of Countable Choice. The model measure on $V$ is ${\mu }_X(B)={\mu }_U(T^{-1}(B))$ for Borel $B\subseteq V$. This value is defined because the inverse $C^1$ diffeomorphism maps Borel sets to Lebesgue-measurable sets under Countable Choice. When this measure has a Lebesgue density, that density is the model density.

Continuously differentiable maps, local inverses, and local diffeomorphisms — Total differentiability, continuous derivative matrices, and local diffeomorphisms.

The measure with density $f$ relative to $\mu$ — The measure defined by integrating a nonnegative measurable density.

The Axiom of Countable Choice (${\mathrm{A}\mathrm{C}}_{\omega }$) — Countable Choice is assumed for the model-measure construction.

Assuming countable choice, every Borel subset of ${\mathbb{R}}^n$ is Lebesgue measurable — Under Countable Choice, Borel subsets of Euclidean space are Lebesgue measurable.

A C^1 diffeomorphism maps Lebesgue measurable sets to Lebesgue measurable sets — A $C^1$ diffeomorphism sends Lebesgue-measurable sets to Lebesgue-measurable sets.

Papamakarios et al., §2.1, PDF pp. 2–3

Sheldon, COMPSCI 688 Lecture 23, PDF pp. 1–5

Assume the Axiom of Countable Choice. Let $d\ge 1$, let $U,V\subseteq {\mathbb{R}}^d$ be nonempty open sets, and let $T:U\to V$ be a bijection such that $T$ and $T^{-1}$ are both $C^1$. Let $p_U:U\to [0,\mathrm{\infty }]$ be Lebesgue measurable with ${\int }_U{p}_{U}d{\lambda }_d=1$, and define ${\mu }_U(A)={\int }_A{p}_{U}d{\lambda }_d$ for Lebesgue-measurable $A\subseteq U$ and ${\mu }_X(B)={\mu }_U(T^{-1}(B))$ for Borel $B\subseteq V$. Then

$q(x)=p_U(T^{-1}(x))\mathrm{\mid }\det D(T^{-1})(x)\mathrm{\mid }$

is Lebesgue measurable and satisfies ${\mu }_X(B)={\int }_{B}qd{\lambda }_d$ for every Borel $B\subseteq V$. Thus $q$ is a version of the density of the transformed measure.

Let $K\ge 1$, let $U_0,\dots ,U_K$ be nonempty open subsets of ${\mathbb{R}}^d$, and let $T_j:U_{j-1}\to U_j$ be a $C^1$ diffeomorphism for $j=1,\dots ,K$. For $T=T_K\circ \cdots \circ T_1$, $u\in U_0$, and $z_0=u$, $z_j=T_j(z_{j-1})$, the log-Jacobian terms add:

$\log \mathrm{\mid }\det DT(u)\mathrm{\mid }={\sum }_{j=1}^K\log \mathrm{\mid }\det D{T}_j(z_{j-1})\mathrm{\mid }\mathrm{.}$

Equivalently, for $z_K=x\in U_K$ and $z_{j-1}=T_j^{-1}(z_j)$,

$\log \mathrm{\mid }\det D{T}^{-1}(x)\mathrm{\mid }={\sum }_{j=1}^K\log \mathrm{\mid }\det D{T}_j^{-1}(z_j)\mathrm{\mid }\mathrm{.}$

The Axiom of Countable Choice (${\mathrm{A}\mathrm{C}}_{\omega }$) — Countable Choice is the axiom used by the measurable change-of-variables supplier.

Papamakarios et al., §2.1, PDF pp. 2–4; §2.2, PDF pp. 4–5

Sheldon, COMPSCI 688 Lecture 23, PDF pp. 2–5

Assume the Axiom of Countable Choice. Let $d\ge 1$ and let $p:{\mathbb{R}}^d\to (0,\mathrm{\infty })$ be a Lebesgue probability density. For $i=1,\dots ,d$, write $x_{<i}=(x_1,\dots ,x_{i-1})$, with $x_{<1}$ the unique element of ${\mathbb{R}}^0$. For each $i$, let $q_i:\mathbb{R}\times {\mathbb{R}}^{i-1}\to (0,\mathrm{\infty })$ be a positive continuous function, written $q_i(t\mid x_{<i})$, and suppose $p(x)={\prod }_{i=1}^d{q}_i(x_i\mid x_{<i}),{\int }_{\mathbb{R}}{q}_i(t\mid x_{<i})dt=1$ for every $x\in {\mathbb{R}}^d$ and every preceding-coordinate vector $x_{<i}$. Define the conditional cumulative distribution functions $F_i(t\mid x_{<i})={\int }_{-\mathrm{\infty }}^t{q}_i(s\mid x_{<i})ds\mathrm{.}$ Assume each $F_i$ is jointly $C^1$ in $(t,x_{<i})$, satisfies ${\mathrm{\partial }}_t{F}_i(t\mid x_{<i})=q_i(t\mid x_{<i})$, and, for every fixed $x_{<i}$, the slice $t\mapsto F_i(t\mid x_{<i})$ is a strictly increasing bijection from $\mathbb{R}$ onto $(0,1)$. In each inverse below, $F_i^{-1}$ means the inverse in its first argument with the preceding coordinates held fixed.

Then there is a global $C^1$ diffeomorphism $T:(0,1{)}^d\to {\mathbb{R}}^d$ whose $i$th output coordinate depends only on the first $i$ input coordinates and which maps the uniform density $1$ on $(0,1{)}^d$ to the density $p$. Its inverse is $S_i(x)=F_i(x_i\mid x_{<i}),$ and $T=S^{-1}$ is determined recursively by $x_1=F_1^{-1}(u_1),x_i=F_i^{-1}(u_i\mid x_{<i})(i=2,\dots ,d)\mathrm{.}$ This is an exact representational existence result; it makes no claim about finite-network approximation, trainability, or numerical performance.

The Axiom of Countable Choice (${\mathrm{A}\mathrm{C}}_{\omega }$) — Countable Choice is used by the change-of-variables and box-volume suppliers.

Papamakarios et al., §2.2, PDF pp. 4–5; §3.1, PDF pp. 10–16

Sheldon, COMPSCI 688 Lecture 23, PDF pp. 4–5