A parameterized neural ordinary differential equation

Let $d,q\ge 1$, let $I\subseteq \mathbb{R}$ and $\mathrm{\Theta }\subseteq {\mathbb{R}}^q$ be open, and let $v:I\times {\mathbb{R}}^d\times \mathrm{\Theta }\to {\mathbb{R}}^d$. For a parameter $\theta \in \mathrm{\Theta }$, write $v_{\theta }(t,z):=v(t,z,\theta )$. The parameterized ordinary differential equation on $I$ is the initial-value problem

$\dot{z}(t)=v_{\theta }(t,z(t)),z(s)=z_s,$

where $s\in I$ and $z_s\in {\mathbb{R}}^d$. A trajectory on an interval $J\subseteq I$ containing $s$ is a differentiable function $z:J\to {\mathbb{R}}^d$ satisfying both displayed conditions. Here $v_{\theta }$ is the state-velocity field.

It is a parameterized neural ordinary differential equation, or neural ODE, when $v_{\theta }(t,z)$ is specified by a neural network with input $(t,z)$ and $d$ outputs. Here a network is a fixed finite directed acyclic graph evaluated in an order in which every edge points to a later vertex: input vertices hold the coordinates of $(t,z)$, each other vertex applies a fixed scalar activation to an affine combination of earlier vertex values, and the output vertices use the identity activation. The weights and biases are specified functions of $\theta$ and may be shared or fixed. No particular depth, width, or hidden activation is prescribed; affine networks with no hidden vertices are allowed.

The general parameterized ODE remains defined for fields without such a network specification. This definition alone asserts neither solution existence beyond its stated interval nor numerical accuracy.

Chen et al., equation (2) and continuous normalizing flows

Papamakarios et al., §4.1