Neural ODEs and continuous normalizing flows — Examples

Given: $T=1$, $u(0)=(1,1)$, $v_{\theta }(t,u)=(-u_1,\theta u_2)$, terminal loss $u_2(1)$, and the $N$-step forward Euler graph for $N\in \mathbb{N}$ with $N\ge 1$.

A parameterized neural ordinary differential equation: a parameterized ODE is the initial-value problem $\dot{u}=v_{\theta }(t,u)$ with specified initial state; the neural specialization additionally specifies a finite acyclic network for the field. Here two identity-activation output nodes with input weights $(-1,0)$ and $(0,\theta )$, zero biases, and unused time input realize $v_{\theta }(t,u)=(-u_1,\theta u_2)$.

A globally Lipschitz neural ODE has a global smooth flow: the stated smooth uniformly globally Lipschitz field has a unique smooth flow on the full finite interval.

Exact continuous adjoint sensitivity for a neural ODE: the continuous procedure solves the terminal adjoint equation and returns its initial, terminal, running, and vector-field parameter contributions.

The continuous ODE adjoint gives the parameter gradient with endpoint terms: under common-tube and C1 objective hypotheses, the continuous adjoint expression is the exact derivative of the continuous objective.

Finite parameterized state-step graphs and objectives: C1 initial and step maps together with a C1 terminal/running objective define one finite real-arithmetic graph and its reduced objective.

The reverse recursion is the exact gradient of a finite state-step objective: the backward recursion returns the exact gradient of that finite C1 graph.

The real exponential function and the number $e$ by a power series: $e=\exp (1)$ and $\exp$ is the real exponential function.

The exponential function is smooth and $(\exp {)}^{\mathrm{\prime }}=\exp$: $\exp$ is smooth and $(\exp {)}^{\mathrm{\prime }}=\exp$.

The exponential is positive and satisfies $\exp (-x)=1\mathrm{/}\exp (x)$: $\exp (x)>0$ for every real $x$.

Sum, product, quotient, and chain rules from difference quotients: the sum, product, chain, and positive-integer power rules apply to the differentiable functions used below.

The exponential addition formula $\exp (x+y)=\exp (x)\exp (y)$: $\exp (x+y)=\exp (x)\exp (y)$ for real $x,y$.

Take $I=\mathrm{\Theta }=U_0=\mathbb{R}$ and $\mathrm{\Omega }=(-101,-99)$, and set $u_0(\eta )=(1,1)$. The field $v(t,u,\eta )=(-u_1,\eta u_2)$ is jointly $C^{\mathrm{\infty }}$. For every nonempty compact $K\subseteq \mathbb{R}$, $L_K=\max \{1,{\sup }_{\eta \in K}\mathrm{\mid }\eta \mathrm{\mid }\}$ and $M_K=0$ give the required state-Lipschitz and origin bounds; the empty compact case is vacuous. By [F7, F8], the exact solution $u_{\eta }(t)=(\exp (-t),\exp (\eta t))$ is continuous on the compact set $[0,1]\times \bar{\mathrm{\Omega }}$, so its trajectory tube is compact. For the continuous adjoint, take $\varphi (u,\eta )=u_2$ on the open set $U_{\varphi }={\mathbb{R}}^2\times \mathbb{R}$ and $\mathrm{\ell }(t,u,\eta )=0$ on the open set $U_{\mathrm{\ell }}=\mathbb{R}\times {\mathbb{R}}^2\times \mathbb{R}$. These objectives are $C^1$, and their first derivatives are constant, hence bounded and uniformly continuous on the terminal slice and common trajectory tube. Thus the continuous and finite adjoint supplier hypotheses hold.