Activation classes for neural-operator universality

Fix an input dimension $q\ge 1$ and an integer $m\ge 0$. Let ${\mathcal{A}}_m^{(q)}$ be the class of continuous scalar activations $\sigma :\mathbb{R}\to \mathbb{R}$ for which there is one depth $n\in \mathbb{N}$ such that, for every compact $K\subset {\mathbb{R}}^q$, the scalar-output depth-$n$ multilayer perceptrons with activation $\sigma$ are dense in $C^m(K)$ in the $C^m$ norm. For $m=0$, $C^0(K)$ has the supremum norm. For $m>0$, $C^m(K)$ means jets of functions defined and $C^m$ on an open neighborhood of $K$: identify two functions when all their derivatives of orders at most $m$ agree on $K$, and use the norm ${\max }_{\mathrm{\mid }\beta \mathrm{\mid }\le m}{\sup }_{x\in K}\mathrm{\mid }{D}^{\beta }f(x)\mathrm{\mid }$. Only networks that are $C^m$ near $K$ enter this density assertion. For vector outputs, stack scalar-output networks of that same depth. The source notation ${\mathcal{A}}_m$ suppresses the input dimension fixed by the finite-dimensional network under consideration. Let ${\mathcal{A}}_m^{L,(q)}$ be the subclass whose members are Borel measurable and have at most linear growth, meaning ${\sup }_{s\in \mathbb{R}}\frac{\mathrm{\mid }\sigma (s)\mathrm{\mid }}{1+\mathrm{\mid }s\mathrm{\mid }}<\mathrm{\infty }\mathrm{.}$ For an integer $q\ge 1$, define ${\mathrm{B}\mathrm{A}}_q$ to consist of those $\sigma \in {\mathcal{A}}_0^{(q)}$ for which, for every compact $K\subset {\mathbb{R}}^q$, every $\epsilon >0$, and every $C\ge {\mathrm{diam}}_2(K)$, there is a finite multilayer perceptron $f:{\mathbb{R}}^q\to {\mathbb{R}}^q$ with $\mathrm{\mid }f(x)-x{\mathrm{\mid }}_2\le \epsilon (x\in K),\mathrm{\mid }f(x){\mathrm{\mid }}_2\le C(x\in {\mathbb{R}}^q)\mathrm{.}$ Here ${\mathrm{diam}}_2(K)={\sup }_{x,y\in K}\mathrm{\mid }x-y{\mathrm{\mid }}_2$, with diameter and all displayed approximation seminorms equal to zero on the empty set. The diameter-based global bound is part of this definition and is retained verbatim; whether it has any members is checked separately.

feedforward neural network

Kovachki et al., Definitions 5–7: activation classes ${\mathcal{A}}_m$, ${\mathcal{A}}_m^L$, and BA