Rough nonpolynomial activations are dense on compact sets

By [F2] and the hypothesis on $\sigma$, some real test function ${\phi }_{\ast }$ supported in $[-1,1]$ has a nonpolynomial convolution $g_{\ast }=\sigma \ast {\phi }_{\ast }$. Indeed, if every real test had polynomial convolution, then for every complex test $\phi ={\phi }_R+i{\phi }_I$ both real convolutions would be polynomials, making every complex convolution polynomial; [F2] would then imply that $\sigma$ is polynomial almost everywhere. By [F3], $g_{\ast }$ is smooth. The activation hypothesis has the content in [F6].

By [F4], choose a finite $g_{\ast }$-network $G(x)=c+{\sum }_{j=1}^m{a}_j{g}_{\ast }(w_j\cdot x+b_j)$ with ${\sup }_{x\in K}\mathrm{\mid }G(x)-f(x)\mathrm{\mid }<\epsilon \mathrm{/}2$. For each $j$ with $a_j\mathrm{\ne }0$ and $w_j\mathrm{\ne }0$, the set $w_j\cdot K+b_j$ is compact and lies in a compact interval $I_j$.

By [F5], for each such $j$ choose a finite sum of translates of $\sigma$ uniformly within $\epsilon \mathrm{/}(2(1+{\sum }_{j=1}^m\mathrm{\mid }{a}_j\mathrm{\mid }))$ of $g_{\ast }$ on $I_j$. Composing with $w_j\cdot x+b_j$ gives a finite sum of units $\sigma (w_j\cdot x+b_j-t)$; units with $w_j=0$ contribute only constants and are absorbed into the output bias. Summing the finitely many approximants yields a $\sigma$-network $N$ with ${\sup }_K\mathrm{\mid }N-G\mathrm{\mid }<\epsilon \mathrm{/}2$.

The triangle inequality gives ${\sup }_K\mathrm{\mid }N-f\mathrm{\mid }\le {\sup }_K\mathrm{\mid }N-G\mathrm{\mid }+{\sup }_K\mathrm{\mid }G-f\mathrm{\mid }<\epsilon$. Hence the $\sigma$-network family is uniformly dense on $K$.