Finite-dimensional universal approximation by smooth nonpolynomial activations

Fix an integer $m\ge 1$ and $a\in {\mathbb{R}}^n$. Since $\sigma$ is not a polynomial, ${\sigma }^{(m)}$ is not identically zero; choose $b_m$ with ${\sigma }^{(m)}(b_m)\mathrm{\ne }0$. For $h\mathrm{\ne }0$ define $R_{m,h}(x):=\frac{1}{{\sigma }^{(m)}(b_m)h^m}{\sum }_{j=0}^m(-1{)}^{m-j}\left(\genfrac{}{}{0px}{}{m}{j}\right)\sigma (b_m+jh(a\cdot x))$. This is a finite linear combination of permitted units. The repeated-integral formula for the $m$th forward difference gives $R_{m,h}(x)=\frac{(a\cdot x{)}^m}{{\sigma }^{(m)}(b_m)}{\int }_{[0,1{]}^m}{\sigma }^{(m)}(b_m+h(a\cdot x)(t_1+\cdots +t_m))dt$. Because $a\cdot x$ is bounded on compact $K$ and ${\sigma }^{(m)}$ is uniformly continuous on a compact interval around $b_m$, $R_{m,h}\to (a\cdot x{)}^m$ uniformly on $K$ as $h\to 0$.

For $m\ge 1$ and arbitrary real $x_1,\dots ,x_m$, the polarization identity $x_1\cdots x_m=\frac{1}{2^{m}m!}{\sum }_{ϵ\in \{-1,1{\}}^m}({\prod }_{j=1}^m{ϵ}_j)({\sum }_{j=1}^m{ϵ}_j{x}_j{)}^m$ expresses each monomial of total degree $m$ in $n$ variables as a finite linear combination of powers of linear forms. Step 1.1 places every such power in the uniform closure of the network family, and that closure is a vector space because finite network sums are represented by concatenating their hidden units. Constants are present through $c_0$. Hence every real polynomial restricted to $K$ belongs to that closure.

The restrictions of real polynomials to $K$ form a unital real function algebra and separate distinct points, since a coordinate function distinguishes any two different points. Compact subsets of Euclidean space are compact Hausdorff, so [L1] makes these polynomial restrictions uniformly dense in $C(K,\mathbb{R})$. The network closure contains them by step 2.1 and is therefore all of $C(K,\mathbb{R})$.