Smooth nonpolynomial activations are dense on compact sets

Fix $k\ge 1$ and $w\in {\mathbb{R}}^d$. If a smooth function has $g^{(k)}\equiv 0$, induction on $k$ shows it is a polynomial of degree at most $k-1$: for $k=1$, [F4] makes it constant; for $k\ge 2$, [F4] makes $g^{(k-1)}$ constant $c$, and subtracting $c{x}^{k-1}\mathrm{/}(k-1)!$ reduces to the induction hypothesis. Since $g$ is not a polynomial, choose $b_k$ with $g^{(k)}(b_k)\mathrm{\ne }0$. For $\delta \mathrm{\ne }0$, repeated application of [F4] gives the finite-difference identity ${\mathrm{\Delta }}_{\delta }^{k}g(y)={\delta }^k{g}^{(k)}(\xi )$ for some $\xi$ between $y$ and $y+k\delta$, where ${\mathrm{\Delta }}_{\delta }g(y)=g(y+\delta )-g(y)$. Indeed, the case $k=1$ is [F4]; for the induction step apply [F4] to $x\mapsto {\mathrm{\Delta }}_{\delta }^{k-1}g(x)$ between $y$ and $y+\delta$, then apply the induction hypothesis to $g^{\mathrm{\prime }}$ at the resulting point. The binomial expansion of ${\mathrm{\Delta }}_{\delta }^{k}g(y)$ is ${\sum }_{j=0}^k(-1{)}^{k-j}\left(\genfrac{}{}{0px}{}{k}{j}\right)g(y+j\delta )$. For $h>0$, define $R_{k,h}(x)=\frac{1}{g^{(k)}(b_k)h^k}{\sum }_{j=0}^k(-1{)}^{k-j}\left(\genfrac{}{}{0px}{}{k}{j}\right)g(b_k+jhw\cdot x),$ a finite linear combination of units from [F1]. If $w\cdot x=0$ then $R_{k,h}(x)=0=(w\cdot x{)}^k$. Otherwise the identity with $\delta =h(w\cdot x)$ gives $R_{k,h}(x)=\frac{(w\cdot x{)}^k}{g^{(k)}(b_k)}{g}^{(k)}({\xi }_{x,h}),\mathrm{\mid }{\xi }_{x,h}-b_k\mathrm{\mid }\le kh\mathrm{\mid }w\cdot x\mathrm{\mid }\mathrm{.}$ By [F3], choose $M<\mathrm{\infty }$ with $\mathrm{\mid }w\cdot x\mathrm{\mid }\le M$ on $K$. Continuity of $g^{(k)}$ at $b_k$ makes $g^{(k)}({\xi }_{x,h})\to g^{(k)}(b_k)$ uniformly over $x\in K$ as $h\downarrow 0$, since $\mathrm{\mid }{\xi }_{x,h}-b_k\mathrm{\mid }\le khM$. Therefore $R_{k,h}\to (w\cdot x{)}^k$ uniformly on $K$.

Constants belong to the network family through its output bias. Step 1.1 puts every positive-degree polynomial ridge function in the uniform closure of the family; finite sums of such approximants are still finite networks. By [F2], that closure contains a dense subspace of $C(K,\mathbb{R})$.

Given $\epsilon >0$, first choose a polynomial ridge combination within $\epsilon \mathrm{/}2$ of $f$ by [F2], then approximate its finitely many ridge terms closely enough using step 1.1 that their total error is below $\epsilon \mathrm{/}2$. The resulting one-hidden-layer network is within $\epsilon$ of $f$.