Feedforward networks and single-hidden-layer universality — Examples

Since sine is odd, $S(0)=c$ and $(S(1)+S(-1))\mathrm{/}2=c$.

For the even target $f(x)=x^2$, uniform error at most $\epsilon$ implies both $\mathrm{\mid }c\mathrm{\mid }=\mathrm{\mid }S(0)-f(0)\mathrm{\mid }\le \epsilon$ and $\mathrm{\mid }c-1\mathrm{\mid }=\mid (S(1)+S(-1))\mathrm{/}2-1\mid \le \epsilon$.

Every real $c$ satisfies $\max (\mathrm{\mid }c\mathrm{\mid },\mathrm{\mid }c-1\mathrm{\mid })\ge 1\mathrm{/}2$. Thus no such sum approximates $x^2$ with error less than $1\mathrm{/}2$, disproving uniform density.

The spike set $\{1\mathrm{/}n:n\in \mathbb{N},n\ge 1\}$ is countable and null by [F3]. Thus $\sigma =H$ almost everywhere and $\sigma$ is locally essentially bounded. Its discontinuities are exactly $\{0\}\cup \{1\mathrm{/}n:n\in \mathbb{N},n\ge 1\}$, a closed null set: away from this set the step part is locally constant and there are no spikes nearby. Hence the activation satisfies [F4].

Consider a finite network $N(x)=c+{\sum }_j{a}_j\sigma (w_{j}x+b_j)$ on $[-1,1]$. Absorb constant affine units into $c$, combine terms with identical affine maps and discard groups with zero combined coefficient. Call a nonconstant affine map active when its zero lies in $[-1,1]$ and it takes positive values arbitrarily close to that zero at points of $[-1,1]$. If an active group remains, fix its zero $x_0$ and choose, among active groups at $x_0$ with nonzero coefficient, one with minimal $\mathrm{\mid }{w}_0\mathrm{\mid }$; write its combined coefficient as $A_0\mathrm{\ne }0$.

The function $\sigma$ is not polynomial almost everywhere. If a polynomial agreed with it almost everywhere, it would vanish almost everywhere on $(-\mathrm{\infty },0)$, so [F2] would force it to be the zero polynomial; this contradicts $\sigma =1$ almost everywhere on $(0,1)$.

Set $x_n=x_0+1\mathrm{/}(n{w}_0)$, choosing $n$ sufficiently large that $x_n\in [-1,1]$. The chosen unit has preactivation $1\mathrm{/}n$ and contributes $A_0(n^2+1)$. For a unit with a different zero, its preactivation tends to a nonzero value; if the limit is negative the unit is eventually negative, and if positive it lies near only finitely many reciprocals, each attained at most once by the nonconstant affine sequence. Thus it spikes at only finitely many $x_n$. At the same zero, an active group with opposite sign has negative preactivation on this sequence; one with the same sign and a distinct affine map has slope ratio $r=\mathrm{\mid }w\mathrm{\mid }\mathrm{/}\mathrm{\mid }{w}_0\mathrm{\mid }>1$. If $r$ is irrational the equation $r\mathrm{/}n=1\mathrm{/}m$ is impossible; if $r=p\mathrm{/}q>1$ in lowest terms it requires $p\mid n$. Choose the infinite sequence $n=1+kP$, where $P$ is the product of the finitely many such numerators $p$ (take $P=1$ if there are none); then none of those other groups spikes. Along a tail of this sequence all other spike terms vanish and all Heaviside contributions remain bounded, so $\mathrm{\mid }N(x_n)\mathrm{\mid }\to \mathrm{\infty }$. Thus a bounded network can have no active group with nonzero coefficient.

For a bounded network, every nonconstant group that remains has no positive threshold crossing in $[-1,1]$. Its Heaviside part is constant on $(-1,1)$, and its spike set in $[-1,1]$ is finite because its positive preactivation values stay bounded away from zero. Constant units are constant as well. Hence $N$ is constant outside a finite subset of $[-1,1]$.

If $N$ is unbounded, it cannot approximate the bounded target $f(x)=x$ uniformly. If it is bounded, choose $x_{-},x_{+}$ outside the finite exceptional set with $x_{+}-x_{-}>1$; then $N(x_{-})=N(x_{+})$, so at least one of $\mathrm{\mid }N(x_{-})-x_{-}\mathrm{\mid }$ and $\mathrm{\mid }N(x_{+})-x_{+}\mathrm{\mid }$ exceeds $1\mathrm{/}2$. Thus no network approximates $x$ on $[-1,1]$ with error below $1\mathrm{/}2$, refuting the pointwise-uniform claim.

The constant activation $\sigma (t)=1$ is a polynomial, and the zero measure is finite, compactly supported, and absolutely continuous with respect to Lebesgue measure by [F3].

Since every measurable function has $L^p(0)$ seminorm zero, the quotient space $L^p(0)$ contains only its zero class by [F1].

The network family contains the zero function by [F2], whose class is the only element of $L^p(0)$. Thus it is dense despite the polynomial activation, refuting the unrestricted “only if” claim.

By [F2], each unit on a univariate input is $(wt+b{)}^q$, a polynomial of degree at most $q$; finite weighted sums and an output bias preserve that degree bound.

Apply [F1] with the polynomial $p(t)=t^q$ to obtain the stated positive lower bound for approximation of $t^{q+1}$.

ReLU and logistic sigmoid are continuous, hence their discontinuity sets are empty; ReLU is bounded on each compact interval and sigmoid is globally bounded between zero and one. Thus both lie in the pointwise locally bounded activation class.

Neither is a polynomial almost everywhere. If a continuous function equals a polynomial almost everywhere on $\mathbb{R}$, their difference is continuous and must vanish everywhere, since any nonzero value would persist on an open interval of positive measure. ReLU vanishes on $(-\mathrm{\infty },0)$ but is positive on $(0,\mathrm{\infty })$; the sigmoid is bounded and nonconstant. These properties exclude a polynomial identity for each.

Applying the pointwise ${\mathcal{M}}_{\mathrm{p}\mathrm{t}}$ branch of [F2] separately to each activation proves the stated compact-uniform approximation property. Both activations and their networks are continuous, so they are already Borel representatives when the theorem's $L^p$ clause is used. The theorem is an existence statement only and supplies no uniform width bound.