Single-hidden-layer universal approximation

If $\sigma \in {\mathcal{M}}_{\mathrm{p}\mathrm{t}}$ is nonpolynomial almost everywhere, [F3] proves pointwise uniform density on every compact $K$. If $\sigma$ is polynomial almost everywhere, [F7] gives a specific compact segment on which density fails.

Now let $\sigma \in \mathcal{M}$ and put $C=\overline{D_{\sigma }}$, its closed null discontinuity set. Definition [F1] supplies the local essential bound and null-closure hypotheses. Define $\tilde{\sigma }=\sigma$ off $C$ and $\tilde{\sigma }=0$ on $C$. This is measurable and equals $\sigma$ almost everywhere. For a compact interval $J$, choose a larger compact interval $J^{\mathrm{\prime }}$ containing $J$ in its interior and an essential bound $M_{J^{\mathrm{\prime }}}$ for $\sigma$ there. At any $t\in J\setminus C$, continuity of $\sigma$ implies $\mathrm{\mid }\sigma (t)\mathrm{\mid }\le M_{J^{\mathrm{\prime }}}$: a strict violation would, by continuity, give an interval of positive measure inside $J^{\mathrm{\prime }}$ on which $\mathrm{\mid }\sigma \mathrm{\mid }>M_{J^{\mathrm{\prime }}}$. Thus $\tilde{\sigma }$ is pointwise locally bounded; outside $C$ it is continuous, so the closure of its discontinuities is contained in $C$. It belongs to ${\mathcal{M}}_{\mathrm{p}\mathrm{t}}$; if $\sigma$ is nonpolynomial almost everywhere, so is $\tilde{\sigma }$. For any nonconstant affine form $T(x)=w\cdot x+b$ and any Lebesgue measurable $A\subseteq \mathbb{R}$, [F16] writes $A=H\cup W$ with $H$ an $F_{\sigma }$ set and $W$ outer null. Then $T^{-1}(H)$ is Borel, hence Lebesgue measurable by [F17], and $T^{-1}(W)$ is measurable and null by [F5]. Thus $\sigma \circ T$ is Lebesgue measurable, as is every finite network; constant affine forms give constant measurable units. This makes the essential-supremum seminorm in the Statement well-defined for raw networks.

For the converse, suppose $\sigma =p_0$ almost everywhere for a polynomial $p_0$ of degree $q$. By [F5] and [F8], every nonconstant unit equals $p_0(w_j\cdot x+b_j)$ for $\mu$-almost every $x$; constant-input units can be absorbed into the output bias. Hence every network represents, in $L^p(\mu )$, a polynomial of total degree at most $q$.

For any finite $\tilde{\sigma }$-network, a unit with $w_j\mathrm{\ne }0$ differs from the corresponding $\sigma$-unit only on the null pullback $\{x:w_j\cdot x+b_j\in C\}$, null by [F5]. Replace the output bias by $c+{\sum }_{w_j=0}{a}_j(\tilde{\sigma }(b_j)-\sigma (b_j))$ to absorb the constant-input units. The resulting $\sigma$-network equals the $\tilde{\sigma }$-network almost everywhere in ${\mathbb{R}}^d$, and hence defines the same Borel $L^p(\mu )$ class under the representative convention when $\mu \ll {\lambda }_d$. For nonpolynomial-almost-everywhere $\sigma$, applying [F3] to $\tilde{\sigma }$ and transferring these approximants proves essential-supremum density for $\mathcal{M}$ on every compact $K$.

Assume $\mu (K)>0$ and let $P_q$ be the vector space of real polynomials in $d$ variables of total degree at most $q$. In a monomial basis its coefficient space is ${\mathbb{R}}^s$ for some finite $s\ge 1$, with its Euclidean norm. Every monomial is bounded on compact $K$, so for coefficient vectors $a,b$ one has ${\sup }_K\mathrm{\mid }{P}_a-P_b\mathrm{\mid }\le {\sum }_{\mathrm{\mid }\alpha \mathrm{\mid }\le q}\mathrm{\mid }{a}_{\alpha }-b_{\alpha }\mathrm{\mid }{\sup }_K\mathrm{\mid }{x}^{\alpha }\mathrm{\mid }$, which tends to zero as $a-b\to 0$. By [F12], $\mathrm{\parallel }{P}_a-P_b{\mathrm{\parallel }}_{L^p(\mu )}\le \mu (K{)}^{1\mathrm{/}p}{\sup }_K\mathrm{\mid }{P}_a-P_b\mathrm{\mid }$; the reverse triangle inequality from [F11] therefore makes $P\mapsto \mathrm{\parallel }P{\mathrm{\parallel }}_{L^p(\mu )}$ continuous in the coefficients. For a nonzero coefficient vector, $P$ is a nonzero polynomial; by [F6] its zero set is Lebesgue null, hence $\mu$-null by [F8]. If its $L^p$ norm were zero, [F11] would make its class zero, so $P=0$ $\mu$-almost everywhere. This would give $\mu (K\setminus \{P=0\})=0$, whereas [F8] and [F10] give $\mu (K\setminus \{P=0\})=\mu (K)>0$, a contradiction. Thus the norm is positive on the coefficient unit sphere. That sphere is nonempty and compact by [F9], and [F13] gives a minimum $a>0$; homogeneity yields $\mathrm{\parallel }P{\mathrm{\parallel }}_{L^p(\mu )}\ge a\mathrm{\mid }\mathrm{coeff}(P)\mathrm{\mid }$. Any $L^p$-Cauchy sequence in $P_q$ therefore has Cauchy coefficient vectors. By [F14], their finitely many coordinates converge to those of a polynomial $P\in P_q$; the coefficient estimate above and [F12] imply convergence to $P$ in $L^p$. If a class lies in the closure of $P_q$, Countable Choice [A1] supplies a sequence from $P_q$ converging to it; the preceding argument and uniqueness of norm limits [F11] put that class in $P_q$. Hence $P_q$ is closed in $L^p(\mu )$.

To justify the Borel convention, $C$ is closed, $\tilde{\sigma }=0$ on $C$, and $\tilde{\sigma }$ is continuous on the open complement of $C$, so $\tilde{\sigma }$ is Borel measurable. For any raw $\sigma$-network $N$, replace only its nonconstant units by $\tilde{\sigma }$-units and keep each constant-input value $a_j\sigma (b_j)$ in the output bias. The resulting Borel function $N^{\ast }$ is bounded on $K$ by pointwise local boundedness of $\tilde{\sigma }$ and compactness of each affine image. It equals $N$ outside the finite union of the closed Lebesgue-null sets $\{x:w_j\cdot x+b_j\in C\}$ with $w_j\mathrm{\ne }0$, by [F5]. Hence $N^{\ast }$ lies in $L^p(\mu )$. Any two Borel representatives of $N$ agreeing with it Lebesgue-almost everywhere differ on a Borel Lebesgue-null set, which is $\mu$-null by [F8], so they define the same class. This also justifies the polynomial representatives in step 1.3 and the transfers in step 2.1 without assuming raw networks are Borel measurable.

The polynomial $r(x)=x_1^{q+1}$ is not in $P_q$ modulo $\mu$-almost-everywhere equality: for every $P\in P_q$, $r-P$ is a nonzero polynomial, so [F6]–[F8] imply $\mu (\{x\in K:r(x)=P(x)\})=0<\mu (K)$. Thus the closed subspace $P_q$ is proper and cannot be dense. Step 1.3 places the whole network family inside it, proving failure of $L^p$ density for polynomial-almost-everywhere activations.

Suppose $\mu (K)>0$. By [F10], $\mu$ is a measure, and the real Borel $L^p$ convention is fixed in the Statement. To prove the positive $L^p$ sufficiency, choose any real class $[f]\in L^p(\mu ;\mathbb{R})$ and $\epsilon >0$. By [F4], choose $u\in C_c({\mathbb{R}}^d,\mathbb{R})$ with $\mathrm{\parallel }f-u{\mathrm{\parallel }}_{L^p(\mu )}<\epsilon \mathrm{/}2$. Apply [F3] to $\tilde{\sigma }$ on $K$ to choose a network $N_{\tilde{\sigma }}$ with ${\sup }_{x\in K}\mathrm{\mid }{N}_{\tilde{\sigma }}(x)-u(x)\mathrm{\mid }<\epsilon \mathrm{/}(2\mu (K{)}^{1\mathrm{/}p})$. By step 2.1 replace it by a $\sigma$-network equal $\mu$-almost everywhere. Using the Borel representative supplied by steps 2.1 and 3.1, equality of classes and [F12] give $\mathrm{\parallel }N-u{\mathrm{\parallel }}_{L^p(\mu )}=\mathrm{\parallel }{N}_{\tilde{\sigma }}-u{\mathrm{\parallel }}_{L^p(\mu )}\le \mu (K{)}^{1\mathrm{/}p}{\sup }_K\mathrm{\mid }{N}_{\tilde{\sigma }}-u\mathrm{\mid }$. The triangle inequality [F11] yields $\mathrm{\parallel }N-f{\mathrm{\parallel }}_{L^p(\mu )}<\epsilon$.

If $\mu =0$, every measurable real function represents the zero class in $L^p(\mu ;\mathbb{R})$, and the network family contains zero by width zero and output bias zero. Thus it is dense regardless of $\sigma$. Together with steps 4.1 and 3.2, this proves the stated $L^p$ equivalence and its zero-measure boundary.

Let $K=[0,1{]}^d$, which is compact by [F9], and define $\mu (A)={\lambda }_d(A\cap K)$ for Borel $A$. By [F15], $\mu (K)=1$. This is a finite Borel measure concentrated on $K$ and absolutely continuous with respect to ${\lambda }_d$. It is regular: inner regularity follows by applying [F18] to $A\cap K$; for outer regularity, apply [F18] to $K\setminus A$ to choose a compact $F\subseteq K\setminus A$ with ${\lambda }_d((K\setminus A)\setminus F)<\epsilon$, and use the open set ${\mathbb{R}}^d\setminus F\supseteq A$, for which $\mu (({\mathbb{R}}^d\setminus F)\setminus A)<\epsilon$. Thus this $\mu$ satisfies the theorem's measure hypotheses. If a polynomial-almost-everywhere activation gave essential-supremum approximation to every continuous target there, [F12] and $\mu (K)=1$ would turn those approximations into $L^p(\mu )$ approximations, contradicting step 3.2 for $f(x)=x_1^{q+1}$. This proves the essential-supremum obstruction on the cube and completes both compact-set equivalences.