One-hidden-layer particle-measure model

Assume AC for the flow clause below. Let $(\mathsf{X},\mathcal{A},{\rho }_X)$ be an input probability space, let $\mathcal{H}=L^2({\rho }_X)$, let $\sigma :\mathbb{R}\to \mathbb{R}$ be measurable, and put $\mathrm{\Theta }={\mathbb{R}}^{p+1}$ for hidden affine parameters $\theta =(w,b)$. Assume ${\varphi }_{\theta }(x)=\sigma (w\cdot x+b)$ belongs to $\mathcal{H}$ for each $\theta$. On $\mathrm{\Omega }=\mathbb{R}\times \mathrm{\Theta }$, define the feature map $\mathrm{\Phi }(a,w,b)=a{\varphi }_{(w,b)}\in \mathcal{H}$. For a finite nonnegative Borel measure $\mu$ on $\mathrm{\Omega }$ for which the Bochner integral exists, its particle-measure predictor is

$f_{\mu }={\int }_{\mathrm{\Omega }}\mathrm{\Phi }(u)d\mu (u)\in \mathcal{H}\mathrm{.}$

The corresponding objective is $F(\mu )=R(f_{\mu })+{\int }_{\mathrm{\Omega }}Vd\mu$, whenever the scalar integral is finite, with $R:\mathcal{H}\to [0,\mathrm{\infty })$ a risk functional and $V:\mathrm{\Omega }\to [0,\mathrm{\infty })$ a regularizer. For the empirical particle law with integer $m\ge 1$ defined on measurable $A\subseteq \mathrm{\Omega }$ by ${\mu }_m(A)=m^{-1}\mathrm{\#}\{j:(m{a}_j,w_j,b_j)\in A\}$, one has $f_{{\mu }_m}(x)={\sum }_{j=1}^m{a}_j\sigma (w_j\cdot x+b_j)$, so every positive-width finite one-hidden-layer network is a particle-measure predictor (with an optional constant output bias added separately). For width $m=0$, the zero finite nonnegative measure $\mu =0$ has Bochner predictor $f_0=0$ and represents the empty-sum network before adding its optional output bias. This zero measure is a finite-predictor representation; it is not a probability law for the Wasserstein-flow clause. A feature-energy Wasserstein gradient flow means an $A{C}_{\mathrm{l}\mathrm{o}\mathrm{c}}^2$ curve of probability measures satisfying the finite-action continuity equation with Borel velocity equal almost everywhere to the negative minimum-norm element of the constrained subdifferential of the first-variation potential. This is the earlier model-flow definition; no existence or uniqueness follows from this definition. The flow use below requires all of the feature-energy regularity hypotheses. In particular, the classical ReLU feature map is not asserted here to meet those hypotheses merely because it is positively homogeneous.

Feature-measure energy and its regularity assumptions

Wasserstein gradient flow for a feature-measure energy

Single-hidden-layer network realization

Population and empirical risk

Chizat and Bach, Sections 2.1–2.3

Ambrosio, Gigli, and Savaré, Chapters 8 and 10